Work snippet
Idempotent webhook intake and a versioned field map in n8n
Concept DesignIllustrative integration walkthrough. Business details, figures and records are worked examples.
Several vendors post webhooks into this service group, and each used to have its own flow and mapping. Now a single n8n entry point checks the signature, keeps the raw payload, maps fields from a versioned table, and writes to the job system or CRM keyed on the event, so a resent event updates the same record instead of adding one.
The problemEach webhook has its own fragile flow, and vendor retries create second records.
What changesOne intake, one versioned field map, and no duplicates from retries.
- Area
- Inbound integration architecture
- Tools
- n8n · ServiceM8 · HubSpot
- Typical scope
- US$1,600-3,000
- Timeline
- About 4 weeks · fixed milestones
The problem
Separate mappings per vendor, several with no unique key and nothing saved, meant a resent event created a duplicate and a failed run had nothing to replay from.
- The same customer fields are spelled differently in each flow
- A vendor retry has already created a second job
- A failed mid-run cannot be replayed because the body is gone
- A live edit to one mapping leaves no current version
What discovery would examine
A review of eight inbound webhooks into n8n for a service group that already has several point-to-point zaps, plus two weeks of execution history.
| ID | Finding | Evidence | Risk if left |
|---|---|---|---|
| F1 | Each webhook has its own mapping, with the same customer fields spelled three ways | Eight workflows, three names for "mobile" | A later change has to be made in several places, and some are missed |
| F2 | Two webhooks have no idempotency key | A vendor retry created a second job and a second contact | Duplicates that operations then has to merge |
| F3 | Payloads are not stored, so a failed mid-run cannot be replayed safely | Execution log has the error, not the body | The vendor must be asked to resend, or the item is lost |
| F4 | A change to a downstream field was edited live in one flow, with no version note | The mapping in n8n disagreed with the written sheet | Nobody could say which was current |
| F5 | Slow vendors time out, and the same event arrives again later | Timeouts followed by a second post | Without a key, the second post is treated as new work |
Approach
- Put every event through one intake
Verify, store, key.
- Hold one mapping table
Versioned, dated, and used by every writer.
- Make every writer idempotent
The same key cannot create a second record.
- Replay from what was stored
After a fix or a mapping change, with the version recorded.
Current state and future state
Current state
Future state
Future-state modules
How it is built
- Module AIntake and verify
One n8n entry workflow checks the signature, rejects stale events, and stores the raw payload.
- Module BIdempotency key
A written key per vendor, usually event id plus vendor name. A known key is not written again.
- Module CMapping table
A single table of source paths to destination fields, with a version and a date.
- Module DIdempotent writers
Each destination uses the same key, so a replay cannot create a second record.
- Module EReplay and versioning
A stored payload can be replayed after a mapping change. The version used is recorded on the run.
Field mapping
| Destination field | Source path (typical) | Transform | If missing |
|---|---|---|---|
| External event key | event.id or delivery id | Prefix with the vendor code | Refuse the event, alert |
| Customer email | customer.email or contact.email | Lower-case, trim | Park if no phone either |
| Mobile | customer.mobile or phone | Normalise to a single format | Park if no email either |
| Site line 1, suburb, postcode | site.address_* | Join only when all required parts exist | Park with the missing part named |
| Requested window | schedule.start / schedule.end | Store as received, in the written timezone | Job is unscheduled |
| Source system | Vendor name from the intake | Never taken from a free-text field | Always set by the intake, not the payload |
Worked exhibits
Exceptions and runbook
| Case | Rule | Who acts | First action |
|---|---|---|---|
| Unknown field in a new payload version | Park the event, do not guess a mapping | Operations and me, if monthly support is in place | Add a versioned row to the map, then replay |
| Missing required field | Park, do not create a half record | Operations | Ask the vendor or complete by hand, then replay |
| Vendor retry after a timeout | The key already exists: do not write again, return success to the vendor if the first write completed | Flow | If the first write did not complete, replay from the stored payload |
| Mapping change | A new version is dated; old stored events can be replayed under the new version only if the owner agrees | Owner | The run record shows which version was used |
How to read the exhibits
- The two-week counts show intake, refusals and parked events across the flow.
- The stored event is what makes a replay safe after a mapping change.
- Source system is set by the intake, not by a free-text field in the payload.
Test plan
| ID | Check | Method | PASS when | FAIL when |
|---|---|---|---|---|
| T1 | Happy path | Post a complete signed event for each vendor | One stored payload, one destination write | A missing store, or a write with no key |
| T2 | Retry | Post the same key twice | One destination record | A second job or contact |
| T3 | Bad signature | Post with a wrong secret | Refused, not stored as a valid event | A record is created |
| T4 | Missing field | Omit the email and the phone | Parked, no destination write | A record with a blank customer |
| T5 | Replay after a map change | Change a mapping version, replay a stored event | The run records the new version; still one destination record | A second record, or the version is not recorded |
Scope
| Area | Included |
|---|---|
| Discovery | Existing webhook flows, keys, and two weeks of history, with written findings |
| Shared intake | Signature checks, payload store and event keys |
| Versioned mapping table | Source paths, transforms and missing-field rules |
| Idempotent writers | To the agreed destinations |
| Replay | From stored payloads, with the mapping version recorded |
| Test plan | The PASS and FAIL checks in this write-up |
| Handover and support | SOPs, walkthrough, 14 days of fixes, optional monthly hour block |
| Not included | Rebuilding the vendors themselves, or adding a new destination that is not agreed in the scope |
Rollout, handover and monthly support
| Milestone | Deliverable | Timing | Share of fee |
|---|---|---|---|
| M1 Discovery and keys | Findings, the key design and the first mapping version | Week 1 | 25% |
| M2 Intake and writers | Store, map and write in a copy, replay safe | Week 2 | 35% |
| M3 Remaining vendors and tests | All agreed vendors on the shared intake | Week 3 | 25% |
| M4 Handover | SOPs, versioning rules, 14 days of fixes | Week 4 | 15% |
Handover
- Written operating procedures for the happy path and each exception in the runbook
- A recorded walkthrough of the architecture, the queue and how to replay an item
- A credentials and ownership list, with every connection in your name
- Fourteen days of fixes after go-live, for defects in the agreed scope
Monthly support
A set block of hours each month, at US$35 per hour, agreed in writing before it starts. Typical use of the block: new vendor fields, mapping versions, and parked events that need a person
- Review of the exception queue and the weekly digest
- Small mapping or rule changes, agreed before they are made
- A short written note on what was done and what is still open
Typical scope for this kind of project: US$1,600-3,000
About 4 weeks · fixed milestones. The systems health check (US$150, fixed) comes first and confirms the scope. Extra hourly work is US$35 per hour, agreed before it starts. After handover, a set block of hours each month is available at the same rate, agreed in writing, to watch the flows, handle changes and work the exception queue.
Discuss an engagement.
Outline the operational problem through the contact form. I respond with a recommended starting point, including whether the US$150 systems health check is the appropriate first step.