Engineering / Field guide
Prevent duplicate orders when an automation retries
A timeout can mean the order failed, or that it succeeded and the acknowledgment never arrived. Those cases need different recovery paths.
2XD Studio · · 2 min read
Identify the business event
Choose a stable key such as supplier, order number, and revision. An upload timestamp makes a poor key because a retry gets a new timestamp. A customer name alone is equally risky because two legitimate orders may come from the same customer.
Separate receipt from completion
Store the incoming event and its processing state. When a duplicate arrives, check whether the earlier run is pending, confirmed, or unresolved. If the destination supports an idempotency key, reuse the same key across attempts. If it does not, look for an existing order reference before creating another order. A local log alone cannot prove what a remote system committed.
Rehearse the lost acknowledgment
In a test environment, let the destination accept an order and interrupt the response. Then retry. The expected result is one destination order and an inspectable processing history. Also try two simultaneous deliveries and a legitimate revision. A design that rejects all repeats may prevent duplicates while silently throwing away corrections.
Read an ambiguous run
At 09:02 the destination creates order 481. At 09:03 the sender times out. A second submission at 09:05 should find the pending event and reconcile it against order 481. Ask the operator to explain the sequence using only the run history. If they must infer success from a missing error message, the recovery process is still incomplete.
Reader questions
Can we use a hash of the file?
It helps identify an identical file, but two exports of the same business event can differ. Use a business key where one exists.
What should an operator do with an uncertain result?
Check the destination record first. Keep manual replay disabled until the operator can distinguish a missing order from an unacknowledged one.
Further reading Amazon Builders’ Library: making retries safe