On the Wednesday, a finance reconciliation flagged two hundred and fourteen orders taken over the previous weekend that had been paid for and never dispatched. Four of those customers had already telephoned. Nothing in any log was an error, no alert had fired, and every dashboard for that weekend is green, because as far as our systems were concerned nothing failed.
The order service writes the order row, commits, and then publishes an event that fulfilment consumes. On Saturday morning the broker was unreachable for twelve minutes during maintenance on its own cluster. The publish threw. Somebody, years ago and for a perfectly humane reason, wrapped that call in a try and logged the failure at warning level, on the grounds that the order is already saved and we should not fail a customer's checkout over a messaging problem. So the customer got a confirmation, the row existed, the payment was captured, and the only trace of the missing half was two hundred and fourteen warning lines in a log nobody queries.
Two writes to two systems inside one request have four outcomes and we had designed for two. The state where the first succeeds and the second does not is not an error condition. It is a system that has recorded something untrue about itself and carried on with complete confidence.
The event is now written into an outbox table in the same transaction as the order, so the two facts commit or fail together. A relay reads that table, publishes, marks the row sent, and retries indefinitely, which means delivery is explicitly at least once and consumers dedupe on the event id. The single most useful thing to come out of it is an alert on the age of the oldest unsent outbox row, which is two minutes and has fired three times since, each time for a reason we would otherwise have met on a Wednesday. We also stopped swallowing that exception.
A system that can only record its successes will lose exactly the work that failed in the middle. If two things have to be true together, they need to be written together, or you need something whose job is to notice that they are not.
– Sergey Shinder
Top comments (0)