The payment bug that hurts is rarely a 500 error. It is the quiet mismatch: a customer was charged but the order still says unpaid, or an order was marked paid twice and shipped twice. Nobody sees it until finance closes the month or a customer calls with a screenshot of their bank statement.
Every payment integration I have worked on, from card gateways to buy-now-pay-later providers, eventually meets the same failure modes. The tools differ; the discipline that prevents lost or doubled money is the same. Here it is.
1. The provider owns the truth, your database caches it
Your system does not decide whether money moved. The payment provider does. Your database holds your best current knowledge of what the provider decided, and that knowledge can be late, incomplete, or arrive out of order.
That framing changes how you model a payment. It is not a boolean is_paid column. It is a small state machine:
created -> pending -> authorized -> captured -> partially_refunded -> refunded
\-> failed
\-> expired
Each transition is triggered by evidence from the provider: a webhook, or a server-side status query. Nothing else moves the state forward.
2. Redirects are hints, webhooks are the record
After checkout, the provider redirects the customer back to your site. It is tempting to mark the order paid right there. Don't.
The redirect can be lost: the customer closes the tab, the mobile app is killed, the network drops. It can also be forged, because it passes through the customer's browser. Treat the redirect as a hint that something happened: show a "confirming your payment" screen, then confirm on the server, either by waiting for the webhook or by querying the provider's API with your own credentials.
The webhook, signed by the provider and delivered server to server, is the record you act on.
3. Idempotency, in both directions
Networks fail in the worst possible place: after the other side did the work but before you heard about it. Retries are therefore mandatory, and retries are only safe when operations are idempotent.
Outbound. When you create a charge or a refund, send an idempotency key. Some providers support an idempotency header directly. For those that don't, generate your own merchant reference per attempt, store it before the call, and on retry query the provider by that reference before creating anything new. A refund you sent twice is a refund you paid twice.
Inbound. Webhooks are delivered at least once, sometimes several times, sometimes hours apart. Record every event ID with a uniqueness guarantee and make the handler a no-op for duplicates:
CREATE TABLE payment_events (
id BIGINT PRIMARY KEY,
provider VARCHAR(32) NOT NULL,
event_id VARCHAR(128) NOT NULL,
payload JSON NOT NULL,
processed_at TIMESTAMP NULL,
UNIQUE (provider, event_id)
);
The unique constraint, not an if exists check in application code, is what protects you when two deliveries of the same event arrive at the same millisecond.
4. Verify, then trust
A webhook endpoint is a public URL that moves money in your books. Treat it like one.
- Verify the signature against the raw request body, before any JSON parsing or framework normalisation changes a single byte.
- Check the timestamp in the signed payload and reject events outside a short tolerance window. That blocks replay of a captured, validly signed request.
- Never trust amounts from the client. Compare the amount and currency in the provider's event with the order total you calculated on the server. A mismatch is an alert, not a payment.
5. Acknowledge fast, process later
Providers expect a 2xx response quickly and retry when they don't get one. If your handler sends emails, updates stock, and calls a delivery API before responding, a slow dependency turns into duplicate deliveries and retry storms.
The pattern that holds up:
- Verify the signature.
- Insert the raw event (the unique constraint deduplicates it).
- Enqueue a job with the event's internal ID.
- Return 200.
The job does the real work inside a database transaction, with retries and backoff, and moves to a dead-letter queue with an alert after the last attempt. A poison message should page a human, not loop forever.
6. Events arrive out of order
A refund notification can arrive before the capture notification it refers to. An "authorized" event can arrive after "captured" because it was retried. If your handler applies every event blindly, state goes backwards.
The state machine from step 1 is your guard. Each handler checks that the transition is valid from the current state. Transitions that would move backwards are recorded and ignored. When an event refers to something you haven't seen yet, fetch the current object from the provider's API instead of guessing.
7. Buy now, pay later is a different shape
Instalment providers usually split the flow into authorize and capture. You authorize at checkout and capture when the order ships. That adds failure modes:
- Authorizations expire. An order that sits in the warehouse too long can no longer be captured, so track the expiry and alert before it passes.
- Captures and refunds can be partial. Model them as separate records linked to the payment, not as edits to a single amount.
- Cancelling an order before shipment means voiding the authorization, not refunding money that was never captured.
Card gateways look simpler until you enable 3-D Secure, saved cards, or partial refunds, and then they look exactly like this.
8. Reconciliation closes the loop
Even with all of the above, something will slip: a webhook endpoint that was down during a deploy, a provider incident, a manual refund done from the provider's dashboard.
Reconciliation is the safety net. Once a day, pull the provider's list of transactions for the period and compare it with your records:
- Paid but not recorded: money arrived, the order doesn't know.
- Recorded but not paid: your system thinks it was paid, the provider disagrees.
- Amount or currency mismatch.
- Refunds made outside your system.
Fix only the safe cases automatically, such as replaying a missed event. Everything else becomes a ticket for a human with the evidence attached. The goal is that by the time finance asks, the answer already exists.
Money is not a float
One last rule underneath all of this: store amounts as integers in the currency's minor unit, together with the currency code, and know each currency's exponent. Most currencies have two decimals; some, such as the Jordanian dinar, have three. A floating-point total that is off by a fraction is still a reconciliation failure, just a slower one.
The checklist
- Payments are a state machine driven by provider evidence, not a boolean.
- Redirects only show status; server-side confirmation moves the state.
- Idempotency keys on every outbound create; unique event IDs on every inbound webhook.
- Signatures verified on the raw body, with a timestamp window.
- Webhooks acknowledged fast, processed in queued jobs with retries and a dead-letter queue.
- Out-of-order events can't move state backwards.
- Authorizations, captures, voids, and partial refunds are separate records.
- A daily reconciliation job reports every mismatch.
None of this is exotic. It is the difference between a payment integration that works in the demo and one that is still correct after a year of real traffic.
