Checkout error rate is high

When this applies

Customers are getting 5xx responses from POST /checkout on web. Web passes checkout's status through, so the errors almost always come from the checkout service or something it calls.

First five minutes

1. Look at the error split by status on web's /checkout route. 500 means checkout threw; 503 means checkout could not get a database connection; 502 or 504 means checkout or payments did not answer.
2. Search checkout's error logs for the most common message and stack trace. One dominant signature usually means one cause.
3. Check what changed in the last hour: deploys (with their commit messages), feature flag changes and config changes. A change that lands one or two minutes before the errors start is the first suspect, but only if the errors match what the change touched.
4. Check whether the errors are limited to some users (region, coupon, flag cohort). Errors for a subset point at data or a flag; errors for everyone point at a dependency or a resource limit.

Mitigations

• Errors started right after a checkout deploy and the stack trace points into code that changed: roll back (see rb-002).
• Errors only for users in a flag cohort: switch the flag off (rb-011).
• 503s with "pool timeout": see rb-004.
• Do not restart checkout to "see if it helps" when the cause is a bad release; it comes back with the same code.