List the failure modes first
Before writing the automation, enumerate how it can break:
- Input failure. Missing, malformed, or unexpected data.
- Dependency failure. An API is down, slow, or returns a changed format.
- Partial failure. Step 3 of 5 succeeded — what is the state of the world now?
- Silent failure. The worst kind: it ran, reported success, and did the wrong thing.
The recovery toolkit
- Retries with backoff for transient failures — but cap them, and never retry non-idempotent actions blindly.
- Dead-letter handling: park failed items somewhere visible instead of dropping them.
- Idempotency: design steps so re-running them is safe; partial failures become recoverable.
- Alerting with context: alert on what failed, with the payload attached — 'step failed' with no data is unactionable.
- Human fallback: a documented manual path for when the automation must be bypassed.
The pre-handoff checklist
- Every external call has a timeout.
- Failures are logged with enough context to diagnose without re-running.
- Someone specific gets the alert, and knows what to do with it.
- The manual fallback has been tested at least once.
- There is a kill switch — a way to stop the automation without deploying code.
Assumptions and limits
- Guidance is general; specific tooling varies by stack.
Common questions
What is a dead-letter queue in automation?
A holding area for items an automation could not process — failed messages, rejected records, errored jobs. Instead of dropping them silently, they wait somewhere visible for inspection and reprocessing. Dead-letter handling is part of the recovery toolkit designed before handoff, alongside retries with backoff, idempotency, alerting with context, and a documented human fallback for items that cannot be recovered automatically.
How many retries should an automation attempt?
Enough to ride out transient blips, few enough to fail fast on real problems — typically a small capped number with exponential backoff. And never retry an action that isn't safe to repeat (like charging a card) without idempotency. Retries sit inside a wider checklist: every external call has a timeout, and failures are logged with enough context to diagnose without re-running.
What is the most common automation failure?
Changed dependencies — an API alters its response format, a website redesigns, a credential expires. Automations rarely break from their own logic; they break because the world around them moved. That is why the failure modes are listed before any code is written — input failure, dependency failure, partial failure, and silent failure, where the automation runs, reports success, and does the wrong thing.
Published · Updated . Automation planning guides for founders and developers. Models are planning tools, not guarantees.