Failure is part of the interface
An automation is not complete when the happy path runs once. It is complete when a future operator can tell what happened, what changed, and what is safe to try next. Scheduled work runs when nobody is watching, so its failure path must carry more context than an interactive button.
I prefer a result type that separates expected operational failures from programmer mistakes. The caller can then decide whether to retry, alert, or stop without parsing arbitrary text.
type RunResult =
| { ok: true; processed: number; cursor?: string }
| {
ok: false
code: 'UPSTREAM_TIMEOUT' | 'INVALID_INPUT' | 'PARTIAL_WRITE'
retryable: boolean
message: string
}
async function runJob(): Promise<RunResult> {
const input = await loadValidatedInput()
try {
const processed = await processBatch(input)
return { ok: true, processed }
} catch (error) {
return classifyOperationalFailure(error)
}
}
The type is only one part of the contract. The job also needs a stable idempotency key for writes, a timeout shorter than the scheduler deadline, structured logs with a run identifier, a bounded retry policy, and a clear rule for partial progress.
| Failure | Unsafe reaction | Polite response |
|---|---|---|
| Upstream timeout | Retry forever | Stop after a bound and preserve the cursor |
| Invalid input | Skip silently | Reject before writing and identify the field |
| Partial write | Restart the full batch | Reconcile the smallest confirmed unit |
| Unknown error | Mark as retryable | Stop, capture evidence, and require review |
Recovery order
- Stop repeated writes when idempotency is uncertain.
- Capture the run identifier and last confirmed cursor.
- Verify whether upstream state changed after the failure.
- Compare intended writes with confirmed writes.
- Retry only the smallest safe unit.
- Record the resolution beside the original alert. A useful alert is specific without exposing secrets. “Job failed” is barely information. “Import stopped after batch 12 because the content API timed out; no writes occurred after cursor 8f2; safe to retry” gives the next person a starting point.
- The same batch can run twice without duplication.
- Secrets and full request bodies never appear in logs.
- Validation completes before the first write.
- A timeout returns a recognizable code.
- A partial write has a documented reconciliation path.
- Retry delays have a maximum attempt count.
- A dry-run mode can describe intended changes.
When should the job throw?
Throw for violated programmer assumptions or states the caller cannot safely interpret. Return a typed operational result for expected external failures. The distinction should remain small and documented; an enormous error hierarchy often hides the same ambiguity it was meant to remove.
The best automation does not require a heroic operator. It leaves a narrow, well-lit path from failure back to a known state.