Twenty Minutes After Launch

A fictional incident review about cache identity, bilingual routes, honest 404s, evidence preservation, and bounded recovery.

Published
14 Nov 2025
Updated
2 Aug 2026
Reading
3 min
Tags
testing, network, web, engineering
阅读中文版本

The symptom

Twenty minutes after launch, the English article URL returned Chinese HTML for some visitors. Refreshing sometimes fixed it, which made the problem look random. It was not random. The response cache used only the article Slug and ignored the requested locale.

The application itself queried content correctly. The error appeared only after a response passed through the shared edge cache, so local development and the first production smoke test both looked healthy.

Timeline

  1. 09:00: The new release reached production.
  2. 09:08: The first Chinese article request populated the edge cache.
  3. 09:20: An English reader reported the wrong language.
  4. 09:24: The team reproduced the issue with a clean browser profile.
  5. 09:31: New cache writes were disabled while existing evidence was preserved.
  6. 09:42: The key was changed to include locale and normalized pathname.
  7. 09:55: Cross-locale verification passed and caching resumed. The useful log record was small:
event=content.response
locale=en
pathname=/blog/sample
cache_key=article:sample
cache_status=HIT

The requested locale and derived cache identity disagreed. That was enough to replace speculation with a testable cause.

Impact

Area Result
Availability Pages returned successfully
Correctness Some English requests received Chinese content
Security No private data was exposed
Duration Approximately 35 minutes
Root cause Locale omitted from cache identity

Why the checks missed it

  • Local development did not use the production edge cache.
  • The smoke test requested only one locale.
  • The cache-key builder had unit coverage for paths but not locale pairs.
  • A successful HTTP status was treated as sufficient verification.
  • The initial test did not compare meaningful body text between languages.

A successful response with the wrong meaning is still a failure.

Corrective actions

  • Include locale and normalized pathname in the cache key.
  • Purge affected responses.
  • Preserve one incorrect response as a sanitized test fixture.
  • Add an automated cross-locale cache isolation test.
  • Log the derived key in non-sensitive diagnostics.
  • Document how to disable cache writes without disabling reads.
  • Require body-language assertions in production smoke checks. The recovery deliberately changed one boundary at a time. First the team stopped creating new incorrect entries, then confirmed the cause, then changed the key, and finally restored caching. Purging everything before collecting evidence would have made the incident harder to understand.

Why the 404 rule mattered

The application already treated a missing translation as a real 404. That rule reduced the repair surface: the cache could be corrected without inventing fallback behavior or deciding which language should silently replace another. Locale identity remained explicit from route to repository.

Bojin Li

Writes about software, systems, and the parts that are still uneven.