CLOUD & DEVOPS / OBSERVABILITY
Why Observability Should Be Designed Before Production
Signals, ownership and useful failure context are architecture concerns.
Key Insight: The question "how will we know this is broken, and why" is an architecture question. Answered after launch, it's always more expensive and always less complete than answered during design.
Observability gets treated as an operations concern — something you add once a system exists, usually right after an incident makes the gap painful. That ordering is backwards. By the time you're retrofitting observability onto a live system, you're guessing at which signals would have mattered, instrumenting code you no longer remember the details of, under pressure, after the fact.
Design the signals alongside the components
Every meaningful component in a system architecture should come with an answer to three questions, decided at design time: what does healthy look like for this component, what does degraded look like, and what's the earliest signal that distinguishes the two? A queue's depth and consumer lag matter more than whether the consumer process is technically "up." An API's p99 latency under real load matters more than its uptime percentage. These are architecture-level decisions about what the system needs to expose about itself, not implementation details to backfill later.
Designing this alongside the component, rather than after, means the code is instrumented to emit the right signal from day one, and the team building it understands what failure actually looks like for what they're building — which is valuable independent of the tooling.
The three pillars only work together
Metrics tell you something is wrong. Logs tell you what happened. Traces tell you where, across a distributed call chain, it went wrong. Systems that only invest in one pillar have a permanent blind spot. A metrics-only system can tell you error rate spiked at 14:32 and nothing about why. A logs-only system without correlation IDs threaded through a request's full path leaves you grepping across a dozen services trying to reconstruct a single user's failed request by hand.
The architectural decision that makes all three pillars actually useful together is boring and essential: propagate a correlation or trace ID through every hop of every request, from the edge to the database, from day one. Retrofitting this into an existing distributed system is genuinely painful. Designing it in from the start costs almost nothing.
Alerts need an owner and a reason to exist
An alert that fires and nobody acts on is worse than no alert — it trains the team to ignore the channel. Every alert should map to something a human can actually do about it, and every alert should have a named owner, not "whoever is on the rotation figures it out." Alert on symptoms that matter to users (latency, error rate, saturation) rather than every possible internal state, and treat a growing pile of unactioned alerts as a design defect, not a discipline problem.
Failure context is a design decision, not a debugging accident
When something breaks in production, the difference between a 10-minute diagnosis and a 3-hour one is almost always whether the system was designed to expose useful context about its own failures. That means: structured logging with the fields that actually matter (user ID, request ID, the specific operation, the specific dependency called), not free-text log lines. It means error messages that state what was attempted and what failed, not generic exceptions. This has to be a deliberate choice made while building the component, because nobody adds this kind of detail retroactively with the same accuracy.
Key takeaways
Decide what healthy and degraded look like for each component during design, not after an incident defines it for you. Propagate correlation IDs through every request from the start — this is nearly impossible to retrofit well. Make sure every alert has an owner and a clear action, or remove it. Build structured, contextual failure information into components as you write them. Observability designed in advance is architecture. Observability added after an outage is archaeology.