Automating Observability Instrumentation
Making every service observable by default — auto-instrumentation, generated dashboards and alerts, and the naming conventions that make generation possible.
Observability that depends on each team remembering to instrument their service produces exactly the coverage you would expect: excellent on the services whose authors care, absent on the ones that page you at 3am. The fix is to make instrumentation a property of the platform rather than a task on a backlog.
The three signals and what each is for
| Signal | Answers | Cost profile | Cardinality tolerance |
|---|---|---|---|
| Metrics | “Is something wrong, and how wrong?” | Cheap, constant | Low — every label combination is a time series |
| Logs | “What exactly happened in this case?” | Expensive at volume | High |
| Traces | “Where in the request path did it go wrong?” | Moderate, usually sampled | High |
The standard workflow is metrics → traces → logs: an alert fires on a metric, a trace identifies the slow or failing span, and logs for that specific trace ID explain why. Instrumentation that does not support that path — logs with no trace correlation, traces with no link from the alert — costs money without shortening any incident.
Auto-instrumentation
Modern instrumentation libraries can capture the majority of useful telemetry without application code changes, by hooking the frameworks a service already uses.
flowchart TD
A[Service starts] --> B{Auto-instrumentation<br/>agent or SDK}
B --> C["HTTP server: request rate,<br/>duration, status codes"]
B --> D["HTTP/gRPC client: outbound<br/>calls, latency, failures"]
B --> E["Database drivers: query<br/>duration, connection pool"]
B --> F["Message consumers: lag,<br/>processing time"]
B --> G["Runtime: memory, GC,<br/>threads, CPU"]
C & D & E & F & G --> H[Standard telemetry,<br/>zero application code]
H --> I["Teams add only<br/>domain-specific signals"]
This should be injected by the platform — as a sidecar, an init container, a base image layer, or a build-time dependency in the service template — not left to each team to add. Teams then supply only what auto-instrumentation cannot know: business events, domain-specific counters, and meaningful span attributes.
Adopting a vendor-neutral standard for this layer matters more than the specific backend. Instrumentation is expensive to write and painful to redo; keeping it independent of the analysis tool means a backend migration is a configuration change rather than a re-instrumentation project.
Naming conventions make generation possible
You cannot generate dashboards and alerts for services whose metric names are unpredictable. Conventions are the prerequisite for every automation in this section.
# Metric naming
<namespace>_<subsystem>_<name>_<unit>
http_server_request_duration_seconds
db_client_connections_open
queue_consumer_lag_messages
# Required labels on every service metric
service.name # matches the service catalogue entry
service.version # the deployed artifact version
deployment.environment
Two rules avoid the most expensive mistakes:
- Never put unbounded values in labels. User IDs, request IDs, full URL
paths, and error messages create one time series per distinct value. This is
the standard way to take down a metrics backend, and it typically happens the
first time someone adds
pathas a label without normalising it to a route template. - Units in the name, base units in the value. Seconds, not milliseconds; bytes, not megabytes. Every dashboard that has to guess the unit eventually guesses wrong.
Generating dashboards and alerts
Once conventions hold and every service has ownership metadata, dashboards and alerts become derived artifacts rather than hand-crafted ones.
flowchart LR
A[Service catalogue entry<br/>name · tier · owner · oncall] --> B[Generator]
C[Standard metric names] --> B
D[Service tier → SLO targets] --> B
B --> E[Golden-signal dashboard<br/>rate, errors, duration, saturation]
B --> F[SLO burn-rate alerts<br/>routed to the owning team]
B --> G[Dependency view<br/>from the catalogue graph]
E & F & G --> H[Committed as code,<br/>reviewable and diffable]
The result is that every service has a competent baseline on the day it is created, and teams customise from a working starting point instead of an empty page. Generated artifacts should be stored as code so they are versioned and reviewable — a dashboard edited by hand in a UI is lost the next time it is regenerated.
Structured logs, correlated
Unstructured log lines are text that must be parsed with fragile regular expressions. Structured logs are queryable data.
{
"timestamp": "2026-08-19T09:14:22.481Z",
"level": "error",
"service.name": "payments-api",
"service.version": "1.14.2",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"message": "payment authorisation failed",
"payment.provider": "acquirer-a",
"error.type": "UpstreamTimeout",
"duration_ms": 5001
}
The trace_id field is what turns three separate telemetry systems into one
investigation. Without it, an engineer correlates by timestamp and hope.
Two operational constraints belong in the logging library, not in each service:
- Automatic redaction of tokens, card numbers, and personal data. Relying on developers to remember is how personal data ends up in a log aggregator with a two-year retention policy.
- Sampling for high-volume paths. Log every error; log a fraction of successes. Logging cost otherwise scales linearly with traffic, and log volume is the line item that surprises people.
Trace sampling
Tracing every request is usually unaffordable, and tracing randomly means the interesting requests are the ones you did not keep.
| Strategy | Decision made | Trade-off |
|---|---|---|
| Head-based | At request start, before the outcome is known | Cheap and simple; misses rare failures |
| Tail-based | After the trace completes, using its outcome | Keeps all errors and slow requests; needs a buffering collector |
| Hybrid | Head-sample a baseline, force-keep errors and slow requests | Practical default for most estates |
Tail-based sampling with “always keep errors and traces above the latency threshold” gives the best signal per unit of cost, because the traces you keep are the ones you would actually look at.
Cost control
Observability bills grow superlinearly with traffic and are frequently the second-largest infrastructure cost after compute. Control it structurally:
- Attribute cost per service using the same ownership labels, so teams see their own spend.
- Set retention by signal and by value: high-resolution metrics for two weeks and downsampled for a year; error logs longer than debug logs.
- Alert on ingestion volume spikes. A logging loop in a new release can add thousands of dollars in a day, and the first sign is usually the invoice.
- Review cardinality regularly. Metrics cardinality only ever grows by accident.
Adoption checklist
- Auto-instrumentation is injected by the platform, not added per team.
- Metric and label naming conventions are documented and enforced in review.
- No unbounded values appear in metric labels.
- Dashboards and alerts are generated from the service catalogue and stored as code.
- Logs are structured and carry trace IDs; redaction is automatic.
- Trace sampling keeps all errors and slow requests.
- Observability cost is attributed per service and reviewed.
Last updated 19 Aug 2026, 00:00 UTC.