Logging, monitoring and observability

There is no such thing as a fully observable system. Nobody logs every event, tracks every metric, or traces every code path — the cost of doing that exceeds the cost of the outage you're trying to prevent. Logging, monitoring and observability are practices you implement on a spectrum, and the useful question is never "are we observable?" but "when this pages me at 3am, how many clicks until I know which line failed?"

That framing matters because the tooling around this topic is loud. Prometheus, Grafana, Loki, Promtail, Jaeger, OpenTelemetry, New Relic, Datadog — the list is long enough to be intimidating, and none of it means anything until you know what the three underlying things are.

The three words

Logging is recording events with metadata. A user logged in. A database query ran. Something threw. Each record carries the context you'll need later: the user ID, the request ID, the latency, the function that fired. It's a journal your application keeps so that when something breaks you can read back what happened and when.

Monitoring is continuously checking the state of the system. CPU, memory, requests per second, open database connections in the pool. "Real time" here is a polite lie — most setups run 10 to 15 seconds behind, because shipping a data point every millisecond would cost more to collect than the system being watched. That delay is a deliberate choice, not a limitation.

Observability is the property that you can determine the internal state of a system from its external outputs. It's a property, not a tool. A system has it or doesn't, and it has it only when three things are in place:

• Logs — what happened
• Metrics — patterns and trends over time
• Traces — how the components interacted

These are the three pillars. Drop one and the property goes away.

Why monitoring alone stopped being enough

A decade ago, monitoring was the whole discipline. You set thresholds, wired alerts, and waited for a dashboard to go red. And it worked, in the narrow sense that it told you something was wrong.

That's the ceiling. Monitoring informs you there's a problem. It cannot tell you what the problem is. You get an alert saying the error rate crossed 80%, and then you're on your own — SSH into boxes, grep logs, guess.

Observability closes that loop. It tells you something is wrong and what is wrong, provided you did the instrumentation work up front. The whole movement exists because backend applications stopped being one process on one server. They're distributed across services, regions and machines, and a request that fails somewhere in that mesh leaves no single place to look.

A trace is a transaction

Of the three pillars, traces are the one people find least intuitive, so: a trace is a transaction that spans components.

A request originates somewhere — a frontend, a load balancer, your handler. From there it touches a validation layer, a service layer, a repository layer, the database. A trace records that whole journey as one linked object. Not five unrelated log lines in five services, but one thing you can open and read top to bottom, seeing where the time went and where it died.

Mechanically it usually works like this: middleware at the edge of your app creates a transaction, stamps it with the service name, environment, IP, user agent, request ID, user ID, tenant ID, and stuffs it into the request context. Every layer downstream pulls the transaction back out of that context, opens a segment for its own work, adds attributes, and ends the segment when it returns.

txn := newrelic.FromContext(ctx)
defer txn.StartSegment("todo.create.service").End()
txn.AddAttribute("user_id", userID)
txn.AddAttribute("todo_title", title)

That's four lines and it's the entire idea. The defer closes the segment on any return path, including the error ones — which are the paths you care about.

Two words you'll meet constantly here. Instrumentation is the practice of measuring attributes of your functions — the act of adding those lines. OpenTelemetry is the open standard for doing it: SDKs, APIs and collectors for every major language, so your instrumentation isn't welded to one vendor. Even on a proprietary backend, running an OTel collector keeps the option open.

Log levels, and where each one belongs

Five levels cover almost everything:

• debug — troubleshooting detail. On in development, off in production, because the volume is overwhelming.
• info — successful operations and business events. A todo was created.
• warn — not success, not your fault. A user typed the wrong password. Worth recording, not worth an alert.
• error — validation failures, failed queries, anything thrown. The reason logging exists.
• fatal — the process is going down. Your infrastructure will restart it.

The level filter is one function, decided by environment:

func getLogLevel(env string) zerolog.Level {
    if env == "production" {
        return zerolog.InfoLevel
    }
    return zerolog.DebugLevel
}

Structured vs unstructured is an environment switch

In development you want colour, alignment, plain English — logs a human reads at a glance. In production you want JSON, because nothing human is reading them. They go into Loki, or the ELK stack, or a vendor's ingest pipeline, and those tools have to pull user_id and request_id out of every line. Regex over prose is slow and it breaks. JSON parses.

Same logger, same call sites, one config value. Flip it and the same log line goes from readable to parseable.

The number of production incidents I've debugged with grep on plain-text logs is exactly the number of times I've wished the format flag had been set correctly on day one.

The workflow the pillars are for

Here's the payoff, end to end. Error rate crosses your threshold. A webhook drops a message in Slack: something is wrong with the API service.

You open the dashboard and look at metrics — throughput, average transaction time, error rate. The numbers confirm it: errors are at 80%.

From that metric you click through to the logs behind it. Not all logs, the ones attached to those failures. You find a 500.

From that log you click into its trace. The request started in middleware, went to the handler, hit the service layer, and died at the database call. You have the user ID, the request ID, the path, the exact segment.

Three clicks, from "something is wrong" to "this line, this request, this user". That's the whole return on the instrumentation.

Tools, briefly

The open-source route is the Grafana stack: Prometheus builds the metrics, Loki and Promtail handle logs, Jaeger handles traces, Grafana is the dashboard on top. Full control, four systems to operate.

The proprietary route is New Relic or Datadog — one agent, one dashboard, metrics-to-logs-to-traces already wired together. You pay for it, and if your team doesn't have someone whose job is running observability infrastructure, that's a reasonable trade.

Either way, none of it works if the code doesn't emit anything. This is a collective effort: the DevOps side sets up collection and storage, but the transaction attributes, the log levels, the spans in the service layer — those are written by whoever writes the handler. No dashboard invents context that was never recorded.