technical

Distributed Tracing in Go

OpenTelemetry, Jaeger, and context propagation reveal where requests travel and stall across distributed Go services.

The first time I had to find a slow request across several services, I had logs from each of them and no way to tell which lines belonged to the same request. What I wanted was Ariadne’s thread: something tied to the request at the door that stays attached through every turn it takes. That is what a trace is.

Why Logs Alone Fall Short

Logs are great inside a single process. In a distributed system they leave gaps:

  • Each service logs on its own. A request ID in a header plus structured logs gets you a long way, but you still reassemble the story by hand, one grep per service.
  • Logs carry timestamps, not a tree. They cannot show that service B spent most of the request waiting on a cache lookup in service C.
  • Background jobs, retries and goroutines drop the ID unless you are careful. Tracing has the same problem, as I show below, but it gives you a standard way to carry the ID across.

Tracing is the request ID idea taken further: the same ID, plus parent and child relationships and durations, in a wire format that every HTTP, gRPC and database library already speaks. Keep the logs. Put the trace ID in every log line, with trace.SpanContextFromContext(ctx).TraceID() in your slog handler, and you can jump from a span in Jaeger to the matching log lines and back.

How Tracing Works

Three concepts carry the whole idea:

  • Trace ID: one identifier for the entire request journey. Every span in the journey shares it.
  • Span: a timed unit of work, such as a database query or an HTTP call to the payment service. Each span knows its start and end time, its parent span (except the root), and optional attributes such as the HTTP status or the SQL statement.
  • Propagation: passing the trace context (trace ID, current span ID and a sampling flag) to the next hop so it can attach its own spans to the same trace. Inside a process that context travels in context.Context. Across processes it travels in the W3C traceparent header.

Together they form a tree. The root span is the entry point, usually an incoming HTTP request. Child spans are the work beneath it, both inside the same service and in the services it calls.

  graph TD
    subgraph Trace[Trace ID: 1234abcd]
        A[Root Span: HTTP Request in Service A] --> C[Span: HTTP Call to Service B]
        C --> D[Span: Handle Request in Service B]
        D --> E[Span: DB Query in Service B]

        A --> F[Span: HTTP Call to Service C]
        F --> G[Span: Handle Request in Service C]
        G --> H[Span: Cache Lookup in Service C]
    end

Reading it top down: Service A handles the request, fans out to B and C, and each of those records its own spans under the same trace ID. A slow query in B shows up as a long bar exactly where it sits in the tree.

The Role of context.Context

Go already uses context.Context to carry deadlines and cancellation through a call chain. Tracing libraries hang the current span on that same context.

When I start a span, the tracer returns a new context with the span inside. Every function that receives that context can start child spans under it. The first thing that bit me was passing context.Background() into a helper and watching its span show up in Jaeger as a separate trace. Wrapping the context with a timeout is fine, because the span travels along as a value. Starting from a fresh context breaks the thread.

func handler(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "handler")
    defer span.End()

    processRequest(ctx)
}

Here processRequest receives the same context and can attach its own spans to the trace.

A Minimal Setup With OpenTelemetry and Jaeger

OpenTelemetry (OTel) is the standard for instrumentation. The backend is a separate choice. I use Jaeger here, but the same code ships traces to Grafana Tempo, Datadog, Honeycomb, New Relic, or anything else that speaks OTLP, the OpenTelemetry protocol. Change the exporter settings and keep the instrumentation.

Start Jaeger first. Jaeger ingests OTLP natively, on port 4318 for HTTP, and serves its UI on 16686:

docker run -d \
  -p 16686:16686 \
  -p 4318:4318 \
  cr.jaegertracing.io/jaegertracing/jaeger:2.22.0

Then the service. I run the same binary twice, so the trace actually crosses a network hop:

package main

import (
    "context"
    "errors"
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.43.0"
)

// The tracer is named after the package that emits the spans, not the
// service. The service name lives on the resource below.
var tracer = otel.Tracer("github.com/victorpierredev/tracing-demo")

// setupTracing wires an OTLP/HTTP exporter into a tracer provider and
// registers the W3C propagator so trace headers cross service boundaries.
// The exporter reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment.
func setupTracing(ctx context.Context, serviceName string) (*sdktrace.TracerProvider, error) {
    exp, err := otlptracehttp.New(ctx)
    if err != nil {
        return nil, err
    }

    res, err := resource.Merge(resource.Default(),
        resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceName(serviceName)))
    if err != nil {
        return nil, err
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{}, propagation.Baggage{}))
    return tp, nil
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer stop()

    serviceName := os.Getenv("SERVICE_NAME")
    downstream := os.Getenv("DOWNSTREAM")

    tp, err := setupTracing(ctx, serviceName)
    if err != nil {
        log.Fatalf("setup tracing: %v", err)
    }

    // The transport injects the traceparent header into outgoing requests.
    client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

    mux := http.NewServeMux()
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        ctx, span := tracer.Start(r.Context(), "handle")
        defer span.End()

        if err := doWork(ctx); err != nil {
            span.RecordError(err)
            span.SetStatus(codes.Error, err.Error())
            http.Error(w, err.Error(), http.StatusInternalServerError)
            return
        }

        if downstream != "" {
            req, _ := http.NewRequestWithContext(ctx, http.MethodGet, downstream, nil)
            resp, err := client.Do(req)
            if err != nil {
                span.RecordError(err)
                span.SetStatus(codes.Error, err.Error())
                http.Error(w, err.Error(), http.StatusBadGateway)
                return
            }
            resp.Body.Close()
        }
        fmt.Fprintln(w, "done")
    })

    // The handler reads the traceparent header from incoming requests.
    srv := &http.Server{
        Addr:              ":" + os.Getenv("PORT"),
        Handler:           otelhttp.NewHandler(mux, "server"),
        ReadHeaderTimeout: 5 * time.Second,
    }
    go func() {
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            log.Printf("server: %v", err)
            stop()
        }
    }()

    <-ctx.Done()

    // Give the server and the exporter a bounded time to drain.
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("server shutdown: %v", err)
    }
    if err := tp.Shutdown(shutdownCtx); err != nil {
        log.Printf("tracer shutdown: %v", err)
    }
}

func doWork(ctx context.Context) error {
    _, span := tracer.Start(ctx, "doWork")
    defer span.End()

    time.Sleep(20 * time.Millisecond)
    return nil
}

Run two copies, one calling the other, and send a request to the first:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
SERVICE_NAME=api PORT=8080 DOWNSTREAM=http://localhost:8081 go run . &
SERVICE_NAME=orders PORT=8081 go run . &
curl localhost:8080

A few choices in that code took me a while to get right:

  • The exporter has no options. The endpoint comes from OTEL_EXPORTER_OTLP_ENDPOINT, and the http:// scheme is what makes it plaintext. If you hard-code the endpoint in Go, it silently wins over the environment variable, which is how a service ends up sending traces to localhost in production.
  • The propagator has to be registered. The global default is a no-op. Without otel.SetTextMapPropagator, otelhttp injects nothing and reads nothing, and every service starts its own trace.
  • otelhttp does the header work. NewHandler reads traceparent from incoming requests and NewTransport writes it on outgoing ones. The request has to be built with http.NewRequestWithContext so the transport can find the span.
  • Shutdown is signal driven. log.Fatal calls os.Exit, which skips deferred functions. The batch processor exports every five seconds, so a demo that exits through log.Fatal can leave Jaeger empty. Waiting on the signal context and calling tp.Shutdown with a timeout flushes what is left.
  • Errors are recorded on the span. A span’s status is unset by default. Jaeger only marks a span as failed when SetStatus(codes.Error, ...) is called, so without it a failed call looks like a fast success.

In Jaeger, search for the api service. The trace shows both services under one trace ID, with the orders spans nested under the HTTP call that reached them. Duration bars make the slow part obvious, and clicking a span shows its attributes and any recorded error.

Common Issues

The spans I lost in practice were lost for one of these reasons:

  • The span was never ended. Spans are exported when End is called, so a missing defer span.End() means a missing span. Cancelling the context does not end a span, I had that wrong for a while. It only fails the work inside the span.

  • The process exited before the flush. The batch processor exports every five seconds. log.Fatal and os.Exit skip deferred functions. Shut down the tracer provider on the way out.

  • The exporter could not reach the collector. Those failures go to OTel’s error handler, which prints to stderr by default. Wire otel.SetErrorHandler to your logger so you see them.

  • A goroutine outlived the request. Passing ctx keeps the goroutine in the trace, but the request context is cancelled when the handler returns, and the goroutine’s database and HTTP calls fail with context canceled. Use context.WithoutCancel, which keeps the span and drops the cancellation:

    go func(ctx context.Context) {
        _, span := tracer.Start(ctx, "send-email")
        defer span.End()
        // ...
    }(context.WithoutCancel(ctx))
  • A job was enqueued without the trace context. The worker starts a fresh trace. The fix is to inject the trace context into the payload with the propagator, and extract it on the other side:

    // Producer: put the trace context into the job.
    carrier := propagation.MapCarrier{}
    otel.GetTextMapPropagator().Inject(ctx, carrier)
    job := Job{OrderID: orderID, Trace: carrier}
    
    // Worker: restore it and link the job span to the request span.
    parent := otel.GetTextMapPropagator().Extract(context.Background(),
        propagation.MapCarrier(job.Trace))
    _, span := tracer.Start(context.Background(), "process-job",
        trace.WithLinks(trace.Link{SpanContext: trace.SpanContextFromContext(parent)}))
    defer span.End()

    Start the job span from parent instead of context.Background() if you want it as a child in the same trace. I use a link when the job runs much later or gets retried, because a child span that starts an hour after its parent makes the whole trace look an hour long.

Before Production

The default sampler keeps every trace. That is fine for a demo and expensive at a few thousand requests per second. One line changes it:

sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)))

ParentBased means the first service decides for the whole trace and the others follow. Keep span names low cardinality too: GET /users/{id}, never GET /users/123. Backends index and aggregate by span name, and the ID belongs in an attribute.

Every one of those failures is the same mistake: letting go of the thread. Ariadne’s trick only works if Theseus never drops it, and a trace only works if the context reaches every function, goroutine and queue the request passes through. Once that habit stuck, Jaeger became the first place I look when something is slow.