technical
Distributed Tracing in Go
OpenTelemetry, Jaeger, and context propagation reveal where requests travel and stall across distributed Go services.
The first time I had to find a slow request across several services, I had logs from each of them and no way to tell which lines belonged to the same request. What I wanted was Ariadne’s thread: something tied to the request at the door that stays attached through every turn it takes. That is what a trace is.
Why Logs Alone Fall Short
Logs are great inside a single process. In a distributed system they leave gaps:
- Each service logs on its own. A request ID in a header plus structured logs gets you a long way, but you still reassemble the story by hand, one grep per service.
- Logs carry timestamps, not a tree. They cannot show that service B spent most of the request waiting on a cache lookup in service C.
- Background jobs, retries and goroutines drop the ID unless you are careful. Tracing has the same problem, as I show below, but it gives you a standard way to carry the ID across.
Tracing is the request ID idea taken further: the same ID, plus parent and child relationships and durations, in a wire format that every HTTP, gRPC and database library already speaks. Keep the logs. Put the trace ID in every log line, with trace.SpanContextFromContext(ctx).TraceID() in your slog handler, and you can jump from a span in Jaeger to the matching log lines and back.
How Tracing Works
Three concepts carry the whole idea:
- Trace ID: one identifier for the entire request journey. Every span in the journey shares it.
- Span: a timed unit of work, such as a database query or an HTTP call to the payment service. Each span knows its start and end time, its parent span (except the root), and optional attributes such as the HTTP status or the SQL statement.
- Propagation: passing the trace context (trace ID, current span ID and a sampling flag) to the next hop so it can attach its own spans to the same trace. Inside a process that context travels in
context.Context. Across processes it travels in the W3Ctraceparentheader.
Together they form a tree. The root span is the entry point, usually an incoming HTTP request. Child spans are the work beneath it, both inside the same service and in the services it calls.
graph TD
subgraph Trace[Trace ID: 1234abcd]
A[Root Span: HTTP Request in Service A] --> C[Span: HTTP Call to Service B]
C --> D[Span: Handle Request in Service B]
D --> E[Span: DB Query in Service B]
A --> F[Span: HTTP Call to Service C]
F --> G[Span: Handle Request in Service C]
G --> H[Span: Cache Lookup in Service C]
end
Reading it top down: Service A handles the request, fans out to B and C, and each of those records its own spans under the same trace ID. A slow query in B shows up as a long bar exactly where it sits in the tree.
The Role of context.Context
Go already uses context.Context to carry deadlines and cancellation through a call chain. Tracing libraries hang the current span on that same context.
When I start a span, the tracer returns a new context with the span inside. Every function that receives that context can start child spans under it. The first thing that bit me was passing context.Background() into a helper and watching its span show up in Jaeger as a separate trace. Wrapping the context with a timeout is fine, because the span travels along as a value. Starting from a fresh context breaks the thread.
func handler(w http.ResponseWriter, r *http.Request) {
ctx, span := tracer.Start(r.Context(), "handler")
defer span.End()
processRequest(ctx)
}Here processRequest receives the same context and can attach its own spans to the trace.
A Minimal Setup With OpenTelemetry and Jaeger
OpenTelemetry (OTel) is the standard for instrumentation. The backend is a separate choice. I use Jaeger here, but the same code ships traces to Grafana Tempo, Datadog, Honeycomb, New Relic, or anything else that speaks OTLP, the OpenTelemetry protocol. Change the exporter settings and keep the instrumentation.
Start Jaeger first. Jaeger ingests OTLP natively, on port 4318 for HTTP, and serves its UI on 16686:
docker run -d \
-p 16686:16686 \
-p 4318:4318 \
cr.jaegertracing.io/jaegertracing/jaeger:2.22.0Then the service. I run the same binary twice, so the trace actually crosses a network hop:
package main
import (
"context"
"errors"
"fmt"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.43.0"
)
// The tracer is named after the package that emits the spans, not the
// service. The service name lives on the resource below.
var tracer = otel.Tracer("github.com/victorpierredev/tracing-demo")
// setupTracing wires an OTLP/HTTP exporter into a tracer provider and
// registers the W3C propagator so trace headers cross service boundaries.
// The exporter reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment.
func setupTracing(ctx context.Context, serviceName string) (*sdktrace.TracerProvider, error) {
exp, err := otlptracehttp.New(ctx)
if err != nil {
return nil, err
}
res, err := resource.Merge(resource.Default(),
resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceName(serviceName)))
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithResource(res),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, propagation.Baggage{}))
return tp, nil
}
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
serviceName := os.Getenv("SERVICE_NAME")
downstream := os.Getenv("DOWNSTREAM")
tp, err := setupTracing(ctx, serviceName)
if err != nil {
log.Fatalf("setup tracing: %v", err)
}
// The transport injects the traceparent header into outgoing requests.
client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
ctx, span := tracer.Start(r.Context(), "handle")
defer span.End()
if err := doWork(ctx); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
if downstream != "" {
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, downstream, nil)
resp, err := client.Do(req)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
http.Error(w, err.Error(), http.StatusBadGateway)
return
}
resp.Body.Close()
}
fmt.Fprintln(w, "done")
})
// The handler reads the traceparent header from incoming requests.
srv := &http.Server{
Addr: ":" + os.Getenv("PORT"),
Handler: otelhttp.NewHandler(mux, "server"),
ReadHeaderTimeout: 5 * time.Second,
}
go func() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Printf("server: %v", err)
stop()
}
}()
<-ctx.Done()
// Give the server and the exporter a bounded time to drain.
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
log.Printf("server shutdown: %v", err)
}
if err := tp.Shutdown(shutdownCtx); err != nil {
log.Printf("tracer shutdown: %v", err)
}
}
func doWork(ctx context.Context) error {
_, span := tracer.Start(ctx, "doWork")
defer span.End()
time.Sleep(20 * time.Millisecond)
return nil
}Run two copies, one calling the other, and send a request to the first:
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
SERVICE_NAME=api PORT=8080 DOWNSTREAM=http://localhost:8081 go run . &
SERVICE_NAME=orders PORT=8081 go run . &
curl localhost:8080A few choices in that code took me a while to get right:
- The exporter has no options. The endpoint comes from
OTEL_EXPORTER_OTLP_ENDPOINT, and thehttp://scheme is what makes it plaintext. If you hard-code the endpoint in Go, it silently wins over the environment variable, which is how a service ends up sending traces to localhost in production. - The propagator has to be registered. The global default is a no-op. Without
otel.SetTextMapPropagator,otelhttpinjects nothing and reads nothing, and every service starts its own trace. otelhttpdoes the header work.NewHandlerreadstraceparentfrom incoming requests andNewTransportwrites it on outgoing ones. The request has to be built withhttp.NewRequestWithContextso the transport can find the span.- Shutdown is signal driven.
log.Fatalcallsos.Exit, which skips deferred functions. The batch processor exports every five seconds, so a demo that exits throughlog.Fatalcan leave Jaeger empty. Waiting on the signal context and callingtp.Shutdownwith a timeout flushes what is left. - Errors are recorded on the span. A span’s status is unset by default. Jaeger only marks a span as failed when
SetStatus(codes.Error, ...)is called, so without it a failed call looks like a fast success.
In Jaeger, search for the api service. The trace shows both services under one trace ID, with the orders spans nested under the HTTP call that reached them. Duration bars make the slow part obvious, and clicking a span shows its attributes and any recorded error.
Common Issues
The spans I lost in practice were lost for one of these reasons:
The span was never ended. Spans are exported when
Endis called, so a missingdefer span.End()means a missing span. Cancelling the context does not end a span, I had that wrong for a while. It only fails the work inside the span.The process exited before the flush. The batch processor exports every five seconds.
log.Fatalandos.Exitskip deferred functions. Shut down the tracer provider on the way out.The exporter could not reach the collector. Those failures go to OTel’s error handler, which prints to stderr by default. Wire
otel.SetErrorHandlerto your logger so you see them.A goroutine outlived the request. Passing
ctxkeeps the goroutine in the trace, but the request context is cancelled when the handler returns, and the goroutine’s database and HTTP calls fail withcontext canceled. Usecontext.WithoutCancel, which keeps the span and drops the cancellation:go func(ctx context.Context) { _, span := tracer.Start(ctx, "send-email") defer span.End() // ... }(context.WithoutCancel(ctx))A job was enqueued without the trace context. The worker starts a fresh trace. The fix is to inject the trace context into the payload with the propagator, and extract it on the other side:
// Producer: put the trace context into the job. carrier := propagation.MapCarrier{} otel.GetTextMapPropagator().Inject(ctx, carrier) job := Job{OrderID: orderID, Trace: carrier} // Worker: restore it and link the job span to the request span. parent := otel.GetTextMapPropagator().Extract(context.Background(), propagation.MapCarrier(job.Trace)) _, span := tracer.Start(context.Background(), "process-job", trace.WithLinks(trace.Link{SpanContext: trace.SpanContextFromContext(parent)})) defer span.End()Start the job span from
parentinstead ofcontext.Background()if you want it as a child in the same trace. I use a link when the job runs much later or gets retried, because a child span that starts an hour after its parent makes the whole trace look an hour long.
Before Production
The default sampler keeps every trace. That is fine for a demo and expensive at a few thousand requests per second. One line changes it:
sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)))ParentBased means the first service decides for the whole trace and the others follow. Keep span names low cardinality too: GET /users/{id}, never GET /users/123. Backends index and aggregate by span name, and the ID belongs in an attribute.
Every one of those failures is the same mistake: letting go of the thread. Ariadne’s trick only works if Theseus never drops it, and a trace only works if the context reaches every function, goroutine and queue the request passes through. Once that habit stuck, Jaeger became the first place I look when something is slow.