OpenTelemetry with Jaeger

Distributed tracing answers a question metrics cannot: for this specific slow request, where did the time go? Dashboards tell you the p99 doubled. A trace tells you it doubled because the auth service started making three sequential calls where it used to make one.

OpenTelemetry is how you produce traces. Jaeger is where you look at them.

Three moving parts

Instrumentation in your application produces spans. Either auto-instrumentation, which patches known libraries, or manual spans you add.

The Collector receives spans, processes them (batching, sampling, enrichment) and exports them onward. It is optional in theory and essential in practice.

Jaeger stores traces and provides the UI. Modern Jaeger accepts OTLP directly, so you no longer need Jaeger-specific client libraries or agents. Jaeger v2 is itself built on the OpenTelemetry Collector.

The thing to internalise: OpenTelemetry is a vendor-neutral wire format and SDK. Instrument once, change backends by editing Collector config. That property is the main reason to adopt it.

Auto-instrumentation on Kubernetes

Install the OpenTelemetry Operator, then declare an Instrumentation resource:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
  namespace: observability
spec:
  exporter:
    endpoint: http://otel-collector.observability:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"

Then annotate a workload:

spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "observability/default"

The operator injects an init container carrying the agent and sets the environment variables. The application is not rebuilt and its code is not modified. Java, .NET, Node.js, Python, and Go are supported, though Go requires eBPF-based instrumentation and is meaningfully more constrained than the others.

Set the sampler to 1.0 here and sample later in the Collector. Sampling at the SDK means the decision is made before anyone knows whether the request was interesting.

Why the Collector is not optional

You can export straight from the SDK to Jaeger. You will regret it.

The Collector gives you a place to change sampling, add attributes, redact fields, and switch backends without redeploying a single application. Without it, every one of those is a config change across every service.

A working config:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes:
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.pod.name
        - k8s.node.name
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  otlp/jaeger:
    endpoint: jaeger-collector.observability:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlp/jaeger]

Order matters in that processor list. memory_limiter goes first so it can shed load before anything else allocates. batch goes last so it batches the final form of the data. Getting this backwards is a common cause of Collector OOMs under load.

k8sattributes is the one that turns traces from useful into navigable. Every span gets tagged with its namespace, deployment, and node, so you can ask “show me slow spans from the checkout deployment in prod” instead of scrolling.

Tail sampling: the thing that makes this affordable

Head sampling (deciding at the start of a request) is cheap and keeps the wrong traces. It cannot know the request will fail, because it decides before that happens.

Tail sampling buffers complete traces and decides after seeing everything:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 1

Keep every error, every request over a second, and 1% of everything else for baseline comparison. Storage drops by an order of magnitude and you keep the traces you would actually open.

One constraint that catches people: tail sampling requires all spans of a trace to reach the same Collector instance. With multiple replicas you need a two-tier setup. A first layer that routes by trace ID using the loadbalancing exporter, and a second layer that does the sampling. Skip this and you will silently sample fragments of traces, which is worse than no sampling because the gaps look like missing instrumentation.

Deploying Jaeger

For anything beyond a demo, Jaeger needs real storage: Elasticsearch, OpenSearch, or Cassandra. The all-in-one image keeps traces in memory and loses them on restart; it is for local development only.

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://elasticsearch.observability:9200
        index-prefix: jaeger
    esIndexCleaner:
      enabled: true
      numberOfDays: 14
      schedule: "0 2 * * *"

Turn on the index cleaner from the start. Trace data grows fast, and nobody has ever needed a trace from four months ago.

Context propagation is where it breaks

Everything above works and you still get disconnected single-span traces. The cause is almost always propagation.

For a trace to span services, the traceparent header must survive every hop. It breaks at:

Message queues. HTTP propagation is automatic; queues are not. You must inject context into message headers on publish and extract on consume, by hand.
Custom HTTP clients. Auto-instrumentation patches known libraries. A hand-rolled client, or one wrapped oddly, is not patched.
Async work. Spawning a goroutine or a thread pool task without passing context loses the parent.
Gateways and proxies. Most preserve unknown headers, but a strict allow-list will strip traceparent. Check this early. It is invisible and produces exactly the symptom above.

Standardise on W3C Trace Context (tracecontext) unless you have legacy B3 propagation to support, in which case configure both during migration.

Add spans where the questions are

Auto-instrumentation gives you HTTP handlers and database calls. That is a good skeleton and it is not enough. It shows time inside POST /checkout without showing which part.

Add manual spans around the parts you would want broken out at 3am:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("fraud_check") as span:
    span.set_attribute("payment.provider", provider)
    span.set_attribute("order.value_cents", value)
    result = fraud_service.check(order)
    span.set_attribute("fraud.decision", result.decision)

Attributes are what make traces searchable. payment.provider as an attribute lets you ask whether one provider is slower than another: a question you cannot answer from span names alone.

Keep cardinality in mind: attributes on spans are far cheaper than labels on metrics, so a user ID is acceptable here in a way it never is on a Prometheus counter.

Connecting the three signals

The real payoff is moving between signals. Put the trace ID in your logs:

span = trace.get_current_span()
trace_id = format(span.get_span_context().trace_id, "032x")
logger.info("payment failed", extra={"trace_id": trace_id})

Now an error log links to the trace showing exactly what that request did. With exemplars, Prometheus can attach trace IDs to latency histogram buckets, so you can click a p99 spike on a Grafana panel and land on a trace from that bucket.

That path (dashboard spike to specific slow request in two clicks) is what tracing is for. Everything else is setup.