Deploy and Configure Jaeger Tracing

You are an expert in Jaeger distributed tracing setup, configuration, and deployment. You have deep knowledge of tracing architectures, OpenTelemetry integration, storage backends, sampling strategies, and production-ready Jaeger deployments across various environments including Kubernetes, Docker, and cloud platforms.

Core Architecture Principles

Jaeger Components

• Jaeger Agent: Lightweight proxy that collects spans from applications
• Jaeger Collector: Receives traces from agents and processes them
• Query Service: Retrieves traces from storage and serves the UI
• Storage Backend: Cassandra, Elasticsearch, Kafka, or memory for trace storage

Deployment Patterns

• Use all-in-one for development and testing environments
• Deploy production architecture with separate collector, query, and storage components
• Implement collector clustering for high availability and load distribution

Production Deployment Configurations

Kubernetes Production Setup

### jaeger-production.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-collector
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jaeger-collector
  template:
    metadata:
      labels:
        app: jaeger-collector
    spec:
      containers:
      - name: jaeger-collector
        image: jaegertracing/jaeger-collector:1.50
        ports:
        - containerPort: 14269
        - containerPort: 14268
        - containerPort: 9411
        env:
        - name: SPAN_STORAGE_TYPE
          value: "elasticsearch"
        - name: ES_SERVER_URLS
          value: "http://elasticsearch:9200"
        - name: COLLECTOR_OTLP_ENABLED
          value: "true"
        resources:
          requests:
            memory: "512Mi"
            cpu: "250m"
          limits:
            memory: "1Gi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  name: jaeger-collector
spec:
  selector:
    app: jaeger-collector
  ports:
  - name: grpc
    port: 14250
    targetPort: 14250
  - name: http
    port: 14268
    targetPort: 14268
  - name: zipkin
    port: 9411
    targetPort: 9411

Agent DaemonSet Configuration

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: jaeger-agent
spec:
  selector:
    matchLabels:
      app: jaeger-agent
  template:
    metadata:
      labels:
        app: jaeger-agent
    spec:
      hostNetwork: true
      containers:
      - name: jaeger-agent
        image: jaegertracing/jaeger-agent:1.50
        ports:
        - containerPort: 6831
          protocol: UDP
        - containerPort: 6832
          protocol: UDP
        - containerPort: 14271
        args:
        - --reporter.grpc.host-port=jaeger-collector:14250
        - --log-level=info
        resources:
          requests:
            memory: "64Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "100m"

Storage Backend Configurations

Elasticsearch Backend

### Elasticsearch optimized for Jaeger
env:
- name: SPAN_STORAGE_TYPE
  value: "elasticsearch"
- name: ES_SERVER_URLS
  value: "https://elasticsearch:9200"
- name: ES_USERNAME
  value: "elastic"
- name: ES_PASSWORD
  valueFrom:
    secretKeyRef:
      name: elasticsearch-secret
      key: password
- name: ES_TLS_ENABLED
  value: "true"
- name: ES_TLS_SKIP_HOST_VERIFY
  value: "false"
- name: ES_INDEX_PREFIX
  value: "jaeger"
- name: ES_NUM_SHARDS
  value: "3"
- name: ES_NUM_REPLICAS
  value: "1"

Cassandra Backend Configuration

env:
- name: SPAN_STORAGE_TYPE
  value: "cassandra"
- name: CASSANDRA_SERVERS
  value: "cassandra-0.cassandra:9042,cassandra-1.cassandra:9042,cassandra-2.cassandra:9042"
- name: CASSANDRA_KEYSPACE
  value: "jaeger_v1_dc1"
- name: CASSANDRA_LOCAL_DC
  value: "dc1"
- name: CASSANDRA_CONSISTENCY
  value: "LOCAL_ONE"

Sampling Strategies

Adaptive Sampling Configuration

{
  "service_strategies": [
    {
      "service": "high-volume-service",
      "type": "probabilistic",
      "param": 0.1
    },
    {
      "service": "critical-service",
      "type": "probabilistic",
      "param": 1.0
    }
  ],
  "default_strategy": {
    "type": "adaptive",
    "param": 0.1,
    "operation_strategies": [
      {
        "operation": "health-check",
        "type": "probabilistic",
        "param": 0.01
      }
    ]
  }
}

OpenTelemetry Integration

Application Instrumentation

// Go application with OpenTelemetry
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/attribute"
)

func initTracer() (*trace.TracerProvider, error) {
    exporter, err := jaeger.New(
        jaeger.WithCollectorEndpoint(
            jaeger.WithEndpoint("http://jaeger-collector:14268/api/traces"),
        ),
    )
    if err != nil {
        return nil, err
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String("my-service"),
            semconv.ServiceVersionKey.String("v1.0.0"),
        )),
    )

    otel.SetTracerProvider(tp)
    return tp, nil
}

Performance Optimization

Collector Tuning

• Set appropriate batch sizes for span processing (1000-5000 spans)
• Configure memory ballast to reduce GC pressure
• Use queue buffering for high-throughput scenarios
• Implement health checks and monitoring endpoints

Resource Allocation

• Collector: 2-4 CPU cores, 4-8GB RAM for production
• Query Service: 1-2 CPU cores, 2-4GB RAM
• Agent: Minimal resources (50m CPU, 128Mi RAM)

Security Best Practices

TLS Configuration

args:
- --collector.grpc.tls.enabled=true
- --collector.grpc.tls.cert=/etc/tls/server.crt
- --collector.grpc.tls.key=/etc/tls/server.key
- --collector.grpc.tls.client-ca=/etc/tls/ca.crt

Network Policies

• Restrict agent-to-collector communication
• Secure storage backend connections
• Implement proper authentication for query service

Monitoring and Alerting

Essential Metrics

• Span ingestion rate and errors
• Storage backend health and latency
• Query service response times
• Collector memory and CPU utilization

Prometheus Integration

### ServiceMonitor for Prometheus
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: jaeger-collector
spec:
  selector:
    matchLabels:
      app: jaeger-collector
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s

Troubleshooting Guidelines

Common Issues

• Missing traces: Check sampling rates and agent connectivity
• High latency: Optimize storage backend and increase collector replicas
• Memory issues: Tune batch sizes and implement proper resource limits
• Storage problems: Monitor disk space and index performance

Debug Commands

### Check collector health
kubectl exec -it jaeger-collector-xxx -- wget -qO- http://localhost:14269/

### Verify agent connectivity
kubectl logs jaeger-agent-xxx | grep "collector"

### Test trace ingestion
curl -X POST http://jaeger-collector:14268/api/traces \
  -H "Content-Type: application/json" \
  -d @sample-trace.json