Jaeger 面试题

传统架构（Agent + Collector 直连）：
  App → Jaeger Agent (UDP) → Collector → Storage → Query → UI

生产架构（Kafka 缓冲）：
  App → Jaeger Agent (UDP) → Collector → Kafka → Ingester → Storage → Query → UI

docker run -d --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/all-in-one:latest

Trace (全局调用链)
  │
  ├── Span (Root: HTTP GET /api)
  │     ├── Tags: http.method=GET, http.status_code=200
  │     ├── Logs: "db.query" {sql: "SELECT..."}
  │     │
  │     ├── ChildOf Reference
  │     │     └── Span (Child: SELECT FROM users)
  │     │
  │     └── FollowsFrom Reference
  │           └── Span (Async: send email)
  │
  └── Span (Root: 独立请求)

{
    "traceID": "0af7651916cd43dd8448eb211c80319c",
    "spanID": "b7ad6b7169203331",
    "operationName": "HTTP GET /api/users",
    "references": [
        {"refType": "CHILD_OF", "traceID": "...", "spanID": "..."}
    ],
    "startTime": 1716700000000000,
    "duration": 150000,
    "tags": [
        {"key": "http.method", "vType": "STRING", "vStr": "GET"},
        {"key": "http.status_code", "vType": "INT64", "vInt64": 200}
    ],
    "logs": [
        {"timestamp": 1716700000050000, "fields": [
            {"key": "event", "vStr": "error"},
            {"key": "message", "vStr": "connection refused"}
        ]}
    ],
    "process": {"serviceName": "user-service", "tags": []}
}

graph TD
    A["Application"]
    B["Jaeger Client SDK"]
    C["Jaeger Agent<br/>(本地守护进程)"]
    D["缓冲 + 批量"]
    E["Collector / Kafka"]
    A --> B -->|"UDP (thrift) → localhost:6831"| C --> D -->|"gRPC"| E

# jaeger-agent.yaml
reporter:
  collectorEndpoint: "http://jaeger-collector:14250"
  # 或使用 Kafka
  # kafka:
  #   bokerServers: kafka:9092

  # 批量配置
  batchSize: 1000
  batchWorkers: 100
  batchLinger: 1s

  # 最大队列大小
  queueSize: 10000

# 日志报告
logging:
  options: []

# 代理端口
process:
  # UDP 端口（接收客户端数据）
  zipkinCompactPort: 5775
  jaegerCompactPort: 6831
  jaegerBinaryPort: 6832

  # 管理端口
  adminPort: 14269

应用 → OTel SDK → OTel Collector → Jaeger (OTLP)

graph TD
    Entry[HTTP/gRPC 入口] --> Receiver[Collector 接收器]
    Receiver --> Processor[Span Processor - 验证/清洗]
    Processor --> Queue[内部队列 - 内存/磁盘]
    Queue --> Direct[直连写入 Storage]
    Queue --> Kafka[Kafka 写入 - 缓冲层]
    Kafka --> Ingester[Ingester]
    Ingester --> Storage2[Storage]

# jaeger-collector.yaml
collector:
  # gRPC 端口
  grpc:
    hostPort: ":14250"

  # HTTP 端口
  http:
    hostPort: ":14268"

  # 内部队列
  queue-size: 20000              # 队列最大大小
  num-workers: 100               # 处理 worker 数

  # Span 处理
  max-span-age: "0s"             # Span 最大年龄
  max-msg-size: "4mib"           # 消息大小限制

  # 直连存储（无 Kafka）
  storage:
    type: elasticsearch           # cassandra / elasticsearch / kafka

  # Kafka 写入
  kafka:
    producer:
      topic: jaeger-spans
      borkers: kafka:9092
      batch-size: 1000
      batch-linger: 1s

Kubernetes Deployment (无状态)：
  replicas: N

水平扩展依据：
  - Span 接收速率
  - 队列积压深度
  - 后端存储写入延迟

建议：
  每个 Collector 处理 10k-50k Spans/s
  通过 HPA 自动伸缩
  使用 PDB 保证最小可用

队列满 → 阻塞接收 → 客户端重试
         ↓
  客户端 SDK 内部的缓冲队列
         ↓
  队列满 → 客户端丢弃 Span

storage:
  type: elasticsearch
  elasticsearch:
    hostPorts: "es-1:9200,es-2:9200,es-3:9200"
    indexPrefix: "jaeger"
    indexDateSeparator: "-"
    username: "elastic"
    password: "password"
    # 索引配置
    numberOfShards: 5
    numberOfReplicas: 1
    # 批量写入
    bulk:
      size: 5000000       # 5MB
      workers: 10
      flushInterval: 1s
    # TTL
    esSpanTTL: "72h"
    esServiceCacheTTL: "12h"
    esDependenciesTTL: "72h"

storage:
  type: cassandra
  cassandra:
    hostPorts: "cassandra-1:9042,cassandra-2:9042"
    keyspace: "jaeger_v1"
    connectionsPerHost: 3
    maxRetryAttempts: 3
    timeout: 2s
    # 表配置
    spanTableCompaction:
      class: "LeveledCompactionStrategy"
    dependenciesTableCompaction:
      class: "LeveledCompactionStrategy"

storage:
  type: badger
  badger:
    directoryKey: "/data/badger/key"
    directoryValue: "/data/badger/data"
    ephemeral: false

graph TD
    A["Application"] -->|"HTTP"| JA["Jaeger Agent"]
    JA -->|"每 60 秒<br/>拉取采样策略"| JA2["Agent<br/>策略拉取"]
    JA2 -->|"gRPC"| C["Collector<br/>策略配置"]
    C --> S["采样策略<br/>默认概率: 0.1<br/>端点特定策略:<br/>/api/health: 0<br/>/api/order: 1<br/>/api/pay: 0.5"]

// 通过 Collector API 配置
POST /api/sampling
{
  "service": "user-service",
  "strategyType": "PROBABILISTIC",
  "probabilisticSampling": {
    "samplingRate": 0.1
  }
}

// 端点级采样策略
{
  "service": "order-service",
  "strategyType": "RATE_LIMITING",
  "rateLimitingSampling": {
    "maxTracesPerSecond": 50
  },
  "operationSampling": {
    "defaultSamplingProbability": 0.1,
    "perOperationSampling": [
      {
        "operation": "GET /api/health",
        "samplingProbability": 0
      },
      {
        "operation": "POST /api/orders",
        "samplingProbability": 1
      }
    ]
  }
}

import "github.com/jaegertracing/jaeger-client-go"

// Const 采样（全部采样）
sampler, _ := jaeger.NewConstSampler(true)

// 概率采样
sampler, _ := jaeger.NewProbabilisticSampler(0.1)

// 速率限制采样
sampler := jaeger.NewRateLimitingSampler(10) // 10 traces/s

// Remote 采样（推荐）
sampler, _ := jaeger.NewRemotelyControlledSampler(
    "user-service",
    jaeger.WithSamplingServerURL("http://jaeger-agent:5778/sampling"),
    jaeger.WithMaxOperations(1000),
    jaeger.WithSamplingRefreshInterval(time.Minute),
)

搜索条件：
  - 服务名 (必选)
  - 操作名 (可选)
  - Tags: http.status_code=200
  - 最短持续时间: 100ms
  - 时间范围
  - 限制返回条数

搜索结果列表：
  - Trace ID
  - 操作名
  - 耗时 (最短/最长/平均)
  - Span 数量
  - 错误状态
  - 时间

排序：By Most Recent / By Longest Duration

甘特图（Gantt Chart）：
  X 轴: 时间线
  Y 轴: Span 列表（按服务分组）
  宽度: Span 持续时间
  颜色: 按服务不同

每个 Span 详情：
  - Tags: 关键属性 (http、db、messaging)
  - Process: 服务名、主机
  - Logs: 时间戳事件列表
  - References: 父 Span 引用

功能：
  - 展开/折叠 Span 树
  - 点击 Tags 跳转到搜索
  - 对比多个 Trace
  - 下载 Trace JSON

选择 2-10 个 Trace 对比
  → 并列显示甘特图
  → 高亮差异部分
  → 统计汇总（平均耗时、慢 Span）

用途：
  对比正常请求和慢请求
  对比新旧版本请求
  对比不同路由的请求

自动生成服务调用拓扑
  节点: 服务 (颜色表示健康状态)
  边: 调用关系 (标注 QPS)
  布局: 力导向图

来源: Spark 作业或 Collector 实时计算

Header: traceparent: 00-traceid-spanid-01
  格式: 版本 - 32 位 traceid - 16 位 spanid - 标志

传播流程：
  服务A → HTTP Header → 服务B
          traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

import (
    "github.com/jaegertracing/jaeger-client-go/config"
    "github.com/opentracing/opentracing-go"
    "github.com/uber/jaeger-lib/metrics/prometheus"
)

cfg := config.Configuration{
    ServiceName: "user-service",
    Sampler: &config.SamplerConfig{
        Type:  jaeger.SamplerTypeRemote,
    },
    Reporter: &config.ReporterConfig{
        CollectorEndpoint: "http://jaeger-collector:14268/api/traces",
    },
    // Headers 配置
    Headers: &config.HeadersConfig{
        JaegerDebugHeader:       "jaeger-debug-id",
        JaegerBaggageHeader:     "jaeger-baggage",
        TraceContextHeaderName:  "traceparent", // W3C
        TraceBaggageHeaderPrefix: "tracestate",
    },
}

tracer, closer, _ := cfg.NewTracer(
    config.Metrics(prometheus.New()),
)

// 注入到出站请求
span := tracer.StartSpan("HTTP POST")
ext.SpanKindRPCClient.Set(span)
ext.HTTPMethod.Set(span, "POST")
tracer.Inject(span.Context(), opentracing.HTTPHeaders, r.Header)

// 从入站请求提取
spanCtx, _ := tracer.Extract(opentracing.HTTPHeaders, r.Header)
span := tracer.StartSpan("HTTP GET", ext.RPCServerOption(spanCtx))

方式一：Exemplar 关联
Jaeger UI → 点击示例 → 跳转到 Prometheus
Prometheus → 查看 Exemplar → 跳转到 Jaeger

方式二：Span 指标导出
Jaeger Collector → 计算 Metrics → Prometheus

方式三：统一仪表盘
Grafana → Jaeger + Prometheus 双数据源

# Jaeger Collector 配置
collector:
  # 启用 Prometheus Metrics
  metricsBackend: prometheus
  metricsHTTPPort: 14271

  # 导出内置指标
  # - jaeger_collector_spans_received_total
  # - jaeger_collector_spans_saved_total
  # - jaeger_collector_traces_received_total
  # - jaeger_query_requests_total
  # - jaeger_query_latency_seconds

Jaeger 数据源：搜索 Trace
Prometheus 数据源：查询 Metrics

典型场景：
  Prometheus 检测到高错误率
  → 查看 Exemplar 的 Trace ID
  → 在 Jaeger 中查看完整 Trace
  → 定位根因

# jaeger-query.yaml
prometheus:
  serverURL: "http://prometheus:9090"
  query:
    # 查询示例标签
    tags:
      - key: "service"
        value: "XQOPEN.ServiceNameXQCLOSE"

# UI 配置
ui:
  config:
    metrics:
      # 关联 Metrics 数据源
      prometheus:
        defaultDS: "Prometheus"
        queries:
          - name: "请求速率"
            query: "rate(request_duration_seconds_count{service='$service'}[5m])"
          - name: "错误率"
            query: "rate(request_duration_seconds_count{status_code=~'5..'}[5m])"

# jaeger-client
reporter:
  # Agent UDP 配置
  logSpans: false
  localAgentHostPort: "localhost:6831"
  flushInterval: 1000ms
  queueSize: 10000
  maxPacketSize: 65000

# jaeger-client 直连 Collector
reporter:
  collectorEndpoint: "http://jaeger-collector:14250"
  # gRPC 配置
  collectorUser: ""
  collectorPassword: ""
  httpHeaders:
    Authorization: "Bearer token"

旧架构：应用 Thrift UDP → Agent Thrift UDP → Collector
       (每部署一个 Agent，UDP 有丢失风险)

新架构：应用 gRPC → OTel Collector → Jaeger
       (可靠传输，统一 OTel 标准)

Span A (Service A: HTTP Client)
  ↓ ChildOf Reference (http.url=http://service-b/api)
Span B (Service B: HTTP Server)

提取依赖关系：
  Source: Service A
  Target: Service B
  Operation: GET /api
  Call Count: 100
  Error Rate: 1%
  Avg Duration: 50ms

# spark-dependencies.yaml
spark:
  type: "spark"
  dependencies:
    # Spark 作业参数
    sparkMaster: "spark://spark-master:7077"
    # 处理间隔
    scheduleInterval: "1h"
    # 时间窗口
    lookback: "1d"

GET /api/dependencies?endTs=1716700000&lookback=1h

响应：
{
    "data": [
        {
            "parent": "user-service",
            "child": "order-service",
            "callCount": 1000,
            "errorCount": 5
        },
        {
            "parent": "order-service",
            "child": "payment-service",
            "callCount": 800,
            "errorCount": 10
        }
    ]
}

力导向图布局：
  - 节点: Service（圆角矩形）
  - 边: 调用方向（箭头）
  - 节点颜色: 正常运行/错误/高延迟
  - 边数字: 调用量

交互：
  - 鼠标悬停 → 显示详细数据
  - 点击节点 → 跳转到服务 Trace 列表
  - 调整时间范围 → 动态更新

graph TD
    LB[Load Balancer - Nginx/ALB] --> Q1[Query-1]
    LB --> Q2[Query-2]
    LB --> Q3[Query-3 - 无状态]
    Q1 --> Storage[Storage - HA 存储]
    Q2 --> Storage
    Q3 --> Storage

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jaeger-query
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jaeger-query
  template:
    metadata:
      labels:
        app: jaeger-query
    spec:
      containers:
        - name: jaeger-query
          image: jaegertracing/jaeger-query:latest
          args:
            - "--storage.type=elasticsearch"
            - "--es.hostPorts=es-cluster:9200"
            - "--es.indexPrefix=jaeger"
          ports:
            - containerPort: 16686 # UI + API
          env:
            - name: SPAN_STORAGE_TYPE
              value: elasticsearch
          resources:
            limits:
              memory: 1Gi
              cpu: "1"
            requests:
              memory: 512Mi
              cpu: 500m
          readinessProbe:
            httpGet:
              path: /
              port: 16686
apiVersion: v1
kind: Service
metadata:
  name: jaeger-query
spec:
  ports:
    - port: 16686
      targetPort: 16686
  selector:
    app: jaeger-query
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jaeger-query-ingress
spec:
  rules:
    - host: jaeger.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: jaeger-query
                port:
                  number: 16686

# jaeger-query.yaml
ui:
  config:
    # 菜单链接
    menu:
      - label: "Grafana"
        url: "https://grafana.example.com"
        anchorTarget: "_blank"
    # Track 深度限制
    dependencies:
      menuEnabled: true
      menuLabel: "服务拓扑"
    # 链路追踪配置
    tracking:
      gaID: ""
      trackErrors: true
    # 搜索配置
    search:
      maxLookback:
        label: "2 天"
        value: "2d"
      maxDuration: 0

Collector → 分析各端点的流量
    │
  计算每个端点的最佳采样率
    │
  高流量端点 → 低采样率
  低流量端点 → 高采样率
  错误端点 → 强制 100%
    │
  下发给 Agent (每 60 秒)
    │
  Agent 缓存策略 → 客户端执行

目标: 每个端点每秒采集 N 条 Trace (如 token=10)

计算：
  traffic = 端点调用量 (traces/s)
  rate = min(1, token / traffic)

效果：
  1k/s 的端点: rate = min(1, 10/1000) = 1%
  5/s 的端点: rate = min(1, 10/5) = 100%
  0.5/s 的端点: rate = min(1, 10/0.5) = 100%

# jaeger-collector.yaml
collector:
  # 自适应采样配置
  sampling:
    # 策略存储
    strategiesFile: "/etc/jaeger/sampling_strategies.json"
    # 自动更新间隔
    strategiesFileReloadInterval: 5m

    # 每端点每秒目标采样数
    defaultTargetPerSecond: 10
    # 最小采样率
    minSamplingRate: 0.001

与传统固定概率采样对比：

固定 1% 采样：
  1000/s 端点 → 10 条 (足够)
  5/s 端点 → 0.05 条 (信息不足)

自适应采样：
  1000/s 端点 → 10 条 (1%)
  5/s 端点 → 5 条 (100%)
  总存储 ≈ 固定采样，但低流量端点信息完整

方式一：OTel SDK → Jaeger (OTLP)
  App → OTel SDK → Jaeger Collector (gRPC:4317)
  Jaeger v1.35+ 原生支持

方式二：OTel SDK → OTel Collector → Jaeger
  App → OTel SDK → OTel Collector → Jaeger (OTLP/Jaeger gRPC)

方式三：Jaeger SDK → Jaeger Agent → Jaeger Collector
  旧架构，已被 OTel 取代

# jaeger-collector.yaml
collector:
  # 直接接收 OTLP 数据
  otlp:
    grpc:
      hostPort: ":4317"
    http:
      hostPort: ":4318"

  # 仍然支持 Jaeger 原生协议
  grpc:
    hostPort: ":14250"

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

exporters:
  otlp:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp]

从 Jaeger SDK → OTel SDK 的迁移：

1. 添加 OTel SDK 依赖
2. 使用 OTel API 替代 Jaeger API
3. 配置 OTLP Exporter 指向 Jaeger
4. 逐步下线 Jaeger SDK

优势：
  - 同一套 API 可同时发给 Jaeger 和其他后端
  - 未来可切换后端无需修改代码
  - 支持 Traces + Metrics + Logs 全套

# Jaeger 组件 Prometheus 指标端口
collector:
  metricsHTTPPort: 14271   # /metrics

query:
  metricsHTTPPort: 16687   # /metrics

agent:
  adminPort: 14269         # /metrics

groups:
  - name: jaeger
    rules:
      - alert: JaegerCollectorDown
        expr: up{job="jaeger-collector"} == 0
        for: 1m

      - alert: JaegerQueueBacklog
        expr: jaeger_collector_queue_length > 10000
        for: 5m

      - alert: JaegerSpansNotStored
        expr: rate(jaeger_collector_spans_received_total[5m]) - rate(jaeger_collector_spans_saved_total[5m]) > 100
        for: 5m

      - alert: JaegerHighTraceLatency
        expr: histogram_quantile(0.99, rate(jaeger_query_latency_seconds_bucket[5m])) > 5
        for: 5m

{
    "title": "Jaeger Overview",
    "panels": [
        {"title": "Span Ingestion Rate", "targets": [{"expr": "rate(jaeger_collector_spans_received_total[5m])"}]},
        {"title": "Storage Savings Rate", "targets": [{"expr": "rate(jaeger_collector_spans_saved_total[5m])"}]},
        {"title": "Queue Length", "targets": [{"expr": "jaeger_collector_queue_length"}]},
        {"title": "Query Latency P99", "targets": [{"expr": "histogram_quantile(0.99, rate(jaeger_query_latency_seconds_bucket[5m]))"}]}
    ]
}

graph TD
    Agent[Agent] --> Collector[Collector]
    Collector --> Kafka[Kafka]
    Kafka --> Ingester[Ingester]
    Ingester --> Storage[Storage]
    Kafka --> D1[解耦生产/消费]
    D1 --> D2[高吞吐写入 - 无背压问题]
    D1 --> D3[异步消费 - 可回溯重试]

# jaeger-collector.yaml
collector:
  kafka:
    producer:
      topic: "jaeger-spans"
      borkers: "kafka-1:9092,kafka-2:9092"
      # 压缩
      compression: "gzip"
      # 批量配置
      batch-size: 10000
      batch-linger: 500ms
      # 写入确认
      required-acks: -1     # all

# jaeger-ingester.yaml
ingester:
  kafka:
    consumer:
      topic: "jaeger-spans"
      borkers: "kafka-1:9092,kafka-2:9092"
      # 消费者组
      group-id: "jaeger-ingester"
      # Offset
      auto-offset-reset: "earliest"
      # 批量消费
      fetch-min: 1000
      fetch-max-wait: 1s
      # 并发
      num-workers: 100
  # 存储写入
  storage:
    type: elasticsearch
    elasticsearch:
      hostPorts: "es-cluster:9200"
      bulk:
        size: 5000000
        workers: 10

Collector 写入 Kafka:
  - 写入失败 → 返回背压到 Agent
  - Kafka 集群故障 → Collector 队列积压

Ingester 消费 Kafka:
  - 消费失败 → 自动重试
  - 处理速度 < 写入速度 → 自动延迟
  - 进程重启 → 从提交的 Offset 继续

# 安装 Operator
kubectl create namespace observability
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/download/v1.57.0/jaeger-operator.yaml -n observability

# 创建 Jaeger 实例

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        hostPorts: "es-cluster:9200"
        indexPrefix: "jaeger"
        indexDateSeparator: "-"
        numberOfShards: 3
        numberOfReplicas: 1
  collector:
    options:
      collector:
        queue-size: 20000
        num-workers: 100
    resources:
      limits:
        memory: 4Gi
        cpu: "2"
      requests:
        memory: 2Gi
        cpu: "1"
  query:
    options:
      query:
        max-clock-skew-adjustment: "0s"
    resources:
      limits:
        memory: 1Gi
        cpu: "1"
    ingress:
      enabled: true
      annotations:
        kubernetes.io/ingress.class: nginx
      hosts:
        - jaeger.example.com
  agent:
    strategy: DaemonSet
    options:
      agent:
        reporter:
          collector:
            hostPort: "jaeger-prod-collector:14250"
  ingester:
    options:
      ingester:
        deadlockInterval: "0s"
    resources:
      limits:
        memory: 2Gi
        cpu: "1"

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm upgrade --install jaeger jaegertracing/jaeger \
  --namespace observability \
  --set provisionDataStore.cassandra=false \
  --set storage.elasticsearch.host=es-cluster \
  --set storage.elasticsearch.port=9200

jaeger-span-2026-05-26    # 每日 Span 索引
jaeger-service-2026-05-26  # 服务信息索引
jaeger-dependencies-2026-05-26  # 依赖图索引

PUT /_ilm/policy/jaeger-span-policy
{
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {"max_age": "1d", "max_size": "50gb"}
                }
            },
            "delete": {
                "min_age": "7d",
                "actions": {"delete": {}}
            }
        }
    }
}

PUT /_index_template/jaeger-span
{
    "index_patterns": ["jaeger-span-*"],
    "template": {
        "settings": {
            "number_of_shards": 3,
            "number_of_replicas": 1,
            "index.lifecycle.name": "jaeger-span-policy",
            "index.lifecycle.rollover_alias": "jaeger-span"
        }
    }
}

# jaeger-collector 和 jaeger-query
storage:
  elasticsearch:
    esSpanTTL: "72h"               # Span 数据保留 3 天
    esServiceCacheTTL: "12h"       # 服务缓存
    esDependenciesTTL: "72h"       # 依赖图数据保留

每个 Span 平均 2-5KB (含索引)

日处理量: 1 亿 Spans
每日 ES 存储: 1亿 × 3KB × 1.3(索引开销) × 2(副本) ≈ 780GB

TTL 7 天: 780GB × 7 ≈ 5.5TB
TTL 3 天: 780GB × 3 ≈ 2.3TB

UI 操作流程：
  1. 搜索列表勾选 2-10 个 Trace
  2. 点击 "Compare Traces"
  3. 并列显示甘特图
  4. 统计汇总表格

对比维度：
  - 总耗时对比
  - 各服务耗时分布
  - 错误 Span 标记
  - Span 数量差异

甘特图并列（时间缩放同步）：
  Trace A: [======50ms=====][------30ms------][==20ms==]
  Trace B: [==10ms=][------150ms------][==15ms==]
                    ↑ 差异热点

统计对比表：
  | Span Name        | Trace A | Trace B | 差异     |
  | HTTP GET /api    | 50ms    | 10ms    | +40ms    |
  | ServiceB/Process | 30ms    | 150ms   | +120ms ✗ |
  | DB Query users   | 20ms    | 15ms    | +5ms     |

1. 慢请求 vs 正常请求 → 定位耗时差异点
2. 错误请求 vs 正常请求 → 找出出错的服务
3. 新旧版本对比 → 验证性能优化效果
4. A/B 实验 → 不同路由的性能差异

# 1. 停止 Span 写入（可选）
kubectl scale deployment jaeger-collector --replicas=0

# 2. 滚动升级 ES 节点
# ES 7.x → 8.x 需注意兼容性

# 3. 重启 Jaeger Query 连接新 ES
kubectl rollout restart deployment jaeger-query

# 4. 恢复 Collector
kubectl scale deployment jaeger-collector --replicas=3

# 使用 reindex 迁移数据
POST _reindex
{
    "source": {"index": "jaeger-span-2026-05-25", "host": "old-es:9200"},
    "dest": {"index": "jaeger-span-2026-05-25", "host": "new-es:9200"}
}

新存储配置需要启动新的 Collector 和 Query
旧数据保留在原存储中，可用于历史查询
新 Collector 写入新存储
两个 Query 实例分别连接新旧存储
过渡期后下线旧存储

1. ES 版本兼容性
   - 建议大版本内升级
   - 跨版本需逐步进行

2. 数据格式兼容
   - Jaeger 新版本可能修改索引 mapping
   - 需验证兼容性

3. 回滚计划
   - 保留旧版本配置
   - 数据备份
   - 迁移路径可逆

Jaeger v1:
  Agent (UDP) → Collector (队列) → Storage → Query → UI
  (多组件，Thrift 协议)

Jaeger v2:
  OTel Collector (OTLP) → Storage → OTel Collector (Query) → UI
  (统一 Collector，OTLP 协议)

1. 统一 OTel 生态
   - 复用 OTel Collector 的 Receiver/Processor/Exporter
   - 与 Metrics/Logs 共享同一 Collector

2. 简化运维
   - 更少的组件
   - 统一的配置
   - 相同的二进制

3. 更强的扩展性
   - OTel Collector 插件体系
   - 社区贡献的 Processor 可直接使用

4. 更好的兼容性
   - OTLP 是所有信号的标准
   - 减少协议转换开销

Web 前端 (HTML/JS)
    │
  HTTP
    │
Customer Service (Go) ──── HTTP ──── Route Service (Go)
    │                                   │
  内部逻辑                              gRPC
    │                                   │
  MySQL (嵌入式)                    Redis (地理位置)

# 使用 all-in-one 镜像自带 Hot Rod
docker run -d --name jaeger \
  -p 16686:16686 \
  -p 8080:8080 \
  jaegertracing/all-in-one:latest

# 访问 Hot Rod 页面
# http://localhost:8080

# 访问 Jaeger UI
# http://localhost:16686

展示 Jaeger 在生产场景中的实际使用方式
验证 Trace 是否完整覆盖了所有服务调用
了解不同协议（HTTP/gRPC）的 Span 传播
观察采样策略对 Trace 数量的影响

服务端 HTTP: {HTTP Method} {Route}
  ✅ "GET /api/users/{id}"
  ❌ "GET /api/users/123"  (含具体 ID)

数据库: {DB Type} {Operation} {Table}
  ✅ "MySQL SELECT users"
  ✅ "Redis GET user:123"

消息队列: {Topic} {Operation}
  ✅ "Kafka orders CONSUME"
  ✅ "RabbitMQ orders PUBLISH"

客户端 (service A)                   服务端 (service B)
    │                                     │
  gRPC Call With Metadata                  │
    ├── traceparent: 00-abc123-def456-01   │
    ├── grpc-timeout: 5000ms               │
    │                                     │
  Extract Metadata                         │
  Create Child Span                        │
    │                                     │
  Handle Request (超时: 5s)               │

import (
    "google.golang.org/grpc"
    "github.com/opentracing-contrib/go-grpc"
)

// 服务端
server := grpc.NewServer(
    grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()),
)

// 客户端
conn, _ := grpc.Dial("service-b:8080",
    grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
    grpc.WithBlock(),
)

客户端 (Deadline 5s)
    │
  gRPC Metadata: grpc-timeout=5000m
    │
服务端接收
    │
  如果 deadline 已过期 → 直接返回超时
  否则 → 创建上下文
    │
  子调用继承剩余 deadline
  客户端取消 → 传播 Cancel 通知

避免级联等待：
  服务A (5s) → 服务B (继承 4.5s) → 服务C (继承 4s)
                                   ↓
                            如果 C 超过 4s → 取消
                            通知 B → 通知 A → 快速失败

import "github.com/opentracing/opentracing-go/log"

// 记录异常事件
span.LogFields(
    log.String("event", "error"),
    log.String("type", "DatabaseError"),
    log.String("message", err.Error()),
    log.String("stack", string(debug.Stack())),
)

// 记录关键审计事件
span.LogFields(
    log.String("event", "audit"),
    log.String("action", "user.login"),
    log.String("user_id", userID),
    log.String("ip", clientIP),
    log.String("result", "success"),
)

import "github.com/opentracing/ext"

// 标记 Span 为错误
ext.Error.Set(span, true)
span.SetTag("error.message", err.Error())
span.SetTag("error.type", "business_error")

记录异常：
  ✓ 记录 event=error + message + stack
  ✓ 标记 Span 为 error=true
  ✓ 记录异常的因果关系

不记录：
  ✗ 重复日志（SDK 已有）
  ✗ 敏感信息（密码、Token）
  ✗ 巨量错误信息（截断处理）

# Collector gRPC TLS
collector:
  grpc:
    hostPort: ":14250"
    tls:
      enabled: true
      cert: "/certs/collector.crt"
      key: "/certs/collector.key"
      clientCa: "/certs/ca.crt"    # 客户端认证（mTLS）

  # HTTP TLS
  http:
    tls:
      enabled: true
      cert: "/certs/collector.crt"
      key: "/certs/collector.key"

reporter:
  collectorEndpoint: "https://jaeger-collector:14268/api/traces"
  tls:
    enabled: true
    cert: "/certs/agent.crt"
    key: "/certs/agent.key"
    serverName: "jaeger-collector"

# Collector HTTP
collector:
  http:
    # 基本认证
    basicAuth:
      enabled: true
      username: "jaeger"
      password: "password"

# Query HTTP
query:
  http:
    basicAuth:
      enabled: true
      username: "admin"
      password: "admin"

storage:
  elasticsearch:
    username: "elastic"
    password: "${ES_PASSWORD}"
    # TLS
    tls:
      enabled: true
      ca: "/certs/es-ca.crt"
      skipHostVerify: false

containers:
  - name: jaeger-collector
    livenessProbe:
      httpGet:
        path: /
        port: 14269
      initialDelaySeconds: 10
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /
        port: 14269
      initialDelaySeconds: 5
      periodSeconds: 5

Collector 面板：
  - Span 接收/存储速率 (rate)
  - 队列积压 (gauge)
  - 队列延迟 (histogram)

Query 面板：
  - QPS (rate)
  - P99 查询延迟 (histogram_quantile)
  - 服务/操作用户分布

Ingester 面板：
  - Kafka 消费速率 (rate)
  - 处理延迟 (histogram)
  - Offset Lag (gauge)

// Agent Metrics
{
    "jaeger_agent_thrift_udp_buffer_size": 32768,
    "jaeger_agent_thrift_udp_read_packets": 100000,
    "jaeger_agent_thrift_udp_read_failed": 50,   // 读取失败
    "jaeger_agent_thrift_processors_batch_size": 1000
}

// Collector Metrics
{
    "jaeger_collector_spans_received_total": 100000,
    "jaeger_collector_spans_saved_total": 99000,  // 高于 1% 差距
}

1. 增大 UDP 接收缓冲区
   sysctl -w net.core.rmem_max=26214400
   sysctl -w net.core.rmem_default=26214400

2. 使用 gRPC 替代 UDP（推荐）
   直连 Collector，可靠传输

3. 增加 Agent 批量处理能力
   reporter.batchSize: 10000
   reporter.batchWorkers: 100

4. 使用 Kafka 缓冲层
   Agent → Collector → Kafka → Ingester

Agent UDP (传统)：
  优势：极低延迟，零资源消耗
  劣势：不可靠，数据丢失
  适用：低吞吐量，可容忍丢失

gRPC 直连 (推荐)：
  优势：可靠传输，批量高效
  劣势：TCP 连接开销
  适用：生产环境

// ❌ 过多标签
span.SetTag("http.method", "GET")
span.SetTag("http.url", "http://api.example.com/api/users?page=1&limit=10")
span.SetTag("http.host", "api.example.com")
span.SetTag("http.path", "/api/users")
span.SetTag("http.query", "page=1&limit=10")
span.SetTag("http.scheme", "https")
span.SetTag("http.target", "/api/users")
span.SetTag("http.flavor", "1.1")
span.SetTag("http.user_agent", "Mozilla/5.0...")
span.SetTag("http.content_length", "1234")

// ✅ 最小必要标签
span.SetTag("http.method", "GET")
span.SetTag("http.url", "/api/users")
span.SetTag("http.status_code", 200)

// 减少非必要 Span
// 内联简单方法调用（不创建 Span）
func processRequest() {
    // 不在此处创建 Internal Span
    validate()
    process()
    save()
}

// 只在关键边界创建 Span
func handleRequest() {          // Entry Span
    result := callServiceB()    // Exit Span
    return result
}

1. 自适应采样
   - 动态调整采样率
   - 低流量端点高采样

2. 远程采样策略
   - 中央下发采样配置
   - 实时调整

3. Trace 对比
   - 并列甘特图
   - 差异分析

4. Hot Rod 演示
   - 完整示例应用
   - 教学演示

1. Brave 客户端完善
   - Java/Spring 生态成熟
   - Sleuth → Spring Cloud 集成

2. MySQL 存储支持
   - 小团队场景
   - 无需额外存储组件

3. 轻量级架构
   - 组件少
   - 易于上手