Metrics Server 面试题

Metrics Server 启动
    ↓
1. 监听 Node / Pod 列表（List-Watch API）
    ↓
2. 对每个 Node 并发请求 Summary API
   GET https://<NodeIP>:10250/stats/summary
    ↓
3. 解析 cgroups 数据 → 计算 CPU 使用率（核数）
    ↓
4. 校验数据新鲜度（默认 1 分钟内）
    ↓
5. 聚合到内存 → 通过 Metrics API 暴露

args:
  - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  - --kubelet-insecure-tls=true          # 自签证书场景
  - --metric-resolution=15s              # 采集间隔
  - --request-timeout=10s                # 单节点请求超时
  - --node-selector=kubernetes.io/os=linux

公式：
  CPU 使用率(核) = node_cpu_usage_seconds_total[1m] rate × 1
  CPU 使用率(%) = CPU 使用率(核) / node_cpu_capacity

底层：
  cgroup cpuacct.usage（纳秒）→ 差分 → 转换为核数

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
    # 1. Resource Metrics（Metrics Server 提供）
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    # 2. Custom Metrics（Custom Adapter 提供）
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "1k"
    # 3. External Metrics（External Adapter 提供）
    - type: External
      external:
        metric:
          name: queue_messages_ready
          selector:
            matchLabels:
              queue: orders
        target:
          type: Value
          value: "30"

desiredReplicas = ceil[currentReplicas × (currentMetricValue / desiredMetricValue)]

注意：仅当 metrics 字段中所有指标均计算后，HPA 才发布新副本数
（多指标取最大副本数）

Metrics Server:
  - HPA 副本伸缩（CPU/内存触发）
  - kubectl top node/pod 实时查看
  - Scheduler 调度（resource 感知）
  - VPA 垂直 Pod 伸缩

kube-state-metrics:
  - Prometheus 采集 → Grafana 仪表盘
  - 告警规则（Pod 长时间未就绪、Deployment 副本不一致）
  - 业务监控（Job 失败次数、StatefulSet 滚动状态）
  - 关联分析（Pod 状态、节点健康、Owner Reference）

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
spec:
  service:
    name: metrics-server
    namespace: kube-system
    port: 443
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true  # 自签证书
  groupPriorityMinimum: 100
  versionPriority: 100

1. kube-apiserver 启动参数：
   --enable-aggregator-routing=true        # 1.18+ 默认开启
   --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
   --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
   --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
   --requestheader-username-headers=X-Remote-User
   --requestheader-group-headers=X-Remote-Group
   --requestheader-extra-headers-prefix=X-Remote-Extra-

2. Aggregator 工作原理：
   - kube-apiserver 监听到 APIService CR
   - 注册路由：/apis/metrics.k8s.io/* → 后端 Service
   - 转发时携带原始请求头 + 客户端 CA 证书
   - 后端 Service 验证请求头中的客户端 CA

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
- apiGroups: [""]
  resources: ["pods", "nodes", "nodes/stats", "namespaces"]
  verbs: ["get", "list", "watch"]
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

# 查看 APIService 注册状态
kubectl get apiservices | grep metrics
kubectl describe apiservice v1beta1.metrics.k8s.io

# 直接访问 Metrics API
kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods

# 验证 aggregator 启用
kubectl -n kube-system get pod -l k8s-app=metrics-server
kubectl logs -n kube-system -l k8s-app=metrics-server

# 添加仓库
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server
helm repo update

# 最小化部署（自签证书环境）
helm upgrade --install metrics-server metrics-server/metrics-server \
  --namespace kube-system \
  --set args[0]="--kubelet-insecure-tls" \
  --set args[1]="--kubelet-preferred-address-types=InternalIP"

# 生产推荐配置
helm upgrade --install metrics-server metrics-server/metrics-server \
  --namespace kube-system \
  --version 3.12.0 \
  --set replicas=1 \
  --set args[0]="--kubelet-preferred-address-types=InternalIP" \
  --set args[1]="--kubelet-insecure-tls=true" \
  --set args[2]="--metric-resolution=15s" \
  --set args[3]="--request-timeout=10s" \
  --set resources.requests.cpu=100m \
  --set resources.requests.memory=200Mi \
  --set resources.limits.cpu=500m \
  --set resources.limits.memory=512Mi \
  --set serviceMonitor.enabled=true \
  --set serviceMonitor.namespace=monitoring \
  --set podDisruptionBudget.enabled=true

# Chart 关键配置项
replicas: 1                    # 不支持 HA
image:
  repository: registry.k8s.io/metrics-server/metrics-server
  tag: v0.7.2
args:
  - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  - --kubelet-insecure-tls=true
  - --metric-resolution=15s
  - --request-timeout=10s
resources:
  requests:
    cpu: 100m
    memory: 200Mi
  limits:
    cpu: 500m
    memory: 512Mi
nodeSelector:
  kubernetes.io/os: linux
tolerations:
  - operator: Exists
    effect: NoSchedule
podDisruptionBudget:
  enabled: true
  minAvailable: 1
priorityClassName: system-cluster-critical
serviceAccount:
  create: true
  name: metrics-server
serviceMonitor:
  enabled: true                 # 由 Prometheus Operator 采集
  namespace: monitoring
  interval: 30s

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 1. Pod 状态
kubectl -n kube-system get pod -l k8s-app=metrics-server

# 2. APIService 状态
kubectl get apiservice v1beta1.metrics.k8s.io
# 输出 AVAILABLE=True 表示正常

# 3. 功能验证
kubectl top node
kubectl top pod -A

# 4. Metrics API 直接调用
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq .

# 5. HPA 联动验证
kubectl get hpa -A

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prometheus-adapter prometheus-community/prometheus-adapter \
  --namespace monitoring \
  --set prometheus.url=http://kube-prometheus-stack-prometheus.monitoring.svc \
  --set rules.default=false \
  --set rules.custom[0].seriesQuery='http_requests_total{namespace!="",pod!=""}' \
  --set rules.custom[0].resources.overrides.namespace.resource=namespace \
  --set rules.custom[0].resources.overrides.pod.resource=pod \
  --set rules.custom[0].name.matches='^(.*)_total$' \
  --set rules.custom[0].name.as='${1}_per_second' \
  --set rules.custom[0].metricsQuery='sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

# prometheus-adapter ConfigMap
rules:
  # 自定义指标 1：HTTP QPS
  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: { resource: "namespace" }
        pod: { resource: "pod" }
    name:
      matches: "^(.*)_total$"
      as: "${1}_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

  # 自定义指标 2：P99 延迟
  - seriesQuery: 'http_request_duration_seconds_bucket{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: { resource: "namespace" }
        pod: { resource: "pod" }
    name:
      matches: "^http_request_duration_seconds_bucket$"
      as: "http_request_duration_seconds_p99"
    metricsQuery: |
      histogram_quantile(0.99,
        sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (le, <<.GroupBy>>)
      )

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa-custom
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 20
  metrics:
    # Custom Metrics：基于 Pod 维度
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "1k"

    # External Metrics：基于外部系统
    - type: External
      external:
        metric:
          name: rabbitmq_queue_messages_ready
          selector:
            matchLabels:
              queue: orders
        target:
          type: AverageValue
          averageValue: "100"

# 1. APIService 状态
kubectl get apiservice | grep -E "custom.metrics|external.metrics"

# 2. Custom Metrics API
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests_per_second

# 3. External Metrics API
kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/default/rabbitmq_queue_messages_ready

# 1. 确认 Pod 状态
kubectl -n kube-system get pod -l k8s-app=metrics-server
kubectl -n kube-system logs -l k8s-app=metrics-server --tail=100

# 2. 确认 APIService 注册
kubectl get apiservice v1beta1.metrics.k8s.io
kubectl describe apiservice v1beta1.metrics.k8s.io

# 3. 直接查询 Metrics API
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes

# 4. 验证 kubelet Summary API 可达性
kubectl -n kube-system exec -it metrics-server-pod -- \
  curl -k https://<NodeIP>:10250/stats/summary

# 5. 抓包分析（深层次排查）
kubectl -n kube-system exec -it metrics-server-pod -- \
  tcpdump -i any port 10250 -w /tmp/cap.pcap

# 在 Master 节点生成 kubelet serving 证书 CSR
kubectl get csr

# 签发（kubelet 证书需包含 node IP / DNS SAN）
cfssl gencert \
  -ca=/etc/kubernetes/pki/ca.crt \
  -ca-key=/etc/kubernetes/pki/ca.key \
  -config=ca-config.json \
  -hostname=<NodeIP>,<NodeDNS>,<NodeName> \
  -profile=kubernetes \
  kubelet-csr.json | cfssljson -bare kubelet

# 签发后移除 --kubelet-insecure-tls 参数

# Metrics Server 配置
args:
  - --metric-resolution=15s          # 采集间隔

# HPA Controller 配置（kube-controller-manager）
--horizontal-pod-autoscaler-sync-period=15s
--horizontal-pod-autoscaler-tolerance=0.1
--horizontal-pod-autoscaler-cpuInitializationPeriod=5m
--horizontal-pod-autoscaler-initial-readiness-delay=30s

# 数据新鲜度必须 < sync-period，否则 HPA 报错
# 推荐：metric-resolution ≤ sync-period / 2

1. 调高 Pod 资源限制
   resources:
     limits:
       cpu: 1
       memory: 1Gi

2. 调大 kubelet 并发与超时
   <!-- --max-requests-per-second 参数不存在；metrics-server 使用 --kubelet-qps 控制 kubelet 并发 -->
   --kubelet-qps=10                  # 控制 kubelet 请求 QPS

3. 调优 APIService（kube-apiserver 侧）
   --max-requests-inflight=2000
   --max-mutating-requests-inflight=1000

4. 启用 ServiceMonitor（监控 Metrics Server 自身健康）
   metrics_server_healthy = up{job="metrics-server"}

5. 监控告警：
   - Metrics Server Pod OOMKill
   - 采集耗时 > 10s
   - 节点 metrics 缺失 > 1%
   - APIService unavailable

[ ] 1. 部署单实例 + PodDisruptionBudget
[ ] 2. 签发合法 kubelet 证书（禁用 insecure-tls）
[ ] 3. 设置 priorityClassName=system-cluster-critical
[ ] 4. 配置 resource.requests/limits
[ ] 5. 启用 ServiceMonitor + 告警规则
[ ] 6. metric-resolution 与 HPA sync-period 匹配
[ ] 7. 部署 prometheus-adapter 支持 Custom / External Metrics
[ ] 8. Windows 节点通过 nodeSelector 排除
[ ] 9. 验证 kubectl top node / pod 正常输出
[ ] 10. 定期演练：HPA 扩容链路全链路压测