Nvidia GPU Operator 面试题

graph TD
    A[Nvidia GPU Operator] --> B[Node Feature Discovery<br/>NFD]
    A --> C[GPU Driver Installer<br/>DaemonSet]
    A --> D[GPU Feature Discovery<br/>GFD]
    A --> E[DCGM Exporter]
    A --> F[Container Runtime<br/>nvidia-container-toolkit]
    A --> G[Device Plugin]
    A --> H[MIG Manager]
    A --> I[GPU Time-Slicing]

    B --> B1[检测 GPU 硬件特征]
    B --> B2[节点标签注入]

    C --> C1[自动安装 GPU 驱动]
    C --> C2[驱动版本管理]

    D --> D1[GPU 型号和显存检测]
    D --> D2[节点 GPU 标签]

    E --> E1[GPU 指标暴露]
    E --> E2[遥测数据采集]

    F --> F1[Docker/containerd/cri-o<br/>运行时配置]
    F --> F2[GPU 设备挂载]

    G --> G1[nvidia.com/gpu<br/>资源注册]
    G --> G2[GPU 分配和调度]

    H --> H1[MIG 配置管理]
    H --> H2[GPU 分区]

    I --> I1[GPU 时间片调度]
    I --> I2[共享 GPU 资源]

1. NFD 检测 GPU 硬件
2. Driver DaemonSet 启动
3. 下载 GPU 驱动包
4. 编译内核模块（dkms）
5. 安装 nvidia.ko
6. 创建 /dev/nvidia* 设备
7. 验证驱动加载
8. 节点 Ready

apiVersion: v1
kind: ConfigMap
metadata:
  name: nvidia-driver-config
data:
  driverVersion: "535.154.05"
  image: nvcr.io/nvidia/driver:535.154.05-ubuntu22.04
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-driver-daemonset
spec:
  selector:
    matchLabels:
      app: nvidia-driver
  template:
    spec:
      containers:
        - name: nvidia-driver
          image: nvcr.io/nvidia/driver:535.154.05-ubuntu22.04
          securityContext:
            privileged: true
          volumeMounts:
            - name: host-root
              mountPath: /host
          env:
            - name: ROOT_MOUNT_DIR
              value: /host
      volumes:
        - name: host-root
          hostPath:
            path: /

graph TD
    A[GPU Device Plugin] -->|1. ListAndWatch 注册 GPU 资源| B[向 Kubelet 注册 nvidia.com/gpu]
    B --> B1[上报 GPU 数量]
    B --> B2[设备 ID 和健康状态]
    A -->|2. Allocate 分配 GPU| C[接收 Pod 的 GPU 请求]
    C --> C1[返回环境变量]
    C --> C2[返回设备卷挂载]
    A -->|3. 健康检测| D[定期检查 GPU 状态]
    D --> D1[检测 GPU 故障]
    D --> D2[异常 GPU 从可用列表移除]

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin-daemonset
spec:
  selector:
    matchLabels:
      name: nvidia-device-plugin-ds
  template:
    spec:
      containers:
        - name: nvidia-device-plugin-ctr
          image: nvcr.io/nvidia/k8s-device-plugin:v0.14.0
          env:
            - name: NVIDIA_VISIBLE_DEVICES
              value: all
          volumeMounts:
            - name: device-plugin
              mountPath: /var/lib/kubelet/device-plugins
      volumes:
        - name: device-plugin
          hostPath:
            path: /var/lib/kubelet/device-plugins

apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-config
data:
  config.yaml: |
    version: v1
    flags:
      migEnabled: true
    migConfigs:
      all:
        - devices: [0,1,2,3,4,5,6,7]
          migEnabled: true
          migDeviceSpecs:
            - devices:
              - parentDevice: 0
                migProfile: 1g.10gb
              - parentDevice: 1
                migProfile: 3g.40gb    

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4
            failRequestsGreaterThanOne: false    

apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-manager-config
data:
  config.yaml: |
    version: v1
    flags:
      migEnabled: true
    migConfigs:
      node-0:
        - devices: ["GPU-xxx"]
          migEnabled: true
          migDeviceSpecs:
            - devices:
                - parentDevice: "GPU-xxx"
                  migProfile: "3g.40gb"    

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: cuda-container
      image: nvidia/cuda:12.0-base
      resources:
        limits:
          nvidia.com/gpu: 1
          nvidia.com/mig-profile: "1g.10gb"

groups:
  - name: gpu-alerts
    rules:
      - alert: GPUHighTemperature
        expr: DCGM_FI_DEV_TEMPERATURE{gpu=~".*"} > 85
        for: 5m
        labels:
          severity: critical

      - alert: GPUXIDError
        expr: rate(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
        for: 1m
        labels:
          severity: critical

      - alert: GPUMemoryLow
        expr: (DCGM_FI_DEV_FB_FREE / (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) * 100 < 10
        for: 5m
        labels:
          severity: warning

# 节点选择（指定 GPU 型号）
apiVersion: v1
kind: Pod
spec:
  nodeSelector:
    nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB

# 拓扑约束
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: nvidia.com/gpu.count
                operator: Gt
                values:
                  - "2"

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: gpu-container
      resources:
        limits:
          nvidia.com/gpu: 2   # 申请 2 块 GPU

NVLink 连接拓扑:
  GPU-0 ── NVLink ── GPU-1
    │                  │
  NVLink             NVLink
    │                  │
  GPU-2 ── NVLink ── GPU-3

NVSwitch（DGX A100/H100）:
  所有 GPU 通过 NVSwitch 全互联
  任意 GPU 间 P2P 带宽: 600GB/s (A100)

apiVersion: v1
kind: ConfigMap
metadata:
  name: topology-config
data:
  topology.yaml: |
    version: v1
    policies:
      - name: best-effort
        # 尽量分配同一 NVSwitch 域的 GPU
        preferences:
          - nvlink
        # 至少分配同一节点
        requirements:
          - numa    

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: training
      resources:
        limits:
          nvidia.com/gpu: 8
          nvidia.com/nvlink: 4

XID 错误类型:
  XID 13: GPU 停止运行（致命）
  XID 31: 显存页面错误
  XID 43: GPU 挂起
  XID 48: 双 ECC 错误
  XID 64: GPU 掉线
  XID 79: GPU 温度过高

处理策略:
  非致命错误（XID 31）:
    标记异常 GPU
    通知上层应用
  
  致命错误（XID 13/43/48）:
    从可用设备列表移除
    驱逐受影响 Pod
    触发节点维护

# 设备插件健康检测配置
apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: "mixed"
      failOnInitError: true
      deviceListStrategy: envvar
      nvidiaDriverRoot: /
    health:
      healthCheckInterval: 10s
      unhealthyDeviceThreshold: 3    

# GPU 故障时的 Pod 驱逐
apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: gpu-workload
          resources:
            limits:
              nvidia.com/gpu: 1
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoExecute

1. 排空 GPU 节点
   kubectl drain <node> --ignore-daemonsets

2. 更新 Driver 版本
   kubectl set image daemonset/nvidia-driver-daemonset \
     nvidia-driver=nvcr.io/nvidia/driver:545.23.08-ubuntu22.04

3. 验证驱动加载
   kubectl logs nvidia-driver-xxxxx
   kubectl exec nvidia-driver-xxxxx -- nvidia-smi

4. 恢复调度
   kubectl uncordon <node>

# Helm 升级
helm upgrade gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --version v23.9.0 \
  --set driver.version=545.23.08

# 验证升级
kubectl get pods -n gpu-operator
kubectl describe pod nvidia-device-plugin-xxxxx

1. 驱动升级需要节点排空
2. 大版本升级需逐步执行
3. 备份配置和 CRD
4. 监控 GPU 工作负载状态
5. 灰度升级（分批节点）

MIG:
  - 多租户 GPU 资源隔离
  - 混合负载（训练 + 推理）
  - 需要确定性性能

vGPU:
  - VDI 桌面虚拟化
  - 开发测试环境
  - 需要 GPU 热迁移

apiVersion: v1
kind: ResourceQuota
metadata:
  name: gpu-quota
  namespace: ai-training
spec:
  hard:
    requests.nvidia.com/gpu: 8
    limits.nvidia.com/gpu: 8
apiVersion: v1
kind: LimitRange
metadata:
  name: gpu-limits
spec:
  limits:
    - max:
        nvidia.com/gpu: 4
      min:
        nvidia.com/gpu: 1
      type: Container

# 多团队 GPU 配额
apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-gpu
  namespace: team-a
spec:
  hard:
    nvidia.com/gpu: 16

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-b-gpu
  namespace: team-b
spec:
  hard:
    nvidia.com/gpu: 8

全局调度层
    │
    ├── 集群 A (GPU: A100×32)
    │   └── GPU Operator
    │
    ├── 集群 B (GPU: A100×64)
    │   └── GPU Operator
    │
    └── 集群 C (GPU: H100×128)
        └── GPU Operator

全局资源视图:
  - Total GPU: 224
  - Total VRAM: 17.5TB

# Volcano 跨集群调度
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: multi-cluster-gpu
spec:
  weight: 1
  capability:
    cpu: "1000"
    memory: "8TB"
    nvidia.com/gpu: "224"

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
spec:
  schedulerName: volcano
  queue: multi-cluster-gpu
  tasks:
    - replicas: 32
      template:
        spec:
          containers:
            - name: training
              resources:
                limits:
                  nvidia.com/gpu: 8

apiVersion: v1
kind: Namespace
metadata:
  name: gpu-workloads
  labels:
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/warn: baseline

# SELinux 上下文配置
apiVersion: v1
kind: Pod
spec:
  securityContext:
    seLinuxOptions:
      level: "s0:c1,c2"
      type: "container_t"
  containers:
    - name: gpu-container
      securityContext:
        privileged: true
        capabilities:
          add:
            - SYS_ADMIN
            - SYS_PTRACE

# NVIDIA Container Toolkit 安全配置
/etc/nvidia-container-runtime/config.toml:
  # 禁用不需要的特性
  disable-require: false
  # 启用 NVIDIA 驱动限制
  nvidia-container-cli:
    # 显存限制
    environment: ["NVIDIA_VISIBLE_DEVICES=all"]
    # 限制计算能力
    capabilities:
      - compute
      - utility

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --set driver.version=535.154.05 \
  --set operator.defaultRuntime=crio \
  --set toolkit.version=1.14.0 \
  --set devicePlugin.version=v0.14.0 \
  --set migManager.enabled=true

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: cluster-policy
spec:
  operator:
    defaultRuntime: containerd
    deployGFD: true
  driver:
    enabled: true
    version: "535.154.05"
  toolkit:
    enabled: true
  devicePlugin:
    enabled: true
    config:
      name: device-plugin-config
  migManager:
    enabled: true
    config:
      name: mig-manager-config
  gfd:
    enabled: true

1. 容器创建请求
       │
2. OCI runtime spec 生成
       │
3. nvidia-container-runtime-hook 拦截
       │
4. 检测 NVIDIA_VISIBLE_DEVICES 环境变量
       │
5. 注入 GPU 设备文件（/dev/nvidia*）
       │
6. 注入 GPU 库依赖（libcuda.so、libnvidia-ml.so）
       │
7. 挂载 nvidia-container-toolkit 驱动目录
       │
8. 容器启动，GPU 可见

# /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

apiVersion: v1
kind: ConfigMap
metadata:
  name: validator-config
data:
  config.yaml: |
    version: v1
    flags:
      precompiled: false
      toolkitEnvEnabled: true
    validator:
      env:
        - name: DISABLE_DEV_CHARACTER_CHECK
          value: "false"
      args:
        - --no-driver
        - --no-runtime    

预检流程:
  1. Validator Job 在每个 GPU 节点上执行
  2. 不满足条件 → 节点标记为 gpu.nvidia.com/status=failed
  3. 满足条件 → 节点标记为 gpu.nvidia.com/status=ready
  4. ClusterPolicy 中可设置 validator.operator.cleanupFailed
     自动重试间隔: 5 分钟

1. Pod 请求 nvidia.com/gpu: 1
2. Device Plugin 分配 GPU-0
3. 注入环境变量 NVIDIA_VISIBLE_DEVICES=GPU-xxx
4. 容器内 CUDA 应用独占 GPU-0
5. 容器内申请显存超出限制
6. CUDA API 返回 out of memory
7. 容器级别 OOM Kill（触发重启策略）

# 显存监控告警规则
groups:
  - name: gpu-memory
    rules:
      - alert: GPUOOMImminent
        expr: DCGM_FI_DEV_FB_USED / (DCGM_FI_DEV_FB_FREE + DCGM_FI_DEV_FB_USED) > 0.95
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "GPU XQOPEN $labels.gpu XQCLOSE 显存使用率超过 95%"

传统路径:
  GPU → CPU Memory → NIC → Network
  延迟: 高，CPU 参与拷贝

GPUDirect RDMA:
  GPU → NIC（PCIe P2P）→ Network
  延迟: 低，绕过 CPU

apiVersion: nvidia.com/v1
kind: ClusterPolicy
spec:
  driver:
    enabled: true
    rdma:
      enabled: true
      useHostMofed: false
  gpuDirect:
    enabled: true
    kernelModule:
      name: nvidia-peer-memory

graph LR
    subgraph GPU Operator
        A[Device Plugin] --> A1[nvidia.com/gpu]
        B[GFD] --> B1[节点标签]
        C[MIG Manager]
    end
    subgraph Volcano
        D[Queue<br/>GPU 队列]
        E[PodGroup<br/>Gang Scheduling]
        F[Job<br/>批量任务]
        G[Scheduler<br/>拓扑感知]
    end

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: gpu-training-queue
spec:
  weight: 1
  capability:
    nvidia.com/gpu: 32
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: distributed-training
spec:
  schedulerName: volcano
  queue: gpu-training-queue
  minAvailable: 8
  tasks:
    - replicas: 4
      name: worker
      template:
        spec:
          containers:
            - name: pytorch
              resources:
                limits:
                  nvidia.com/gpu: 8

分布式训练场景:
  4 个 worker，每个需要 8 块 GPU
  minAvailable: 4（4 个 worker 必须同时调度）
  任一 worker 无法调度 → 整体等待
  全部 worker 就绪 → 同时启动

# kubelet 配置
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
topologyManagerPolicy: single-numa-node
featureGates:
  TopologyManager: true

apiVersion: v1
kind: Pod
metadata:
  name: gpu-topology-pod
spec:
  containers:
    - name: training
      resources:
        limits:
          nvidia.com/gpu: 4
          memory: 128Gi
          cpu: 32
        requests:
          nvidia.com/gpu: 4
          memory: 128Gi
          cpu: 32
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule

apiVersion: nvidia.com/v1
kind: ClusterPolicy
metadata:
  name: cluster-policy
spec:
  operator:
    defaultRuntime: containerd
    deployGFD: true
    cleanupState:
      driver: false
      toolkit: true
    validator:
      plugin:
        env:
          - name: DISABLE_DEV_CHARACTER_CHECK
            value: "false"
  driver:
    enabled: true
    version: "535.154.05"
    repoConfig:
      configMapName: driver-repo-config
    licensingConfig:
      configMapName: licensing-config
    virtualTopology:
      config: ""
    env: []
  toolkit:
    enabled: true
    env:
      - name: CONTAINERD_CONFIG
        value: /etc/containerd/config.toml
  devicePlugin:
    enabled: true
    config:
      name: device-plugin-config
      default: default
  mig:
    strategy: mixed
  migManager:
    enabled: true
    config:
      name: mig-manager-config
  gfd:
    enabled: true
  dcgm:
    enabled: true
  dcgmExporter:
    enabled: true
    config:
      name: dcgm-exporter-config
  nodeStatusExporter:
    enabled: false
  gpuFeatureDiscovery:
    enabled: true
  sandboxDevicePlugin:
    enabled: false
  vgpuManager:
    enabled: false
  vgpuDeviceManager:
    enabled: false

kubectl get clusterpolicy cluster-policy -o yaml
# status 字段展示各组件运行状态
# status.state: ready / notReady / ignored

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-driver-daemonset
spec:
  template:
    spec:
      hostPID: true
      containers:
        - name: nvidia-driver
          securityContext:
            privileged: true
          env:
            - name: NVIDIA_DRIVER_VERSION
              value: "535.154.05"
            - name: NVIDIA_DRIVER_SKIP_REBUILD
              value: "false"

# 虚拟化环境下跳过驱动安装
apiVersion: nvidia.com/v1
kind: ClusterPolicy
spec:
  driver:
    enabled: false  # 使用宿主机预装驱动
  validator:
    driver:
      env:
        - name: DISABLE_DEV_CHARACTER_CHECK
          value: "true"

Level 1: MIG 硬件隔离（A100/H100）
  GPU → 多个 GI（GPU Instance）
  每个 GI → 独立的 SM、显存、缓存
  隔离强度: 最高

Level 2: Time-Slicing 时间片隔离
  GPU → 多容器时间片轮转
  隔离强度: 中等

Level 3: vGPU 软件隔离
  GPU → 多 VM 通过 GPU Manager 共享
  隔离强度: 基础

Level 4: CUDA MPS（Multi-Process Service）
  GPU → 单 Context 内多进程并发
  隔离强度: 最低

apiVersion: v1
kind: ConfigMap
metadata:
  name: mig-config
data:
  config.yaml: |
    version: v1
    flags:
      migEnabled: true
    migConfigs:
      gpu-node-1:
        - devices: ["GPU-0", "GPU-1"]
          migEnabled: true
          migDeviceSpecs:
            - devices:
              - parentDevice: "GPU-0"
                migProfile: "2g.20gb"
              - parentDevice: "GPU-1"
                migProfile: "3g.40gb"    

# GPU 基本信息
nvidia-smi

# GPU 拓扑
nvidia-smi topo -m

# GPU 进程
nvidia-smi pmon -c 1

# MIG 状态
nvidia-smi mig -lgi
nvidia-smi mig -lci

# GPU ECC 错误
nvidia-smi -q -d ECC

# GPU 温度与功耗
nvidia-smi -q -d TEMPERATURE,POWER

# 持久化模式
nvidia-smi -pm 1

# 锁定 GPU 频率
nvidia-smi -lgc 1530,1530

1. 核对 nvidia-smi 是否正常输出
2. 检查 /dev/nvidia* 设备文件
3. 核对 nvidia.ko 内核模块加载
4. 检查 nvidia-container-toolkit 配置
5. 验证 Device Plugin 注册
6. 检查容器内 CUDA 可用性

节点扩容:
  1. 新 GPU 插入
  2. 驱动重新扫描 PCIe 总线
  3. nvidia-smi 检测到新 GPU
  4. Device Plugin ListAndWatch 上报新数量
  5. Kubelet 更新节点 allocatable 资源
  6. 新 Pod 调度到节点

节点缩容:
  1. GPU 故障或移除
  2. Device Plugin 健康检测失败
  3. 异常 GPU 从可用列表移除
  4. 该 GPU 上 Pod 被驱逐
  5. Kubelet 更新节点 capacity

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
data:
  config.yaml: |
    version: v1
    flags:
      migStrategy: mixed
      failOnInitError: false
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4    

apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: inference
          resources:
            limits:
              nvidia.com/gpu: 1
      tolerations:
        - key: nvidia.com/gpu.unhealthy
          operator: Exists
          effect: NoExecute
          tolerationSeconds: 30
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0

apiVersion: nvidia.com/v1
kind: ClusterPolicy
spec:
  operator:
    repository: private-registry.local/nvidia
  driver:
    repository: private-registry.local/nvidia
    image: driver
    version: "535.154.05"
  toolkit:
    repository: private-registry.local/nvidia
    image: container-toolkit
  devicePlugin:
    repository: private-registry.local/nvidia
    image: k8s-device-plugin
  dcgmExporter:
    repository: private-registry.local/nvidia
    image: dcgm-exporter
  gfd:
    repository: private-registry.local/nvidia
    image: gpu-feature-discovery
  migManager:
    repository: private-registry.local/nvidia
    image: mig-manager
  validator:
    repository: private-registry.local/nvidia
    image: gpu-operator-validator

apiVersion: v1
kind: Secret
metadata:
  name: nvidia-registry-secret
  namespace: gpu-operator
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded-config>
apiVersion: nvidia.com/v1
kind: ClusterPolicy
spec:
  operator:
    imagePullSecrets:
      - nvidia-registry-secret

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-device-plugin
spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: kubernetes.io/arch
                    operator: In
                    values:
                      - amd64
                      - arm64
      containers:
        - name: nvidia-device-plugin
          image: nvcr.io/nvidia/k8s-device-plugin:v0.14.0

apiVersion: v1
kind: ConfigMap
metadata:
  name: driver-config-arm64
data:
  config.yaml: |
    version: v1
    flags:
      driverVersion: "550.54.14"
      image: "nvcr.io/nvidia/driver:550.54.14-grace-hopper-ubuntu22.04"
      arch: "arm64"    

高可用配置:
  - Operator Deployment replicas ≥ 2
  - Device Plugin DaemonSet 每个 GPU 节点一个实例
  - DCGM Exporter 作为 DaemonSet 部署
  - MIG Manager 按需启用，灰度切换 MIG 配置

资源规划:
  - Operator: CPU 200m, Memory 256Mi
  - Device Plugin: CPU 100m, Memory 128Mi
  - DCGM Exporter: CPU 200m, Memory 512Mi
  - Driver DaemonSet: CPU 500m, Memory 1Gi

1. 驱动升级遵循分批灰度策略（10%→30%→100%）
2. GPU 节点维护前执行 drain 操作
3. 每季度审查 GPU 资源利用率报告
4. MIG 配置变更需在维护窗口内执行
5. 保持 GPU Operator 版本与驱动版本联动
6. 定期演练 GPU 故障场景下的 Pod 迁移
7. nvidia-smi 健康检查纳入巡检脚本
8. GPU 节点标签规范化管理
9. GPU 资源配额与 LimitRange 覆盖所有租户
10. GPU 审计日志集中收集