Volcano 面试题

graph TD
    KC["kubectl / vcctl"] --> ADM["Admission (Webhook)<br/>- Job 验证<br/>- 默认注入"]
    KC --> SCH["Scheduler"]
    ADM -->|"校验与注入"| SCH
    SCH --> SES["Sessions"]
    SES --> PLG["Plugins"]
    SES --> ACT["Actions"]
    SCH --> CTRL["Controller<br/>- Job<br/>- PodGroup<br/>- Queue"]
    SCH --- Q["Queue (CRD)"]

    style SCH fill:#e1f5fe
    style SES fill:#e1f5fe
    style PLG fill:#e1f5fe
    style ACT fill:#e1f5fe

graph TD
    A["提交 Job（minAvailable: 8）"] --> B["Scheduler 检查集群 GPU 可用量"]
    B --> C{"GPU 可用 >= 8 ?"}
    C -->|否| D["Job 进入 Pending 状态<br/>所有 8 个 Pod 全部等待"]
    C -->|是| E["一次分配 8 个 GPU 槽位<br/>8 个 Pod 同时启动<br/>进入 Running 状态"]

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: pytorch-training-pg
spec:
  minMember: 8
  minResources:
    nvidia.com/gpu: "8"
    cpu: "64"
    memory: "256Gi"
  queue: ai-training
  priorityClassName: high-priority

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: ai-training
spec:
  weight: 3
  capability:
    cpu: "200"
    memory: "1024Gi"
    nvidia.com/gpu: "32"
  reclaimable: true
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: inference
spec:
  weight: 1
  capability:
    cpu: "50"
    memory: "256Gi"
    nvidia.com/gpu: "8"
  reclaimable: false

graph LR
    Pending --> Inqueue --> Running --> Completed
    Running --> Unknown --> Failed

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
  name: llm-fine-tuning
spec:
  minMember: 4
  minResources:
    nvidia.com/gpu: "16"
    cpu: "128"
    memory: "512Gi"
  minTaskMember:
    master: 1
    worker: 3
  queue: ai-training
  priorityClassName: high-priority

graph TD
    Pending --> Running --> Completed
    Running --> Failed --> Restarting --> Running
    Running --> Terminating --> Terminated
    Running --> Aborting --> Aborted
    Running --> Completing --> Completed

Volcano Job
  │
  ├── PodGroup（自动创建）
  │   └── minMember / minResources
  │
  ├── Tasks[]（多任务类型）
  │   ├── master
  │   │   ├── replicas: 1
  │   │   └── template: PodSpec
  │   │
  │   └── worker
  │       ├── replicas: 8
  │       └── template: PodSpec
  │
  ├── Policies（策略）
  │   ├── Event: PodRunning / PodFailed /…
  │   └── Action: RestartJob / AbortJob / CompleteJob
  │
  └── Plugins（扩展）
      ├── svc / ssh / env / tensorflow
      └── 自定义插件（生成 ConfigMap、Service 等）

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: pytorch-dist-training
spec:
  schedulerName: volcano
  queue: ai-training
  minAvailable: 4
  tasks:
    - replicas: 1
      name: master
      template:
        spec:
          containers:
            - name: master
              image: pytorch/pytorch:2.0.0-cuda11.8
              command: ["torchrun", "--nproc_per_node=1", "--nnodes=4", "train.py"]
              resources:
                limits:
                  nvidia.com/gpu: 4
    - replicas: 3
      name: worker
      template:
        spec:
          containers:
            - name: worker
              image: pytorch/pytorch:2.0.0-cuda11.8
              command: ["torchrun", "--nproc_per_node=1", "--nnodes=4", "train.py"]
              resources:
                limits:
                  nvidia.com/gpu: 4
  policies:
    - event: PodEvicted
      action: RestartJob
    - event: PodFailed
      action: RestartJob

Fair-Share 计算示例

三个 Queue:
  Queue-A  weight: 3, 已分配: 30 GPU
  Queue-B  weight: 2, 已分配: 10 GPU
  Queue-C  weight: 1, 已分配: 5 GPU

集群总 GPU: 64

计算过程:
  总权重: 3 + 2 + 1 = 6
  Queue-A 应得: 64 × 3/6 = 32 GPU, 份额: 30/32 = 0.9375
  Queue-B 应得: 64 × 2/6 = 21 GPU, 份额: 10/21 = 0.476
  Queue-C 应得: 64 × 1/6 = 11 GPU, 份额: 5/11  = 0.455

  份额最低的 Queue-B、Queue-C 优先调度

Binpack 节点打分:

  节点 A: CPU 已用 80%, GPU 已用 75%  ← 高分（利用率高）
  节点 B: CPU 已用 30%, GPU 已用 25%  ← 低分（利用率低）
  节点 C: CPU 已用 50%, GPU 已用 100% ← 低分（GPU 已满，不可调度）

  调度结果: Pod 优先落到节点 A

apiVersion: v1
kind: ConfigMap
metadata:
  name: volcano-scheduler-config
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
        - name: priority
        - name: gang
        - name: conformance
    - plugins:
        - name: binpack
          arguments:
            binpack.weight: 10
            binpack.cpu: "1"
            binpack.memory: "1"
            binpack.nvidia.com/gpu: "2"    

DRF 计算示例:

  Job-A: 需要 {CPU: 4, GPU: 8}
    ├── CPU 占比:  4/100 = 4%
    └── GPU 占比:  8/32  = 25% ← 主导资源

  Job-B: 需要 {CPU: 20, GPU: 2}
    ├── CPU 占比:  20/100 = 20% ← 主导资源
    └── GPU 占比:  2/32  = 6.25%

  调度决策:
    Job-A 的 GPU 份额 (25%) > Job-B 的 CPU 份额 (20%)
    → Job-B 优先调度（主导资源份额更低）

data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
        - name: drf
          arguments:
            drf.weight: 5    

graph TD
    subgraph NUMA0["NUMA Node 0"]
        CPU0["CPU 0-31"]
        G0["GPU 0"] ---|"NVLink"| G1["GPU 1"]
        MEM0["Memory: 256GB"]
    end
    subgraph NUMA1["NUMA Node 1"]
        CPU1["CPU 32-63"]
        G2["GPU 2"] ---|"NVLink"| G3["GPU 3"]
        MEM1["Memory: 256GB"]
    end
    NOTE["调度决策: 同一 Job 的 Worker 优先调度到同一 NUMA Node<br/>跨 NUMA 时通过 NVSwitch 或 PCIe Switch 保证带宽"]

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: gpu-dist-training
spec:
  schedulerName: volcano
  tasks:
    - replicas: 8
      name: worker
      template:
        spec:
          containers:
            - name: training
              resources:
                limits:
                  nvidia.com/gpu: 8
              env:
                - name: NCCL_SOCKET_IFNAME
                  value: "eth0"
                - name: NCCL_DEBUG
                  value: "INFO"
      topologyPolicy: restricted

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: tf-dist-training
spec:
  schedulerName: volcano
  minAvailable: 5
  tasks:
    - replicas: 1
      name: ps
      template:
        spec:
          containers:
            - name: parameter-server
              resources:
                limits:
                  cpu: "4"
                  memory: "16Gi"
    - replicas: 4
      name: worker
      template:
        spec:
          containers:
            - name: worker
              resources:
                limits:
                  nvidia.com/gpu: 4
      taskTopology:
        affinity:
          pod:
            - task: ps
              topologyKey: kubernetes.io/hostname
              weight: 100

graph LR
    A["数据预处理"] --> B["模型训练"] --> C["模型评估"]
    B --> D["模型导出"]

apiVersion: flow.volcano.sh/v1alpha1
kind: JobFlow
metadata:
  name: ml-pipeline
spec:
  flows:
    - name: data-preprocess
      jobTemplate:
        name: preprocess-job
    - name: model-training
      dependsOn:
        targets:
          - data-preprocess
      jobTemplate:
        name: training-job
    - name: model-evaluation
      dependsOn:
        targets:
          - model-training
      jobTemplate:
        name: eval-job
    - name: model-export
      dependsOn:
        targets:
          - model-training
      jobTemplate:
        name: export-job

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: gpu-training
spec:
  schedulerName: volcano
  queue: gpu-queue
  minAvailable: 4
  tasks:
    - replicas: 4
      name: worker
      template:
        spec:
          containers:
            - name: training
              image: pytorch/pytorch:2.0.0-cuda11.8-cudnn8-devel
              command:
                - torchrun
                - --nproc_per_node=8
                - --nnodes=4
                - --node_rank=$(VC_TASK_INDEX)
                - --master_addr=$(VC_MASTER_HOST)
                - --master_port=29500
                - train.py
              resources:
                limits:
                  nvidia.com/gpu: 8
                  cpu: "64"
                  memory: "256Gi"
              env:
                - name: NCCL_SOCKET_IFNAME
                  value: "eth0"
                - name: NCCL_IB_DISABLE
                  value: "0"
                - name: NCCL_NET_GDR_LEVEL
                  value: "5"
              volumeMounts:
                - name: dataset
                  mountPath: /data
          volumes:
            - name: dataset
              persistentVolumeClaim:
                claimName: training-dataset

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: pytorch-ddp
spec:
  schedulerName: volcano
  plugins:
    pytorch: ["--master=master", "--worker=worker"]  # Task 角色映射
  minAvailable: 4
  tasks:
    - replicas: 1
      name: master
      template:
        spec:
          containers:
            - name: pytorch
              image: pytorch/pytorch:2.0.0
              command: ["torchrun", "--nnodes=4", "train.py"]
    - replicas: 3
      name: worker
      template:
        spec:
          containers:
            - name: pytorch
              image: pytorch/pytorch:2.0.0
              command: ["torchrun", "--nnodes=4", "train.py"]

plugins:
  mpi: ["--master=master", "--worker=worker"]
  ssh: []
  svc: []

Reservation 流程

  1. Session 开始，Scheduler 收集集群资源快照
  2. 遍历 Queue 内 Pending PodGroup
  3. 为满足 minMember 的 PodGroup 预留资源
  4. 所有 Pod 分配完毕后，统一标记为 Allocated
  5. Session 结束，Controller 绑定 Pod 到 Node

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: resv-example
spec:
  schedulerName: volcano
  minAvailable: 8  # 8 个 Pod 必须全部预留成功才分配
  tasks:
    - replicas: 8
      name: worker
      template:
        spec:
          containers:
            - name: worker
              resources:
                limits:
                  nvidia.com/gpu: 1

graph TD
    A["高优 Job<br/>Priority: 100<br/>Pending"] --> B["Scheduler 检查<br/>集群可用资源"]
    B --> C{"资源充足？"}
    C -->|"充足"| D["直接分配"]
    C -->|"不足"| E["触发 Preempt Action"]
    E --> F["选择牺牲者 Victim"]
    F --> F1["低优先级队列"]
    F --> F2["同队列低优先级 Job"]
    F --> F3["reclaimable=true 的队列"]
    F1 & F2 & F3 --> G["满足 minMember<br/>后停止选择"]
    G --> H["驱逐牺牲者 Job<br/>释放资源分配给高优 Job"]

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000
globalDefault: false
description: "高优先级 AI 训练任务"
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: critical-training
spec:
  schedulerName: volcano
  queue: ai-training
  minAvailable: 4
  priorityClassName: high-priority
  tasks:
    - replicas: 4
      name: worker
      template:
        spec:
          priorityClassName: high-priority

Backfill 示例

  队列中有 Job-A (minMember: 8, 需 8 GPU)
  集群当前空闲: 6 GPU
  队列中还有 Job-B (minMember: 1, 需 1 GPU)

  无 Backfill:
    Job-A 等 8 GPU, 6 GPU 空闲浪费
    Job-B 排在 Job-A 后面，也被阻塞

  有 Backfill:
    Job-A 不能调度（不足 8 GPU）
    Backfill 发现 Job-B 只需要 1 GPU
    Job-B 被调度到空闲 GPU 上
    6 GPU 中有 1 GPU 被利用

data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"    

data:
  volcano-scheduler.conf: |
    tiers:
    - plugins:
        - name: priority
        - name: gang
    - plugins:
        - name: nodeorder
          arguments:
            plugin.weight: 10
            plugin.least-task.weight: 5
            plugin.binpack.weight: 8
            plugin.binpack.nvidia.com/gpu: "2"    

graph TD
    A["高优 Queue-A<br/>资源不足"] --> B["Scheduler 扫描<br/>reclaimable=true 的 Queue"]
    B --> C{"可回收？"}
    C -->|"Queue-B<br/>reclaimable=true<br/>有空闲资源"| D["驱逐 Queue-B 中<br/>低优先级 Pod"]
    D --> E["资源分配给 Queue-A"]
    C -->|"Queue-C<br/>reclaimable=false"| F["资源不可回收<br/>跳过"]

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: training-with-ttl
spec:
  schedulerName: volcano
  minAvailable: 4
  ttlSecondsAfterFinished: 3600  # 完成后 1 小时自动清理
  maxRetry: 3                      # 最大重试次数
  tasks:
    - replicas: 4
      name: worker
      template:
        spec:
          containers:
            - name: training
              image: pytorch/pytorch:2.0.0

graph TD
    subgraph GPUOP["GPU Operator"]
        DRV["Driver"]
        DP["Device Plugin"]
        GFD["GFD"]
        DCGM["DCGM Exporter"]
    end

    DP -->|"nvidia.com/gpu"| NODE["节点标签 + 资源注册"]
    GPUOP --> NODE

    subgraph VOL["Volcano Scheduler"]
        Q["Queue"]
        PG["PodGroup"]
        BP["Binpack"]
        TOPO["Topology"]
    end

    NODE --> VOL

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
spec:
  queue: gpu-training
  tasks:
    - replicas: 4
      template:
        spec:
          nodeSelector:
            nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB  # GFD 注入的标签
          containers:
            - name: training
              resources:
                limits:
                  nvidia.com/gpu: 8

graph TD
    A["Job 请求<br/>nvidia.com/gpumem: 10 GB<br/>nvidia.com/gpucores: 20"] --> B["Volcano Scheduler<br/>检查节点可用 GPU 资源"]
    B --> C["通过 HAMI 资源维度<br/>找到满足条件的节点<br/>HAMI Device Plugin 注册<br/>gpumem、gpucores"]
    C --> D["Volcano 执行<br/>Gang Scheduling"]
    D --> E["Pod 分配到指定 GPU<br/>的共享槽位"]
    E --> E1["HAMI Core<br/>绑定 CUDA Context"]
    E --> E2["HAMI Memory<br/>限制 vRAM 上限"]

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: shared-gpu-inference
spec:
  schedulerName: volcano
  queue: inference-queue
  minAvailable: 2
  tasks:
    - replicas: 2
      name: inference
      template:
        spec:
          containers:
            - name: inference
              image: vllm/vllm-openai:latest
              resources:
                limits:
                  nvidia.com/gpumem: 10  # 申请 10GB vRAM
                  nvidia.com/gpucores: 20 # 申请 20% 算力

graph TD
    subgraph KARMADA["Karmada Control Plane"]
        KS["Scheduler"]
        RD["Resource Detector"]
    end

    KS --> CA["Cluster A<br/>Volcano<br/>GPU: 64"]
    RD --> CA
    KS --> CB["Cluster B<br/>Volcano<br/>GPU: 128"]
    RD --> CB

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: volcano-job-policy
spec:
  resourceSelectors:
    - apiVersion: batch.volcano.sh/v1alpha1
      kind: Job
  placement:
    clusterAffinity:
      clusterNames:
        - cluster-a
        - cluster-b
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames: [cluster-a]
            weight: 1
          - targetCluster:
              clusterNames: [cluster-b]
            weight: 2

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: volcano
  namespace: volcano-system
spec:
  selector:
    matchLabels:
      app: volcano-scheduler
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

groups:
  - name: volcano-alerts
    rules:
      - alert: VolcanoJobPendingTooLong
        expr: volcano_job_pending_duration_seconds > 1800
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Job XQOPEN $labels.job_name XQCLOSE Pending 超过 30 分钟"

      - alert: VolcanoQueueGPUSaturation
        expr: volcano_queue_allocated{nvidia_com_gpu != ""} / volcano_queue_capability{nvidia_com_gpu != ""} > 0.95
        for: 10m
        labels:
          severity: warning

层次 1：Job 状态检查
  kubectl get job -n <namespace> <job-name> -o yaml
  kubectl describe podgroup -n <namespace> <podgroup-name>

  关注: status.state, status.conditions, events

层次 2：调度器日志分析
  kubectl logs -n volcano-system deploy/volcano-scheduler | grep <job-id>
  # 关键日志前缀:
  #   "Session Open" - 调度周期开始
  #   "PodGroup <name> is inqueue" - 入队
  #   "PodGroup <name> is pending reason: insufficient resources" - 资源不足
  #   "Allocate task <name> to node <node>" - 分配成功

层次 3：调度器配置诊断
  kubectl get configmap volcano-scheduler-config -n volcano-system -o yaml
  # 核对: actions 链路、plugin 参数、queue 配置

弹性配额模型

  Queue-A:
    guarantee:   {cpu: 20,  nvidia.com/gpu: 8}
    capability:  {cpu: 100, nvidia.com/gpu: 64}
    reclaimable: true

  集群实际分配:
    保底（guarantee）: 8 GPU   ← 永远保留
    弹性（capability - guarantee）: 最多 56 GPU  ← 可被回收
    超出 capability: 不允许

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: elastic-ai-training
spec:
  weight: 2
  guarantee:
    nvidia.com/gpu: "16"
    cpu: "64"
    memory: "256Gi"
  capability:
    nvidia.com/gpu: "64"
    cpu: "256"
    memory: "1024Gi"
  reclaimable: true
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: burst-inference
spec:
  weight: 1
  guarantee:
    nvidia.com/gpu: "4"
  capability:
    nvidia.com/gpu: "32"
  reclaimable: false  # 独占弹性区间的资源，不被回收

SLO 约束类型:

  Deadline SLO:
    Job 必须在指定时间前完成
    调度器优先为临近 Deadline 的 Job 分配资源

  Throughput SLO:
    Job 需维持最小吞吐量
    调度器为低于吞吐目标的 Job 增加资源倾斜

  Availability SLO:
    多副本 Job 需保持 N 个副本可用
    调度器在副本不足时触发快速恢复调度

apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
  name: slo-training
  annotations:
    scheduling.volcano.sh/slo-deadline: "2025-06-01T00:00:00Z"
spec:
  schedulerName: volcano
  queue: ai-training
  minAvailable: 4
  tasks:
    - replicas: 4
      name: worker
      template:
        spec:
          containers:
            - name: training
              resources:
                limits:
                  nvidia.com/gpu: 8

SLO 紧急度 = 剩余工作量 / (Deadline - 当前时间)

数值越高 → 紧急度越高 → 调度优先级越高

当多个 Queue 竞争资源时，SLO-Aware 权重覆盖 Fair-Share：
  priority_score = fair_share_score × (1 - SLO_slack_factor)

# Volcano Helm 安装
helm repo add volcano-sh https://volcano-sh.github.io/helm-charts
helm install volcano volcano-sh/volcano \
  --namespace volcano-system \
  --create-namespace \
  --set basic.schedulerConfigFile="volcano-scheduler.conf" \
  --set controller.replicas=2 \
  --set scheduler.replicas=2 \
  --set admission.replicas=2

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app: volcano-scheduler
        topologyKey: kubernetes.io/hostname

data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
        - name: priority
        - name: gang
        - name: conformance
    - plugins:
        - name: binpack
          arguments:
            binpack.weight: 10    

apiVersion: v1
kind: ConfigMap
metadata:
  name: volcano-scheduler-config
data:
  volcano-scheduler-kubeconfig: |
    apiVersion: kubescheduler.config.k8s.io/v1beta3
    kind: KubeSchedulerConfiguration
    percentageOfNodesToScore: 30    

# 查看调度器性能指标
kubectl logs -n volcano-system deploy/volcano-scheduler | grep "Session.*duration"

# Session 耗时过高的标志
# "Session Open" 到 "Session Close" 超过 2s 时需关注

Volcano Job（旧版）:
  spec.tasks[].template.spec.containers[].resources.limits

VCJob（新版）:
  spec.tasks[].resources.limits  ← 扁平化，字段更少

apiVersion: batch.volcano.sh/v1alpha1
kind: VCJob
metadata:
  name: advanced-vcjob
spec:
  schedulerName: volcano
  queue: ai-training
  minAvailable: 4
  runPolicy:
    maxRetry: 3
    ttlSecondsAfterFinished: 7200
    cleanUpTaskPolicy:
      - action: Delete
        task: all
        when: Completed
  plugins:
    svc: []
    env: []
    pytorch: ["--master=master", "--worker=worker"]
  tasks:
    - replicas: 1
      name: master
      restartPolicy: OnFailure
      resources:
        limits:
          nvidia.com/gpu: 8
          cpu: "64"
          memory: "256Gi"
      lifecycle:
        postStart:
          exec:
            command: ["/bin/sh", "-c", "nvidia-smi && echo 'GPU ready'"]
        preStop:
          exec:
            command: ["/bin/sh", "-c", "kill -SIGTERM $PID"]
      pod:
        image: pytorch/pytorch:2.0.0-cuda11.8
        command: ["torchrun", "--nnodes=4", "train.py"]
        volumes:
          - name: shm
            emptyDir:
              medium: Memory
              sizeLimit: 32Gi

集成架构

  Kubeflow
  ├── Pipelines → Volcano JobFlow（工作流 DAG）
  ├── Training Operator → Volcano Job（Gang Scheduling）
  │   ├── PyTorchJob → Volcano PyTorch Plugin
  │   ├── TFJob → Volcano TensorFlow Plugin
  │   └── MPIJob → Volcano MPI Plugin
  ├── Katib（超参搜索） → Volcano Queue（资源隔离）
  └── Notebooks → Volcano Queue + Elastic Quota

apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
  name: pytorch-dist
spec:
  pytorchReplicaSpecs:
    Master:
      replicas: 1
      restartPolicy: OnFailure
      template:
        spec:
          schedulerName: volcano  # 指定 Volcano 调度器
          containers:
            - name: pytorch
              image: pytorch/pytorch:2.0.0
              resources:
                limits:
                  nvidia.com/gpu: 8
    Worker:
      replicas: 3
      restartPolicy: OnFailure
      template:
        spec:
          schedulerName: volcano
          containers:
            - name: pytorch
              image: pytorch/pytorch:2.0.0
              resources:
                limits:
                  nvidia.com/gpu: 8
  runPolicy:
    schedulingPolicy:
      minAvailable: 4          # Volcano Gang Scheduling
      queue: ai-training        # Volcano Queue
      priorityClass: high-priority

from kfp import dsl

@dsl.pipeline(name="ml-training-pipeline")
def training_pipeline():
    preprocess = dsl.implement(
        volcano_jobflow_component(
            job_template="preprocess-job",
            queue="ai-training"
        )
    )

    train = dsl.implement(
        volcano_jobflow_component(
            job_template="training-job",
            queue="ai-training"
        )
    )

    evaluate = dsl.implement(
        volcano_jobflow_component(
            job_template="eval-job",
            queue="ai-training"
        )
    )

    train.after(preprocess)
    evaluate.after(train)