Manifesto

Manifesto renders shareable Kubernetes manifests and Slurm batch scripts for LLM deployments.

It is built for the workflow where an engineer wants to describe a concrete vLLM deployment, render plain YAML, inspect or hand-edit it, and then apply exactly that artifact to a cluster.

Manifesto keeps intelligence in resolution, not in the artifact: model specs express intent, feature contracts supply required runtime details, and the result remains ordinary, minimal Kubernetes YAML.

The project is packaged as llm-manifesto, with manifesto as the Python package and CLI command name.

What It Does

Manifesto takes two YAML inputs:

• a model spec: model image, topology, roles, parallelism, routing, vLLM args
• a cluster profile: GPU shape, storage paths, fabric env, llm-d images

It emits raw Kubernetes manifests:

• Deployment or LeaderWorkerSet model-server workloads, depending on node count
• InferencePool and endpoint picker deployment
• standalone Envoy routing by default, or optional Gateway API objects
• per-pod monitoring sidecars
• instance-scoped names, labels, selectors, and cache paths

For Slurm profiles, it emits self-contained sbatch scripts for direct vLLM, llm-d load-aware routing, and prefill/decode disaggregation. See Slurm for runtime choices, remote submission, and supported topologies.

The rendered YAML starts with provenance comments:

# Generated by:
#   uvx --from git+https://github.com/neuralmagic/llm-manifesto@<commit> manifesto render manifest \
#     <(cat <<'MANIFESTO_MODEL'
# <expanded model YAML>
# MANIFESTO_MODEL
#     ) \
#     --cluster <(cat <<'MANIFESTO_CLUSTER'
# <expanded cluster YAML>
# MANIFESTO_CLUSTER
#     )
# To set identity or placement, append --user USER and/or --namespace NAMESPACE.
# Source: https://github.com/neuralmagic/llm-manifesto/tree/<commit>
# Safe to edit before applying.
---
apiVersion: v1
kind: ServiceAccount
...

That header is intentional: rendered manifests should be easy to paste into a PR, send to another engineer, edit manually, and regenerate later.

License

Licensed under the Apache License, Version 2.0. See LICENSE.

Quick Start

Bootstrap namespace prerequisites declared by the cluster profile (such as a shared cache PVC):

manifesto render bootstrap --cluster clusters/example-gb200.yaml --namespace "$MANIFESTO_NAMESPACE"
manifesto deploy bootstrap --cluster clusters/example-gb200.yaml --namespace "$MANIFESTO_NAMESPACE"

Render a manifest:

uv run manifesto render manifest models/qwen/aggregated.yaml \
  --cluster clusters/example-gb200.yaml \
  --user "$USER"

Render to a file, edit it, diff it, then apply it:

manifesto render file models/deepseek-v4/1P-EP8-1D-EP8.yaml
$EDITOR /tmp/manifesto.yaml
manifesto file diff
manifesto file apply

Deploy directly:

manifesto deploy models/deepseek-v4/1P-EP8-1D-EP8.yaml
manifesto ready models/deepseek-v4/1P-EP8-1D-EP8.yaml

Stop the instance:

manifesto servers
manifesto stop                         # interactive picker
manifesto stop models/deepseek-v4/1P-EP8-1D-EP8.yaml
manifesto stop --instance "$USER-wide-ep-1p-ep8-1d-ep8"
manifesto stop models/deepseek-v4/1P-EP8-1D-EP8.yaml --now

Repository Layout

manifesto/                  Python renderer implementation
  cli.py                CLI entrypoints
  spec.py               model/deployment spec schema
  cluster.py            cluster profile schema
  instance.py           instance-scoped names, labels, selectors, paths
  resolve.py            concrete env, paths, ports, and vLLM args per role
  render/               Kubernetes object emitters

clusters/               cluster profiles
models/                 model deployment specs
tests/                  renderer and UX tests

User Configuration

Manifesto also reads a private user catalog from ~/.config/llm-manifesto (or $XDG_CONFIG_HOME/llm-manifesto). Keep local cluster and model configuration outside the repository with this layout:

~/.config/llm-manifesto/
  .env                         optional local defaults
  clusters/
    my-context.yaml
  models/
    model_provider/model.yaml
  routing/
    my-epp-profile.yaml

Catalog entries can be referenced by name, with the .yaml suffix optional:

manifesto render manifest model_provider/model --cluster my-context

If a file under clusters/ is named for the current kube context or kube cluster, render and deployment commands select it automatically, so --cluster can be omitted. Explicit paths and the existing MANIFESTO_CLUSTER and MANIFESTO_CLUSTER_MAP settings remain supported. Set MANIFESTO_CONFIG_HOME to override the entire user configuration path. User catalog entries take precedence over bundled entries with the same name.

Use the config commands to discover that effective catalog and validate a model/cluster pairing before rendering or deploying:

manifesto config home
manifesto config list models
manifesto config list clusters
manifesto config list routing
manifesto config resolve models deepseek-v4/1P-EP8-1D-EP8
manifesto config resolve routing wide-ep-lws-config
manifesto config edit models deepseek-v4/1P-EP8-1D-EP8
manifesto config export models deepseek-v4/1P-EP8-1D-EP8 -o model.yaml
manifesto config import models model.yaml --name experiments/model
manifesto config validate deepseek-v4/1P-EP8-1D-EP8 --cluster example-gb200

config list reports whether each winning entry is from the user or bundled catalog. Use --output name for shell scripts or --output json to also see when a user entry shadows a bundled entry. config validate loads both schemas, applies cluster defaults, and exercises the full renderer without contacting or changing the cluster.

config edit opens $EDITOR (or vi) on the user-catalog copy. Editing a bundled model first copies it and any relative extends parents into the user catalog, so the bundled source remains untouched. A new name creates a new user config. The edited file is schema-validated when the editor exits; an invalid file is retained so it can be fixed with the same command.

config export writes a self-contained YAML document with model inheritance flattened. config import validates and installs that portable form in the user catalog. Both refuse to overwrite files unless --force is passed.

Specs

Model specs live under models/.

Examples:

models/qwen/aggregated.yaml
models/deepseek-v4/1P-EP8-1D-EP8.yaml
models/deepseek-v4/3P-EP8-1D-EP16.yaml

A spec chooses the topology, roles, parallelism, vLLM args, and routing behavior:

release: wide-ep
topology: pd
accelerator: gb200

model:
  id: deepseek-ai/DeepSeek-V4-Pro
  image_ref: vllm.standard

vars:
  max_concurrency: 1024
  mtp_size: 1

roles:
  - name: decode
    workload_name: vllm-ep8-decode
    lws: {size: 4}
    parallelism: {tp: 1, pp: 1, dp: 16, ep: true}
    computed:
      env:
        MAX_TOKENS: max_concurrency
      vllm:
        max_num_batched_tokens: max_concurrency * mtp_size
        max_num_seqs: max_concurrency
        max_cudagraph_capture_size: max_concurrency

tp, pp, and dp are global tensor-, pipeline-, and data-parallel sizes. Each engine replica consumes tp × pp GPUs. Local model/DP groups, port fanout, and per-pod launch arguments are derived from the workload size and GPUs per pod. GPUs per pod is derived as tp × pp × dp ÷ lws.size, treating disabled DP as one, and must fit the selected accelerator profile's gpus_per_node capacity. Single-node roles render as Kubernetes Deployments; roles spanning multiple nodes render as LeaderWorkerSets. Set a role's workload to deployment or leaderworkerset to override that default. An explicit one-node LeaderWorkerSet can be useful when an admission controller integrates with LeaderWorkerSet rather than Deployment. Multi-node roles cannot select Deployment.

For example, this runs one TP2 × PP2 engine across four GPUs. Increase lws.size and divide those four GPUs evenly across pods to span nodes; Manifesto renders the native vLLM node-rank and headless-worker flags.

roles:
  - name: decode
    lws: {size: 1}
    parallelism: {tp: 2, pp: 2, dp: false}

Manifesto derives where DP ranks sit and which nodes host HTTP endpoints. Each DP rank uses tp × pp GPUs and either fits on one node or spans whole nodes. With internal DP load balancing, node 0 hosts the HTTP endpoint. With external DP load balancing, each DP rank exposes an endpoint on its first node. Launch scripts, readiness checks, routing, and idle shutdown use this same endpoint placement.

Configure PP only through parallelism.pp. Manifesto rejects pipeline_parallel_size in vllm:, computed vLLM arguments, or vllm_raw_args so GPU allocation and the vLLM worker topology cannot disagree.

Unknown role keys are rejected at load time, so typos fail loudly instead of being silently ignored.

Keep EPP scheduling policy in the routing catalog, separate from model specs. A routing profile is an EndpointPickerConfig, for example routing/kv-aware.yaml:

apiVersion: llm-d.ai/v1alpha1
kind: EndpointPickerConfig
plugins:
  - {type: approx-prefix-cache-producer, name: gpu-prefix-cache-producer}
  - type: prefix-cache-scorer
    parameters: {prefixMatchInfoProducerName: gpu-prefix-cache-producer}
schedulingProfiles:
  - name: default
    plugins:
      - {pluginRef: prefix-cache-scorer, weight: 3}

Choose the profile when rendering or deploying; the model remains independent of the routing policy:

manifesto render manifest deepseek-v4/1P-EP8-1D-EP8 \
  --routing-profile wide-ep-lws-config
manifesto deploy deepseek-v4/1P-EP8-1D-EP8 \
  --routing-profile wide-ep-lws-config

Profiles resolve from the user catalog first and then bundled routing/ files. Manifesto embeds the resolved profile in the rendered EPP ConfigMap, so the Kubernetes artifact remains self-contained. Set MANIFESTO_ROUTING_PROFILE to choose a default without repeating the flag. Advanced specs can still use plugin_configs with plugins_config_file to bundle and select multiple files; existing routing.plugin_config specs continue to render as plugins.yaml.

Routed specs use a standalone Envoy sidecar in the EPP pod by default. The EPP Service exposes port 80, while the InferencePool remains the source of model endpoints and rank ports. Select the Gateway API frontend explicitly when shared Gateway infrastructure is required:

routing:
  kind: pd
  target_role: decode
  frontend: gateway  # standalone (default) or gateway

manifesto deploy removes resources left by the previous frontend after a successful apply. The low-level manifesto file apply command intentionally applies only the saved file and cannot prune resources omitted from it; use manifesto stop before that workflow when changing frontends.

workload_name is optional. When set, it controls the Deployment or LeaderWorkerSet name and therefore the Kubernetes pod name prefix, while full release-specific instance labels still scope routing and selectors.

Image versions are centralized in config/images.yaml. Model specs should use model.image_ref, with vllm.standard as the shared vLLM image reference, unless a spec needs an explicit one-off model.image.

Set model.revision to pin the model repository to a branch, tag, or commit. Manifesto renders it as vLLM's --revision argument for every role; it is not accepted in a role's vllm:, computed vLLM arguments, or vllm_raw_args.

Each cluster profile declares its available accelerator profiles and a default. Specs inherit the selected cluster's default unless they set accelerator. The selected entry controls accelerator allocation, per-node GPU capacity, accelerator-specific cache paths, and the development build architecture.

For model authors, accelerator allocation is cluster-owned: model specs keep the same role GPU counts whether the cluster uses extended resources or Dynamic Resource Allocation (DRA). Only cluster operators select or change the backend.

An extended-resource profile declares that backend explicitly:

accelerators:
  default: b200
  profiles:
    b200:
      gpus_per_node: 8
      allocation:
        extended_resource:
          resource_name: nvidia.com/gpu
      presence_label: nvidia.com/gpu.present
      gpu_arch: b200
      torch_cuda_arch_list: "10.0+PTX"

To use DRA instead, the cluster operator changes only allocation:

    b200:
      gpus_per_node: 8
      allocation:
        dra:
          device_class_name: gpu.nvidia.com
      presence_label: nvidia.com/gpu.present
      gpu_arch: b200
      torch_cuda_arch_list: "10.0+PTX"

allocation must contain exactly one of extended_resource or dra.

DRA operator requirements

A DRA profile renders one resource.k8s.io/v1 ResourceClaimTemplate for each GPU-bearing role. Each template requests the role's GPU count with ExactCount, and only the model container receives the resulting claim. CPU, memory, ephemeral storage, and RDMA remain ordinary container resources. Template names include a digest of the complete immutable claim specification, so a release can safely reuse a stable workload name when its claim metadata changes.

The cluster must serve resource.k8s.io/v1 and provide the configured DeviceClass. When Kueue is enabled, configure its DRA integration and a deviceClassMappings entry for that class, then give the mapped logical resource ClusterQueue quota. Manifesto verifies the DRA API and DeviceClass before deployment; Kueue remains authoritative for mapping and quota accounting.

Manifesto applies shared vLLM arguments before role-specific vllm: and computed arguments. By default, Uvicorn access logs omit the high-frequency /health, /v1/models, and /metrics endpoints while preserving inference request logs. A role can replace the excluded endpoint list with vllm.disable_access_log_for_endpoints, or set it to null to omit the Manifesto default entirely.

Feature contracts

Role settings activate typed serving features before rendering. Data, pipeline, and expert parallelism, P/D, and llm-d are features. Connector and all-to-all implementations are reported separately as backends; workload kind and platform resources are derived consequences. For example:

• routing.kind: disabled selects direct vLLM, while load_aware or pd selects llm-d.
• data parallelism uses external DP when llm-d is selected and internal DP for direct vLLM.
• P/D implies llm-d and a routing proxy for now.
• a NIXL connector backend supplies VLLM_NIXL_SIDE_CHANNEL_HOST from the Pod IP unless the role provides an explicit value; other connector backends are preserved without being treated as NIXL.
• a parallel shape spanning multiple nodes selects a LeaderWorkerSet; single-node shapes select a Deployment.
• DeepEP is reported as the selected all-to-all backend for EP, and an IMEX claim is derived from the cluster platform capability when configured.

Dependencies are resolved transitively and the registry rejects implication cycles. They do not add framework annotations or dependency metadata to the rendered objects. Inspect the internal resolution separately:

manifesto explain models/deepseek-v4/1P-EP8-1D-EP8.yaml \
  --cluster clusters/example-gb200.yaml \
  --user "$USER"

The explanation includes each role's enabled features, selected backends, workload and fabric profile, environment provenance, and resource claims.

Lower-level integrations can append exact command-line fragments with vllm_raw_args. Unlike the vllm: mapping, these strings are not renamed, quoted, or otherwise interpreted, and they are emitted after structured and computed arguments. Each entry is trusted shell syntax, so quote spaces and shell metacharacters within the entry when needed:

roles:
  - name: decode
    vllm_raw_args:
      - --trust-remote-code
      - --attention-config.backend=FLASH_ATTN

Specs can also extend a base YAML file and override only the fields that differ:

extends: wide-ep-base.yaml
release: wide-ep-2p-ep8-1d-ep8

roles:
  prefill:
    lws: {replicas: 2}

Overrides deep-merge mappings. roles may be written as a map keyed by role name in override files; each role override is merged into the matching base role before normal schema validation.

Every deployment includes an idle-shutdown controller by default. Once all expected vLLM API servers are ready, it watches request counters and running or queued requests across all roles. After 45 idle minutes it scales the model workloads and endpoint picker to zero, deletes the instance Gateway when the Gateway frontend is selected so its platform-managed proxy replicas are released, and scales itself to zero. Applying or deploying the spec again restores all of those resources. Change the timeout or opt out in the runtime configuration:

runtime:
  idle_shutdown:
    timeout_minutes: 90
    # enabled: false

Metric or Kubernetes API failures reset the idle timer, so the controller does not shut down an instance when it cannot establish that the instance is idle. For one-off operations, the CLI can override either setting without editing the model spec:

manifesto deploy models/qwen/qwen3-0.6b.yaml \
  --context example-context \
  --idle-timeout 15m

manifesto render manifest models/qwen/qwen3-0.6b.yaml \
  --no-idle-shutdown

DeepSeek V4 wide-EP specs use filenames that encode only the parallel layout: <prefill-replicas>P-EP<width>-<decode-replicas>D-EP<width>.yaml. For example, 3P-EP8-1D-EP16.yaml means three prefill LWS replicas at EP8 and one decode LWS replica at EP16. Backend choices stay inside the YAML because optimal backends can change independently of the parallel layout.

Cluster Profiles

Cluster profiles live under clusters/.

Three synthetic profiles are included for documentation and tests:

clusters/example-gb200.yaml
clusters/example-h200.yaml
clusters/example-stateless-b200.yaml

They deliberately omit real network, storage, scheduling, and provider details. Put usable profiles in the private user catalog described above; clusters/*.yaml is ignored by git except for files named example-*.yaml.

The stateless example intentionally declares no cache or logging filesystem. Its model pods omit persistent cache env, storage mounts, and the log tee wrapper; libraries use container-local defaults and logs stream to stdout. An external vllm-envs worktree cannot be selected unless its absolute path is covered by a model-pod volume mount.

Manifesto also omits Kubernetes fields whose defaults are sufficient. Set pod_defaults.image_pull_secrets, image_pull_policy, termination_grace_period_seconds, working_dir, or container_security_context only when cluster or image policy requires an explicit value. Pull secrets are existing Secret names in the workload namespace; Manifesto references them but does not create credential data. They are attached to every generated PodSpec, including routing and idle-shutdown pods. For example:

pod_defaults:
  image_pull_secrets:
    - example-registry-credentials

A dedicated model-server ServiceAccount is generated only when an OpenShift SCC must be bound to it.

Set pod_defaults.ephemeral_storage in a Kubernetes cluster profile to request and limit scratch storage for each model container. A role's explicit resources.ephemeral_storage overrides this default. This setting also sizes the pod-local JIT cache when one is used.

Cluster profiles own environment that should not be repeated in every model spec:

• available accelerators and the cluster default
• GPUs per node
• shared and local volume mounts
• optional namespace bootstrap configuration for shared PVCs
• pod annotations, scheduling constraints, extra devices, and security context
• RDMA extended-resource requests and OpenShift SCC authorization
• user, log, and cache path templates
• llm-d release
• UCX/NCCL/NVSHMEM/IMEX fabric env profiles
• optional Kueue LocalQueue selection for GPU LeaderWorkerSets

When a role omits resources.cpu or resources.memory, Manifesto derives the request from the role's inferred GPUs per pod (N) using built-in defaults:

• CPU: 6 + 2N
• memory: max(64Gi * N, 128Gi)

For example:

GPUs per pod	CPU	Memory
1	8	128Gi
2	10	128Gi
4	14	256Gi
8	22	512Gi

Set either value directly on a role when a model needs a different request:

roles:
  - name: decode
    parallelism: {tp: 4}
    resources: {memory: 224Gi}

Explicit role resource values always win independently, so the example keeps the built-in 14-CPU request while using exactly 224Gi of memory.

Reusing cluster workload settings

Controllers that submit other GPU workloads can reuse the cluster-owned part of a Manifesto profile without depending on model topology or routing:

from manifesto.cluster import load_cluster
from manifesto.workload import workload_settings

settings = workload_settings(load_cluster("clusters/my-cluster.yaml"))
accelerator = settings.accelerator("gb200")

The portable projection contains accelerator resource or DeviceClass names and node selectors, the default Kueue LocalQueue, accelerator-attached platform claims such as an NVIDIA compute-domain channel, and pod placement defaults such as affinity, tolerations, DNS, annotations, and image pull policy. The projection carries only each claim's portable name and template reference; storage, the rest of the fabric configuration, and launch settings remain part of Manifesto's serving-specific cluster model.

Manifesto also owns a controller-neutral workload IR and its Kubernetes object lowering. This lets tools describe a pod template and lifecycle policy while Manifesto consistently applies cluster placement and Kueue metadata:

manifesto render workload examples/non-indexed-job.yaml \
  --cluster clusters/my-cluster.yaml \
  --accelerator gb200

The job backend emits an ordinary, non-indexed batch/v1 Job by default and can explicitly describe Indexed Job completion policy for distributed work. A workload may optionally declare a normal or headless Service. The same IR lowers Deployment and LeaderWorkerSet backends, including explicit LWS leader templates; Grove is reserved as an extension point until its rendering contract is implemented.

An explicit LWS leader template is intentionally same-shaped: Manifesto applies the workload's accelerator count, accelerator container, and cluster-owned claims to both leader and worker templates. It also removes Kueue queue labels from both Pod templates and keeps queue selection on the LeaderWorkerSet, as required by Kueue's LWS integration.

Kueue admission

GPU roles can be admitted through a Kueue LocalQueue selected in the cluster profile:

kueue:
  local_queue: example-gpu-queue

When configured, Manifesto preserves each role's native workload kind. A single-node role remains a Deployment and receives kueue.x-k8s.io/queue-name on both the Deployment and its pod template. Each Deployment pod is therefore admitted independently. A role whose topology requires a LeaderWorkerSet receives the queue label on the LeaderWorkerSet, whose complete leader/worker group is admitted atomically. The queue is cluster- and namespace-specific operational configuration, so it belongs in the private cluster profile rather than a shareable model spec. Omitting kueue preserves the same workload kinds without admission metadata. CPU-only supporting resources, including the in-cluster end-to-end probe Job, do not receive Kueue metadata or suspension.

Before applying a queued workload, manifesto deploy verifies that the Kueue APIs are served, the selected LocalQueue is Active=True, and its referenced ClusterQueue is Active=True. It also prints every init-container and container request in each rendered workload pod template so quota requirements are visible before mutation, and rejects request resource names that the ClusterQueue does not cover. Deployments do not require the LeaderWorkerSet API; native LeaderWorkerSets are checked for it.

Changing queue admission on an existing LeaderWorkerSet is intentionally a recreate operation because its admitted group is atomic. Enabling, changing, or disabling its queue deletes it before applying the replacement. Deployment queue changes use the Deployment rollout and replace independently admitted pods without changing workload kind. Moving a same-named role between native Deployment and LeaderWorkerSet shapes deletes the obsolete controller first. LWS recreation and kind transitions interrupt serving; use a distinct release name for a parallel, zero-downtime rollout.

The workflow CLI requires a cluster profile for commands that render a spec. Set MANIFESTO_CLUSTER directly, pass --cluster, or set MANIFESTO_CLUSTER_MAP in .env as a local lookup table keyed by kube context or kube cluster name:

MANIFESTO_CLUSTER=clusters/example-gb200.yaml manifesto deploy models/qwen/aggregated.yaml
manifesto deploy models/qwen/h200-aggregated.yaml

The namespace defaults to the selected kube context namespace, falling back to default when the context has no namespace. Use --context to select a context without changing kubectl's current context, or set MANIFESTO_NAMESPACE in .env to override the namespace.

Shared storage is configured without assuming a filesystem implementation:

storage:
  shared_volume:
    persistentVolumeClaim:
      claimName: model-cache
  shared_mount_path: /mnt/shared
  local_nvme_path: /mnt/local

shared_volume accepts any Kubernetes volume source, such as persistentVolumeClaim, hostPath, csi, nfs, or emptyDir. Profiles backed by separate host-local cache volumes can omit it entirely.

When Manifesto should provision a referenced shared PVC in each namespace, declare storage.shared_claim with its storage class, access modes, and size, inspect it with manifesto render bootstrap, then run manifesto deploy bootstrap. The apply command uses kubectl apply, so it is safe to repeat. PVCs managed outside Manifesto should omit shared_claim.

Set platform: openshift for OpenShift clusters. This adds a stable USER fallback for Python libraries when containers run under an arbitrary UID, without affecting standard Kubernetes profiles. It also omits the node-exporter sidecar because its /sys and /proc hostPath volumes are incompatible with OpenShift's restricted SCC; dcgm-exporter remains enabled when requested. An SCC can be granted to each release-specific model service account:

openshift:
  scc: custom-scc

A model role can override automatic fabric selection when its backend requires one explicitly:

roles:
  - name: decode
    fabric_profile: custom_ep

Fabric profiles provide backend-specific environment settings:

fabric:
  profiles:
    deepep_decode:
      env:
        VLLM_USE_NCCL_SYMM_MEM: "1"

Select a non-default Gateway API implementation in the cluster profile:

gateway:
  class_name: data-science-gateway-class
  service_type: ClusterIP

Compiled-cache paths derive their cache key from the resolved model image tag or digest. Custom and dev builds can force a fresh namespace explicitly:

cache:
  key: dev-build-42

For Deployment model pods, writable JIT caches live on a size-limited pod emptyDir (using the role's ephemeral_storage, then the cluster's pod_defaults.ephemeral_storage, or 32Gi if neither is configured). They survive container restarts and disappear when Kubernetes removes the pod, so rollouts do not leave old cache directories on the persistent filesystem. The Hugging Face model cache remains on the configured shared or host volume. LeaderWorkerSet roles continue using the configured persistent JIT cache paths.

When persistent cache storage is configured, model pods clear their JIT and compilation caches after a failed process exit, or when the same container restarts after terminating without running its exit handler. Stateless pods omit this machinery. FlashInfer autotuning results under $VLLM_CACHE_ROOT/flashinfer_autotune_cache are preserved. Disable this default for a deployment only when retaining possibly corrupt compiled artifacts is intentional:

cache:
  cleanup_on_crash: false

Model-server stdout/stderr is persisted by the generated launch script. Configure the backing PVC and root path in the cluster profile:

logging:
  pvc: example-shared-cache
  mount_path: /mnt/shared
  root: /mnt/shared/{user}/logs

The rendered pods tee logs to {root}/{role}, for example /mnt/shared/<user>/logs/decode and /mnt/shared/<user>/logs/prefill.

Slurm

Select platform: slurm in a cluster profile. Start with clusters/example-slurm.yaml for Apptainer/Singularity. Copy the profile into your private cluster catalog and set its partition, GPU type, runtime, and filesystem paths.

Use the same model spec for either platform. The cluster profile selects the backend and default accelerator; the shared Qwen example has no platform-specific variant. For example, only --cluster changes here:

manifesto render manifest models/qwen/qwen3-0.6b.yaml --cluster example-stateless-b200
manifesto render manifest models/qwen/qwen3-0.6b.yaml --cluster example-slurm

To save and submit a batch script:

manifesto render slurm models/qwen/qwen3-0.6b.yaml --cluster my-slurm -o qwen.sbatch
bash -n qwen.sbatch
sbatch --test-only qwen.sbatch
sbatch qwen.sbatch

The regular render manifest, render file, and deploy commands also select Slurm from the cluster profile. render file defaults to /tmp/manifesto.sbatch. Rendering and config validate are offline and need no Kubernetes tools.

The Slurm workflow commands run locally on a login node, or over SSH when the profile sets slurm.ssh_host: user@login-host (an SSH config alias also works):

manifesto slurm submit models/qwen/qwen3-0.6b.yaml --cluster my-slurm --test-only
manifesto deploy models/qwen/qwen3-0.6b.yaml --cluster my-slurm
manifesto slurm servers --cluster my-slurm
manifesto slurm stop 12345 --cluster my-slurm
manifesto slurm stop 12345_0 --cluster my-slurm  # just one replica

Submission prints the Slurm job ID. Each submit creates a new job, including when the release name is unchanged. servers lists the authenticated user's Manifesto jobs; --output name prints job IDs and --output json includes state, nodes, and scheduling reason. Scheduler validation uses sbatch --test-only; it does not launch a container or test model compatibility. SSH submission sends the complete batch script over stdin and does not install Manifesto remotely. SSH authentication must already work without a password prompt.

Allocation and launch

Slurm supports direct vLLM, llm-d load-aware routing, and NIXL prefill/decode disaggregation. TP, PP, DP, EP, computed arguments, model revision, pre-launch hooks, profiling, and existing vLLM environments use the same model schema and launch logic as Kubernetes.

Model setting	Slurm behavior
lws.size	Nodes per serving replica, one srun task per node
lws.replicas	Direct serving: job array elements. llm-d: replicas in one routed allocation
parallelism	GPUs per node derived from tp × pp × dp ÷ lws.size
resources.cpu	CPUs per task, rounded up to a whole CPU
resources.memory	Memory per node, converted to MiB and rounded up
slurm.time	Maximum lifetime of each replica

Each replica uses its first allocated host for rendezvous and prints that host's serving URL in the batch log. srun propagates failures to the other nodes in the replica. Manifesto passes Slurm's CUDA_VISIBLE_DEVICES unchanged into each container, including noncontiguous device IDs and UUIDs. One vLLM launcher per node starts the local DP engines and assigns their devices. For DP, vLLM's coordinator manages the group, with the API on the first node and headless workers on the remaining nodes for direct serving. llm-d uses external DP balancing and discovers each API endpoint, excluding headless nodes.

Cluster accelerator allocation uses slurm: {gres: gpu} or a typed value such as slurm: {gres: "gpu:nvidia_gr100"}. Omit the count; Manifesto derives it. slurm.partition, account, qos, and constraint map to scheduler options. Jobs request exclusive nodes by default because vLLM binds host ports. If you disable slurm.exclusive for direct serving, ensure concurrently running jobs use distinct ports. llm-d requires exclusive nodes because its router and P/D services use fixed host ports.

llm-d routing and prefill/decode

The same routed model works on Kubernetes and Slurm. For example:

manifesto render manifest models/qwen/qwen3-0.6b-pd.yaml --cluster my-slurm > pd.sbatch
manifesto slurm submit models/qwen/qwen3-0.6b-pd.yaml --cluster my-slurm --test-only
manifesto deploy models/qwen/qwen3-0.6b-pd.yaml --cluster my-slurm

For aggregated load-aware serving, set routing.kind: load_aware and choose lws.replicas in the model. Routing profiles from --routing-profile work on both platforms. Cluster configuration owns the runtime and endpoint discovery.

Manifesto follows llm-d's non-Kubernetes deployment path:

• Envoy and the endpoint picker run on the first serving node. The batch log prints their shared URL, using slurm.port (default 8081).
• Each serving role gets a heterogeneous Slurm job component with its own CPU, memory, GPU, and node requests. Replicas use consecutive groups of nodes in that component. P/D therefore requires Slurm's backfill scheduler and heterogeneous-job support; aggregated serving uses a regular allocation.
• The batch script resolves allocated hostnames to IPv4 addresses and writes the endpoint picker's discovery file. It lists only API nodes and all their DP ports. The allocation is fixed for the job's lifetime.
• Decode proxies run beside API nodes and use the same NIXLv2 arguments as Kubernetes. Both P/D roles must configure NixlConnector. The allocated node address supplies VLLM_NIXL_SIDE_CHANNEL_HOST unless explicitly overridden.
• Slurm reserves extra resources for auxiliary services: slurm.router_cpus (8) and router_memory (16Gi) on each node of the first role, plus proxy_cpus (2) and proxy_memory (4Gi) on each proxy role's node. The router runs only on the first node; unused overhead on other nodes keeps each role's allocation uniform. srun --overlap lets these CPU services share the reserved nodes with vLLM.
• All components share one job lifetime. A service failure stops the other steps. Cancel the numeric leader job ID to stop the whole P/D deployment.

The batch node needs Python 3 for config generation and IPv4 DNS resolution. Configs are staged on that node and mounted into the router containers; no shared config directory or Manifesto installation on the cluster is required. With slurm.runtime: native, install epp, envoy, and pd-sidecar on PATH alongside vLLM. With container runtimes, cluster.llm_d selects their images.

The image catalog now uses llm-d router v0.11.0 and its current image names, parser, label-filter, picker, and sidecar options. Custom routing profiles and pinned images must use compatible plugin/flag schemas. Kubernetes-dependent llm-d features (InferenceObjective controllers, model rewriting, PodMonitor discovery, Gateway API, and router HA) are outside the Slurm deployment path.

Containers, storage, and lifetime

• slurm.runtime: apptainer (default) or singularity uses exec --nv --no-eval. Bare OCI image names get a docker:// prefix; absolute SIF paths are supported. Use a runtime version supporting --no-eval.
• slurm.runtime: pyxis uses srun --container-image. Registry-qualified OCI names are converted to Enroot's REGISTRY#IMAGE syntax; absolute .sqsh paths work directly. The image entrypoint is bypassed so Manifesto starts vLLM. Pyxis/Enroot also requires /bin/sh inside each image during startup. For distroless llm-d/Envoy images, use shell-equipped images in cluster.llm_d.images or bind a static shell of the node's architecture to /bin/sh through slurm.binds. A dynamically linked host shell also needs its runtime libraries.
• slurm.runtime: native uses vLLM from the host or runtime.vllm_env.
• slurm.binds contains {source: /host/path, target: /container/path} entries with optional read_only: true. Paths must exist on every allocated node. An external runtime.vllm_env must be covered by a bind when using containers.
• slurm.setup contains trusted shell commands run on the batch host before srun, for example loading runtime modules or setting Enroot paths. Model runtime.pre_launch and role hooks run inside each serving task instead.
• Use cache.hf_home, paths.cache_root, and paths.log_root for filesystem caches and logs. Writable compilation caches are isolated by job, array element, and hostname. Slurm also writes stdout/stderr to manifesto-RELEASE-JOB_ARRAYINDEX.out for direct serving or manifesto-RELEASE-JOB.out for llm-d in the submission directory.
• Export HF_TOKEN in the submitting environment when needed. Ambient tokens are not embedded in scripts. With SSH, authentication and environment must be configured on the remote login host; local credentials are not forwarded.

For Rubin, select an ARM64 image built for the GPU and driver, such as an appropriate CUDA 13.4 vLLM build. The generic model example's standard image is not a promise of Rubin compatibility. Enroot extraction needs filesystem features such as extended attributes; use node-local temporary/data paths when the shared home filesystem is NFS. See the vLLM Rubin image guidance and Pyxis runtime options.

Kubernetes sidecars and idle shutdown are omitted when left at implicit defaults. Explicitly requesting them raises an error. Kubernetes pod storage defaults belong in the cluster's pod_defaults, so the model needs no Slurm variant or lifecycle overrides. Kubernetes workload overrides, PVCs, Kueue, DRA, and Kubernetes ephemeral-storage/shared-memory requests are unsupported. Use Slurm wall time and slurm stop for lifecycle control. ready, test e2e, and file apply/diff remain Kubernetes commands; inspect a Slurm endpoint with /health and submit saved scripts with sbatch.

Manual Manifest Workflow

The file workflow is the preferred path when you want to share or tweak exactly what will be deployed:

manifesto render file models/deepseek-v4/1P-EP8-1D-EP8.yaml
$EDITOR /tmp/manifesto.yaml
manifesto file diff
manifesto file apply

manifesto file diff compares the rendered file against the live cluster with kubectl diff. manifesto file apply applies that file as-is. Use -o/--output with render file, or set MANIFESTO_RENDER_OUT, to choose a different file.

The low-level manifesto render manifest command performs no cluster I/O when passed an explicit --cluster and --namespace (or with MANIFESTO_NAMESPACE set). Workflow commands such as deploy, ready, file diff, and file apply are the cluster-touching commands.

Python rendering API

manifesto.render.render() returns ready-to-write text: Kubernetes YAML or a Slurm batch script, selected by cluster.platform.

from manifesto.cluster import load_cluster
from manifesto.render import render
from manifesto.spec import load_spec

cluster = load_cluster("clusters/example-slurm.yaml")
spec = load_spec("models/qwen/qwen3-0.6b.yaml", cluster)
artifact = render(spec, user="tester", cluster=cluster)

Python callers that need Kubernetes object dictionaries should use render_kubernetes(); render_to_yaml() serializes those objects. Existing callers that used render() for an object list should switch to render_kubernetes().

Lifecycle Commands

manifesto deploy SPEC *ARGS          # render and apply a full stack
manifesto servers                    # list live servers in the namespace
manifesto stop [SPEC] [--now]        # discover and delete live objects
manifesto stop --instance ID         # stop by live instance identity
manifesto ready SPEC                 # wait for pods and routing frontend
manifesto test e2e SPEC              # fresh-namespace dev + inference integration test
manifesto deploy routing SPEC *ARGS  # update routing only

manifesto stop is stateless: it finds live objects through their Manifesto instance labels and does not depend on a stored manifest or deployment inventory. With no target it opens an fzf picker, or a numbered picker when fzf is unavailable. Selecting an entry starts teardown immediately. Explicit spec and --instance targets remain noninteractive and are safe for scripts.

Teardown removes routing, model workloads, supporting objects, and remaining instance pods. It does not drain active requests or delete shared logs, caches, or model storage. --now retains the force-delete behavior.

Enable parser-driven completion for commands, flags, model configs, cluster and routing profiles, accelerator names, live deployment instance IDs, and local paths:

source <(manifesto completion zsh)   # zsh
source <(manifesto completion bash)  # bash
manifesto completion fish | source  # fish

For scripts, manifesto servers --output name prints one instance ID per line, and --output json includes the exact resources associated with each instance.

Slow clusters

manifesto servers and manifesto stop find live objects by listing every Manifesto-managed resource type. Those lists run concurrently, one request per type, so discovery costs roughly one API round trip of wall-clock time instead of one per type. There is no kubectl api-resources preflight: a type the cluster does not serve simply lists as zero objects, and pruning the set in advance would not save any wall clock once the lists run in parallel.

Discovery reads are bounded and retried on transient faults. Tune them when an API server is unusually slow or flaky:

MANIFESTO_KUBECTL_TIMEOUT=300  # seconds per read attempt; 0 disables the bound
MANIFESTO_KUBECTL_RETRIES=4    # extra attempts after a transient failure
MANIFESTO_TRACE=1              # log every kubectl invocation and its duration

MANIFESTO_TRACE=1 is the fastest way to find which call is slow:

MANIFESTO_TRACE=1 manifesto stop --instance ID

Shell completion of live instance IDs uses a short, non-retrying timeout, so a slow cluster yields no candidates instead of stalling the prompt.

Examples:

manifesto deploy models/qwen/aggregated.yaml
manifesto ready models/qwen/aggregated.yaml
manifesto test e2e models/qwen/qwen3-0.6b.yaml --cluster my-cluster
manifesto deploy models/deepseek-v4/1P-EP8-1D-EP8.yaml \
  --vllm-env /mnt/shared/$USER/vllm-envs/feature
manifesto deploy routing models/deepseek-v4/1P-EP8-1D-EP8.yaml
manifesto stop

manifesto test e2e owns the complete test lifecycle. By default it creates a fresh namespace, replaces profile storage with emptyDir volumes, deploys the vLLM bundled in the model image, waits for it to become ready, and creates an unprivileged Job that calls /v1/models followed by a real /v1/completions request. The Job targets the standalone router Service or generated Gateway when routing is enabled, and the model Service otherwise.

Pass --vllm-env /absolute/worktree/path to test an existing vllm-envs worktree instead. That mode preserves the cluster profile's normal storage so the worktree remains visible; its volume references must be usable in the fresh test namespace. Manifesto never creates, synchronizes, or modifies the worktree.

The namespace is deleted at the end, including on failure. Pass --namespace to choose its name (the command refuses an existing namespace), --keep-namespace to retain all test resources for inspection, --timeout to change the inference deadline, or --image to use a mirrored Python image. HF_TOKEN and a cluster profile are required, as with normal deployments.

Every rendered object is scoped by the release-based instance identity:

{release}

Names, labels, and selectors derive from that identity. User-scoped storage paths and the llm-d.ai/owner label still retain the user. Deployments must have unique release names within a namespace. Clusters whose users share release names can enable user-prefixed identities in their profile:

naming:
  user_prefix: true

External vLLM Environments

Use vllm-envs to create, synchronize, build, and remove development environments. Manifesto only points model pods at an existing vLLM worktree whose .venv has already been initialized:

manifesto deploy models/deepseek-v4/1P-EP8-1D-EP8.yaml \
  --vllm-env /mnt/shared/$USER/vllm-envs/feature

The path must be absolute and covered by a volume mount in every model pod. Manifesto fails before launching vLLM if the worktree or .venv/bin/activate is missing. Set the same pointer declaratively with runtime.vllm_env in a model spec. Without either setting, the deployment uses vLLM from the model image.

Monitoring

The monitoring/ directory contains namespace-scoped Prometheus and Grafana configuration. Per-pod exporters such as DCGM exporter are part of rendered model manifests and can be toggled from the model spec.

Requirements

Local tools:

• uv
• kubectl

Slurm workflows use sbatch, srun, squeue, scontrol, and scancel on the login host. With slurm.ssh_host, only ssh is needed locally alongside Manifesto; Kubernetes tools are unnecessary.

Expected .env values:

HF_TOKEN=replace-me
KUBECONFIG=/path/to/kubeconfig
MANIFESTO_CLUSTER_MAP=my-context=clusters/example-h200.yaml
MANIFESTO_NAMESPACE=workload-ns  # optional, defaults to current kube namespace

manifesto deploy and manifesto file apply create or update the namespace's hf-secret from HF_TOKEN before launching pods. The token is sent to kubectl apply over stdin and is not included in rendered workload files. Pure manifesto render commands remain side-effect free; manage the secret explicitly when applying rendered stdout with kubectl.

Test

uv run --extra test pytest
manifesto --help

Design Bias

Manifesto is intentionally not a controller, operator, CRD, or matching engine.

It is a client-side renderer:

model spec + cluster profile + user -> raw Kubernetes YAML
                                  -> Slurm batch script

The rendered file is the artifact. Inspect it, edit it, share it, then apply the YAML or submit the batch script.