Environment variables a Managed Inference Job reads, including vLLM flags and Parakeet server settings.
When a Managed Inference Job starts, the inference agent reads the job's environment variables. Each variable either sets a model server option or changes an agent setting, such as the startup timeout. This page lists the variables that each job type reads.
The job type determines where you can set a variable.
| Location | Setting | vLLM Managed Inference Job | Parakeet Managed Inference Job |
|---|
| Web interface | Environment variables field | Yes | No |
| CLI | --env flag of cosmicac jobs create | Yes | No |
| Model master | inference_param_overrides.env list | Yes | Yes |
For a Parakeet job, cosmicac jobs create ignores the --env flag.
CosmicAC adds each variable in a model master's inference_param_overrides.env list to every new job for that model. If the job already sets a variable with the same name, CosmicAC keeps the job's value.
When a job starts, CosmicAC waits a set time for the model server to become ready. If the model server isn't ready in that time, CosmicAC restarts it. The following variables set the timeout and the restart limit.
| Variable | Default | Description |
|---|
| DOCKER_STARTUP_TIMEOUT_MS | 1800000, or 30 minutes | The time, in milliseconds, that the model server has to become ready. On a multi-node vLLM job, the default is 3600000, or 60 minutes. To turn off the timeout, set the value to 0. |
| DOCKER_MAX_RESTARTS | 3 | The number of times CosmicAC restarts the model server before the job fails. |
CosmicAC sets the following variables on every Managed Inference Job. The Your value column shows what CosmicAC does with a value you set.
- Used: your value replaces the value that CosmicAC sets.
- Discarded: your value has no effect, and CosmicAC keeps its own.
| Variable | Description | Your value |
|---|
| DOCKER_IMAGE | The model server image. On a vLLM job, the Runtime image (CUDA) field also sets this value. | Used |
| DOCKER_GPU | Whether the model server can use the job's GPUs. The default is true. | Used |
| DOCKER_TRANSPORT | How the agent connects to the model server. The default is a Unix domain socket, or TCP on a multi-node vLLM job. | Used |
| DOCKER_STARTUP_TIMEOUT_MS | The startup timeout. See Startup timeout. | Used |
| DOCKER_MAX_RESTARTS | The restart limit. See Startup timeout. | Used |
| INFERENCE_MODE | The agent's run mode. The value is always docker. | Discarded |
| MODEL_NAME | The model to serve. The Model field sets this value. | Discarded |
| ENDPOINT_NAME | The endpoint name. The Endpoint name field sets this value. | Discarded |
| REQUIRE_AUTH_HEADER | Whether callers must send an API key. The Require Authorization header field sets this value. | Discarded |
| COSMICAC_INSTANCE_ID | The job ID that CosmicAC uses to label the job's logs and metrics. | Discarded |
| JOB_ID | The job ID. | Discarded |
| REPLICA_ID | The replica ID. | Discarded |
| HOST_AGENT_DIR | The directory that the agent shares with the host agent. | Discarded |
Besides the general variables, a vLLM Managed Inference Job reads vllm serve options and multi-node settings.
Each variable in this section sets one vllm serve flag on the model server. The variable name comes from the flag. To get the name, remove the leading --, replace each remaining hyphen with an underscore, and change every letter to uppercase. For example, --enable-prefix-caching becomes ENABLE_PREFIX_CACHING.
The variable's value controls what CosmicAC adds to the vllm serve command.
true: the flag with no value. For example, TRUST_REMOTE_CODE set to true adds --trust-remote-code.false or an empty value: nothing. CosmicAC leaves the flag out.- Any other value: the flag followed by the value. For example, SWAP_SPACE set to
0 adds --swap-space 0.
The following tables list the 142 supported vllm serve flags and their variables, grouped by area. For details about each flag, see Engine arguments in the vLLM documentation.
| Variable | vLLM flag |
|---|
| MODEL | --model |
| RUNNER | --runner |
| CONVERT | --convert |
| TASK | --task |
| TOKENIZER | --tokenizer |
| TOKENIZER_MODE | --tokenizer-mode |
| TRUST_REMOTE_CODE | --trust-remote-code |
| DTYPE | --dtype |
| SEED | --seed |
| HF_CONFIG_PATH | --hf-config-path |
| ALLOWED_LOCAL_MEDIA_PATH | --allowed-local-media-path |
| ALLOWED_MEDIA_DOMAINS | --allowed-media-domains |
| REVISION | --revision |
| CODE_REVISION | --code-revision |
| TOKENIZER_REVISION | --tokenizer-revision |
| MAX_MODEL_LEN | --max-model-len |
| QUANTIZATION | --quantization |
| ENFORCE_EAGER | --enforce-eager |
| MAX_LOGPROBS | --max-logprobs |
| LOGPROBS_MODE | --logprobs-mode |
| DISABLE_SLIDING_WINDOW | --disable-sliding-window |
| DISABLE_CASCADE_ATTN | --disable-cascade-attn |
| SKIP_TOKENIZER_INIT | --skip-tokenizer-init |
| ENABLE_PROMPT_EMBEDS | --enable-prompt-embeds |
| SERVED_MODEL_NAME | --served-model-name |
| CONFIG_FORMAT | --config-format |
| HF_TOKEN | --hf-token |
| HF_OVERRIDES | --hf-overrides |
| POOLER_CONFIG | --pooler-config |
| OVERRIDE_POOLER_CONFIG | --override-pooler-config |
| LOGITS_PROCESSOR_PATTERN | --logits-processor-pattern |
| GENERATION_CONFIG | --generation-config |
| OVERRIDE_GENERATION_CONFIG | --override-generation-config |
| ENABLE_SLEEP_MODE | --enable-sleep-mode |
| MODEL_IMPL | --model-impl |
| OVERRIDE_ATTENTION_DTYPE | --override-attention-dtype |
| LOGITS_PROCESSORS | --logits-processors |
| IO_PROCESSOR_PLUGIN | --io-processor-plugin |
| Variable | vLLM flag |
|---|
| LOAD_FORMAT | --load-format |
| DOWNLOAD_DIR | --download-dir |
| SAFETENSORS_LOAD_STRATEGY | --safetensors-load-strategy |
| MODEL_LOADER_EXTRA_CONFIG | --model-loader-extra-config |
| IGNORE_PATTERNS | --ignore-patterns |
| USE_TQDM_ON_LOAD | --use-tqdm-on-load |
| PT_LOAD_MAP_LOCATION | --pt-load-map-location |
| Variable | vLLM flag |
|---|
| REASONING_PARSER | --reasoning-parser |
| REASONING_PARSER_PLUGIN | --reasoning-parser-plugin |
| Variable | vLLM flag |
|---|
| DISTRIBUTED_EXECUTOR_BACKEND | --distributed-executor-backend |
| PIPELINE_PARALLEL_SIZE | --pipeline-parallel-size |
| MASTER_ADDR | --master-addr |
| MASTER_PORT | --master-port |
| NNODES | --nnodes |
| NODE_RANK | --node-rank |
| TENSOR_PARALLEL_SIZE | --tensor-parallel-size |
| DECODE_CONTEXT_PARALLEL_SIZE | --decode-context-parallel-size |
| DCP_KV_CACHE_INTERLEAVE_SIZE | --dcp-kv-cache-interleave-size |
| CP_KV_CACHE_INTERLEAVE_SIZE | --cp-kv-cache-interleave-size |
| PREFILL_CONTEXT_PARALLEL_SIZE | --prefill-context-parallel-size |
| DATA_PARALLEL_SIZE | --data-parallel-size |
| DATA_PARALLEL_RANK | --data-parallel-rank |
| DATA_PARALLEL_START_RANK | --data-parallel-start-rank |
| DATA_PARALLEL_SIZE_LOCAL | --data-parallel-size-local |
| DATA_PARALLEL_ADDRESS | --data-parallel-address |
| DATA_PARALLEL_RPC_PORT | --data-parallel-rpc-port |
| DATA_PARALLEL_BACKEND | --data-parallel-backend |
| DATA_PARALLEL_HYBRID_LB | --data-parallel-hybrid-lb |
| DATA_PARALLEL_EXTERNAL_LB | --data-parallel-external-lb |
| ENABLE_EXPERT_PARALLEL | --enable-expert-parallel |
| ALL2ALL_BACKEND | --all2all-backend |
| ENABLE_DBO | --enable-dbo |
| DBO_DECODE_TOKEN_THRESHOLD | --dbo-decode-token-threshold |
| DBO_PREFILL_TOKEN_THRESHOLD | --dbo-prefill-token-threshold |
| DISABLE_NCCL_FOR_DP_SYNCHRONIZATION | --disable-nccl-for-dp-synchronization |
| ENABLE_EPLB | --enable-eplb |
| EPLB_CONFIG | --eplb-config |
| EXPERT_PLACEMENT_STRATEGY | --expert-placement-strategy |
| MAX_PARALLEL_LOADING_WORKERS | --max-parallel-loading-workers |
| RAY_WORKERS_USE_NSIGHT | --ray-workers-use-nsight |
| DISABLE_CUSTOM_ALL_REDUCE | --disable-custom-all-reduce |
| WORKER_CLS | --worker-cls |
| WORKER_EXTENSION_CLS | --worker-extension-cls |
| Variable | vLLM flag |
|---|
| BLOCK_SIZE | --block-size |
| GPU_MEMORY_UTILIZATION | --gpu-memory-utilization |
| KV_CACHE_MEMORY_BYTES | --kv-cache-memory-bytes |
| SWAP_SPACE | --swap-space |
| KV_CACHE_DTYPE | --kv-cache-dtype |
| NUM_GPU_BLOCKS_OVERRIDE | --num-gpu-blocks-override |
| ENABLE_PREFIX_CACHING | --enable-prefix-caching |
| PREFIX_CACHING_HASH_ALGO | --prefix-caching-hash-algo |
| CPU_OFFLOAD_GB | --cpu-offload-gb |
| CALCULATE_KV_SCALES | --calculate-kv-scales |
| KV_SHARING_FAST_PREFILL | --kv-sharing-fast-prefill |
| MAMBA_CACHE_DTYPE | --mamba-cache-dtype |
| MAMBA_SSM_CACHE_DTYPE | --mamba-ssm-cache-dtype |
| MAMBA_BLOCK_SIZE | --mamba-block-size |
| KV_OFFLOADING_SIZE | --kv-offloading-size |
| KV_OFFLOADING_BACKEND | --kv-offloading-backend |
| Variable | vLLM flag |
|---|
| LIMIT_MM_PER_PROMPT | --limit-mm-per-prompt |
| ENABLE_MM_EMBEDS | --enable-mm-embeds |
| MEDIA_IO_KWARGS | --media-io-kwargs |
| MM_PROCESSOR_KWARGS | --mm-processor-kwargs |
| MM_PROCESSOR_CACHE_GB | --mm-processor-cache-gb |
| MM_PROCESSOR_CACHE_TYPE | --mm-processor-cache-type |
| MM_SHM_CACHE_MAX_OBJECT_SIZE_MB | --mm-shm-cache-max-object-size-mb |
| MM_ENCODER_TP_MODE | --mm-encoder-tp-mode |
| MM_ENCODER_ATTN_BACKEND | --mm-encoder-attn-backend |
| INTERLEAVE_MM_STRINGS | --interleave-mm-strings |
| SKIP_MM_PROFILING | --skip-mm-profiling |
| VIDEO_PRUNING_RATE | --video-pruning-rate |
| Variable | vLLM flag |
|---|
| ENABLE_LORA | --enable-lora |
| MAX_LORAS | --max-loras |
| MAX_LORA_RANK | --max-lora-rank |
| LORA_DTYPE | --lora-dtype |
| MAX_CPU_LORAS | --max-cpu-loras |
| FULLY_SHARDED_LORAS | --fully-sharded-loras |
| DEFAULT_MM_LORAS | --default-mm-loras |
| Variable | vLLM flag |
|---|
| SHOW_HIDDEN_METRICS_FOR_VERSION | --show-hidden-metrics-for-version |
| OTLP_TRACES_ENDPOINT | --otlp-traces-endpoint |
| COLLECT_DETAILED_TRACES | --collect-detailed-traces |
| Variable | vLLM flag |
|---|
| MAX_NUM_BATCHED_TOKENS | --max-num-batched-tokens |
| MAX_NUM_SEQS | --max-num-seqs |
| MAX_NUM_PARTIAL_PREFILLS | --max-num-partial-prefills |
| MAX_LONG_PARTIAL_PREFILLS | --max-long-partial-prefills |
| LONG_PREFILL_TOKEN_THRESHOLD | --long-prefill-token-threshold |
| SCHEDULING_POLICY | --scheduling-policy |
| ENABLE_CHUNKED_PREFILL | --enable-chunked-prefill |
| DISABLE_CHUNKED_MM_INPUT | --disable-chunked-mm-input |
| SCHEDULER_CLS | --scheduler-cls |
| DISABLE_HYBRID_KV_CACHE_MANAGER | --disable-hybrid-kv-cache-manager |
| ASYNC_SCHEDULING | --async-scheduling |
| STREAM_INTERVAL | --stream-interval |
| Variable | vLLM flag |
|---|
| CUDAGRAPH_CAPTURE_SIZES | --cudagraph-capture-sizes |
| MAX_CUDAGRAPH_CAPTURE_SIZE | --max-cudagraph-capture-size |
| Variable | vLLM flag |
|---|
| SPECULATIVE_CONFIG | --speculative-config |
| KV_TRANSFER_CONFIG | --kv-transfer-config |
| KV_EVENTS_CONFIG | --kv-events-config |
| EC_TRANSFER_CONFIG | --ec-transfer-config |
| COMPILATION_CONFIG | --compilation-config |
| ADDITIONAL_CONFIG | --additional-config |
| STRUCTURED_OUTPUTS_CONFIG | --structured-outputs-config |
| Variable | vLLM flag |
|---|
| DISABLE_LOG_STATS | --disable-log-stats |
| AGGREGATE_ENGINE_LOGGING | --aggregate-engine-logging |
On a multi-node vLLM job, CosmicAC also sets the following variables. For what Used and Discarded mean, see Reserved names.
| Variable | Description | Your value |
|---|
| DOCKER_SHM_SIZE | The shared memory size for the model server. The default is 10.24g. | Used |
| RAY_PORT | The Ray port. The default is 6379. | Used |
| EXPECTED_GPUS | The number of GPUs that the replica waits for before it serves requests. CosmicAC sets this value from the job's GPU count. | Discarded |
| VMI_NAME | The name that identifies the virtual machine instance (VMI) to its peers. | Discarded |
| NODE_INDEX | The node's position in the replica. | Discarded |
| NODE_COUNT | The number of nodes in the replica. | Discarded |
| OVERLAY_IFACE | The overlay network interface, which CosmicAC discovers when the VMI starts. | Discarded |
| MOUNT_INFINIBAND | Whether to mount the InfiniBand devices. CosmicAC always sets this value on a multi-node job. | Discarded |
Besides the general variables, a Parakeet Managed Inference Job reads Parakeet server settings. You set these variables in the model master. See Where to set a variable.
The inference agent passes each variable in this section to the Parakeet server.
| Variable | Description |
|---|
| BATCH_SIZE | The number of audio chunks the Parakeet server transcribes in one batch. |
| NUM_WORKERS | The number of worker processes the Parakeet server runs. |
| SAMPLE_RATE | The audio sample rate, in hertz, that the Parakeet server uses. |
| LOG_LEVEL | The Parakeet server's log level. |
On a Parakeet job, CosmicAC also sets the following variables. Most of them come from the job fields, listed in Parakeet Managed Inference Job configuration. For what Used and Discarded mean, see Reserved names.
| Variable | Description | Your value |
|---|
| INFERENCE_BACKEND | The inference backend. The value is always the Parakeet backend. | Discarded |
| PORT | The Parakeet server port. The value is always 8777. | Discarded |
| TRANSCRIBE_CHUNK_LEN | The audio chunk length. The Chunk duration field sets this value. | Discarded |
| TRANSCRIBE_OVERLAP | The overlap between chunks. The Chunk overlap field sets this value. | Discarded |
| MAX_FILE_SIZE_MB | The maximum upload size. The Max file size field sets this value. | Discarded |
