← Toutes les actualitésLire l'article ↗
Deploying vLLM
Synthèse
Enter the following command to deploy the vLLM inference server and its service, adapted from the vLLM Kubernetes deployment documentation: VLLM_IMAGE=public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: vllm-server spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: vllm template: metadata: labels: app.kubernetes.io/name: vllm spec: containers: - n
Enter the following command to deploy the vLLM inference server and its service, adapted from the vLLM Kubernetes deployment documentation: VLLM_IMAGE=public. ecr. aws/q9t5s3a7/vllm-cpu-release-repo:latest cat < kubernetes. io/name: vllm spec: containers: - name: vllm image: $VLLM_IMAGE command: ["/bin/sh", "-c"] args: [ "vllm serve meta-llama/Llama-3. 2-1B-Instruct --gpu-memory-utilization 0. 80" ] env: - name: HF_TOKEN valueFrom: secretKeyRef: name: hf-token-secret key: token ports: - containerPort: 8000 volumeMounts: - name: llama-storage mountPath: /root/. cache/huggingface volumes: - name: llama-storage persistentVolumeClaim: claimName: vllm-models --- apiVersion: v1 kind: Service metadata: name: vllm-server spec: selector: app. kubernetes. io/name: vllm ports: - protocol: TCP port: 8000 targetPort: 8000 type: ClusterIP EOF 📝 NOTE: Even on a CPU-only deployment, vLLM uses the --gpu-memory-utilization flag to govern how aggressively it reserves memory for processing requests. Without this flag, vLLM defaults to 92% of available memory, which caused startup failures in my testing environment. Setting it to 0. 80 provided enough headroom in my environment for the engine to initialize successfully.
Source originale
docs.vllm.ai