DEV Community

#vllm

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
vLLM ignores LoRA rank_pattern and alpha_pattern and serves the adapter at the wrong scale
homelabpm profile

vLLM ignores LoRA rank_pattern and alpha_pattern and serves the adapter at the wrong scale

#llm #vllm #lora #selfhosted
Comments
5 min read
vLLM's weight cache can serve another checkpoint's weights when the tensor layout matches
homelabpm profile

vLLM's weight cache can serve another checkpoint's weights when the tensor layout matches

#llm #vllm #selfhosted #debugging
Comments
4 min read
Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default
renolu profile

Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default

#vllm #localllm #speculativedecoding #quantization
Comments
4 min read
Evaluating Speculative Decoding in vLLM on AMD MI300X GPUs
rammehta1899 profile

Evaluating Speculative Decoding in vLLM on AMD MI300X GPUs

#vllm #amd #mi300x #rocm
Comments
1 min read
Gemma 4 on Amazon SageMaker: The NVIDIA T4 Decodes at 0.8x of the L4 With the Same Answers
xbill profile

Gemma 4 on Amazon SageMaker: The NVIDIA T4 Decodes at 0.8x of the L4 With the Same Answers

#aws #sagemaker #gemma #vllm
15
Picked as gem Comments 1
9 min read
Gemma 4 on Amazon SageMaker: 4-Bit Embeddings Decode up to 1.39x Faster on One L4
xbill profile

Gemma 4 on Amazon SageMaker: 4-Bit Embeddings Decode up to 1.39x Faster on One L4

#aws #sagemaker #gemma #vllm
1
Comments
11 min read
Gemma 4 on Amazon SageMaker or a VM? The Same Model Server at 1.40x the Price
xbill profile

Gemma 4 on Amazon SageMaker or a VM? The Same Model Server at 1.40x the Price

#aws #sagemaker #gemma #vllm
1
Comments
8 min read
Gemma 4 on a Tesla T4, Part 3: Int4 Embeddings Serve E2B in 2.86 GiB at 2.30x bf16
xbill profile

Gemma 4 on a Tesla T4, Part 3: Int4 Embeddings Serve E2B in 2.86 GiB at 2.30x bf16

#gemma #vllm #cuda #machinelearning
9
Comments
11 min read
Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4
xbill profile

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

#aws #sagemaker #gemma #vllm
Comments
9 min read
Gemma 4 on Amazon SageMaker: The NVIDIA T4 Decodes at 0.8x of the L4 With the Same Answers
xbill profile

Gemma 4 on Amazon SageMaker: The NVIDIA T4 Decodes at 0.8x of the L4 With the Same Answers

#aws #sagemaker #gemma #vllm
1
Comments
9 min read
Run vLLM on Kubernetes with Minikube, WSL2 and NVIDIA GPU
vishnuhd profile

Run vLLM on Kubernetes with Minikube, WSL2 and NVIDIA GPU

#ai #localllm #vllm #kubernetes
Comments
13 min read
KV Cache on 16 GB GPUs: Making Long Context Actually Fit
rosgluk profile
Rost

KV Cache on 16 GB GPUs: Making Long Context Actually Fit

#llm #llamacpp #vllm #ollama
Comments 1
22 min read
Inside vLLM: Following One Request from the API to GPU Execution
yuan_lei_e631e36865ed370b profile

Inside vLLM: Following One Request from the API to GPU Execution

#vllm #llm #inference #python
1
Comments 2
24 min read
Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4
xbill profile

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

#aws #sagemaker #gemma #vllm
7
Comments
9 min read
Gemma 4 on a Tesla T4, Part 2: The Minimum GCE VM and a Script to Drive It
xbill profile

Gemma 4 on a Tesla T4, Part 2: The Minimum GCE VM and a Script to Drive It

#gemma #vllm #gcp #cuda
13
Comments
13 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.