Vllm - DEV Community

Skip to content

DEV Community

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Oct 6

KV Cache Quantization in LLM Serving: FP8 and INT8 Tradeoffs, the Silent config.json Trap, and How to Measure It Fairly

#llm #vllm #performance #devops

7 min read

Oct 6

What Does It Actually Cost to Self-Host an LLM? The Batching Math Nobody Shows You

#llm #vllm #performance #devops

6 min read

Oct 5

Five Ways Your LLM Serving Benchmark Is Lying to You (and How to Catch Each One)

#llm #vllm #performance #devops

15 min read

The Homelab Postmortem

Oct 3

vLLM's weight cache can serve another checkpoint's weights when the tensor layout matches

#llm #vllm #selfhosted #debugging

4 min read

The Homelab Postmortem

Oct 3

vLLM ignores LoRA rank_pattern and alpha_pattern and serves the adapter at the wrong scale

#llm #vllm #lora #selfhosted

5 min read

Reno Lu

Oct 2

Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default

#vllm #localllm #speculativedecoding #quantization

4 min read

ram.mehta.1899@gmail.com

Oct 1

Evaluating Speculative Decoding in vLLM on AMD MI300X GPUs

#vllm #amd #mi300x #rocm

1 min read

xbill for AWS Community Builders

Sep 30

Gemma 4 on Amazon SageMaker: 4-Bit Embeddings Decode up to 1.39x Faster on One L4

#aws #sagemaker #gemma #vllm

11 min read

xbill for AWS Community Builders

Sep 30

Gemma 4 on Amazon SageMaker or a VM? The Same Model Server at 1.40x the Price

#aws #sagemaker #gemma #vllm

8 min read

xbill for AWS Community Builders

Sep 25

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

#aws #sagemaker #gemma #vllm

9 min read

xbill for AWS Community Builders

Sep 30

Gemma 4 on Amazon SageMaker: The NVIDIA T4 Decodes at 0.8x of the L4 With the Same Answers

#aws #sagemaker #gemma #vllm

9 min read

Vishnu Hari Dadhich

Sep 14

Run vLLM on Kubernetes with Minikube, WSL2 and NVIDIA GPU

#ai #localllm #vllm #kubernetes

13 min read

Rost

Sep 11

KV Cache on 16 GB GPUs: Making Long Context Actually Fit

#llm #llamacpp #vllm #ollama

22 min read

yuan lei

Sep 7

Inside vLLM: Following One Request from the API to GPU Execution

#vllm #llm #inference #python

24 min read

xbill for Google Developer Experts

Sep 26

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

#aws #sagemaker #gemma #vllm

9 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.