vllm

Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin

vLLM TT plugin enables efficient LLM serving on Tenstorrent hardware. · for ml engineers, infrastructure teams

Building and Serving vLLM with Rust

Demonstrates building and serving high-throughput vLLM inference pipelines using Rust for speed. · for backend engineers, ML infrastructure teams

Not Every LLM Needs vLLM: A Kubernetes Engineer's Guide to Serving Engines

Explains non-vLLM approaches to serve LLMs on Kubernetes, guiding ops and ML platform teams. · for devops engineers, ml platform teams

Self-hosting a lite agent backend on one TPU: Gemma 4 E2B + vLLM on a v5e-1

Run lite agent backend on one TPU · for devtools founders

Gemma 4 on a Tesla T4, Part 3: Int4 Embeddings Serve E2B in 2.86 GiB at 2.30x bf16

Gemma 4 on T4 delivers int4 embeddings at 2.30× bf16 speed in 2.86 GiB. · for ml engineers

Watermarking in vLLM

vLLM introduces watermarking to help identify AI-generated text. · for ai researchers, platform engineers

Speculative Decoding in vLLM on AMD GPUs

Speculative decoding in vLLM can accelerate LLM inference on AMD GPUs. · for ML engineers

The Complete Guide to Local LLM Inference Tools in July 2026: llama.cpp, Ollama, vLLM, SGLang, and Beyond

Local LLM tools compared · for ai devs, researchers

The Cheapest CUDA GPU on AWS Has an Arm CPU — and You Probably Want the Intel One

Cheapest AWS CUDA GPU uses Arm CPU, but Intel may be preferable for performance. · for ml ops

Serving Gemma4 with Rust on vLLM 🦀

Running Gemma4 on vLLM with Rust shows high-performance inference on AWS GPU instances. · for ML engineers

Comparing Open-Source LLM Gateways in 2026 to Run Enterprise AI at Scale

Compares open-source LLM gateways to help choose scalable enterprise inference stacks. · for ML platform teams

Gemma 4 on Amazon SageMaker: QAT Weights Decode 2.05x Faster Than bf16 on One L4

Quantization-aware weight decoding runs 2.05× faster than bf16 on a single L4 GPU. · for ml engineers, performance engineers

Installing Rust for vLLM on Graviton: a G5g walk-through 🦀

Guide to installing Rust-based vLLM on Graviton G5g instances for cost-effective inference. · for ML infrastructure engineers

Gemma 4 on Amazon SageMaker or a VM? The Same Model Server at 1.40x the Price

Running Gemma 4 on SageMaker versus a VM costs about 1.4× more. · for ML engineers, cloud ops

Renting Out One T4 to Three Customers: Gemma 4, vLLM Multi-LoRA and a Rust Gateway

Shares one T4 GPU among three workloads via Gemma 4, vLLM LoRA, and a Rust gateway. · for platform engineers, devops

Self-Hosting vLLM on Cloud GPUs in 2026: Sub-180ms LLM Inference for Autonomous AI Agents (Full Production Guide)

Guide details self-hosting vLLM on cloud GPUs for sub-180 ms LLM inference, supporting low-latency agents. · for ml engineers, platform teams

Small Apache-2.0 Models Against Gemma 4 E2B on One TPU v5e: Which Ones Keep Up?

Compares open-source models to Gemma on a single TPU v5e. · for ml engineers

vLLM v0.28.0

vLLM 0.28.0 improves LLM inference performance and resource usage. · for ml engineers, devops

How vLLM Actually Manages KV Cache (vs the Toy Version I Built)

LLM cache management · for ai engineers

Gemma 4 on a Tesla T4: QAT Weights Decode 1.79x Faster Than bf16

Gemma-4 on Tesla T4 with QAT decodes 1.79× faster than bf16. · for ML engineers, inference platform teams