llamacpp

Gemma 4's MTP Drafter, Explained: Speculative Decoding and a 4-Bit Drafter at 60% of the Size

Speculative decoding lets a 4-bit drafter run at 60% size, lowering inference costs. · for ml engineers, ai developers

Qwen and Gemma With Thinking On: Thinking Pays Qwen on Tool Calls, Gemma Still Leads

Qwen adds tool-call support while Gemma remains top performer. · for ml engineers, ai researchers

Do Drafters Work on Small Models? Gemma 4 E2B and E4B on Two Small GPUs

Benchmarks show Gemma models run on small GPUs, guiding low-cost deployment decisions. · for ml engineers, platform teams

Comparing Qwen and Gemma on the Low End: Qwen Is Faster Without Drafting, Gemma Has Better Quality

Low-end tests show qwen runs faster while gemma yields higher quality outputs. · for ml engineers, ai developers

A 4 GB Laptop GPU vs a 6-Core CPU on Gemma 4, Re-Measured in ABBA Order: 4.1x

Benchmark compares 4 GB GPU vs 6-core CPU on Gemma 4 using llama.cpp and CUDA. · for ml engineers, performance testers

Can a QAT 12B Fit an 8 GB GPU? Gemma 4 12B on an RTX 3050

Shows 12B model can run on 8 GB GPU, guiding resource planning for ML workloads. · for ml engineers, devops

Georgi Gerganov on llama.cpp/ggml future after Nvidia acquisition of HuggingFace

Georgi Gerganov discusses llama.cpp/ggml roadmap after Nvidia buys HuggingFace. · for AI researchers, devs

Llama.cpp on a jailbroken PS5: run any GGUF model natively on the console

Run GGUF models locally on PS5 for edge AI without cloud reliance. · for AI engineers

The Complete Guide to Local LLM Inference Tools in July 2026: llama.cpp, Ollama, vLLM, SGLang, and Beyond

Local LLM tools compared · for ai devs, researchers

Two Rust Clients for Gemma 4: Calling the Endpoint vs. Calling the MCP Server 🦀

Rust devs can choose endpoint or MCP server to integrate Gemma 4 models. · for rust developers

Gemma 4 at Over 70 Tokens/s on a 2021 Laptop's 4 GB GPU: The Live Demo, Step by Step

Gemma 4 runs at 70 tokens/s on a 4 GB laptop GPU, enabling local inference. · for ml engineers

Train Gemma 4 on Its Own Cut-Short Answers: Half the Tokens on an 8 GB RTX 3050

Fine-tune Gemma-4 on an RTX 3050, halving token usage for inference. · for ml engineers, AI founders

How Much Context Fits Beside a 12B on an 8 GB GPU? Gemma 4 12B's Context Cache on an RTX 3050

Gemma 4 12B model fits 8 GB GPU context cache using llama.cpp and CUDA on RTX 3050. · for ml engineers

SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated-GGUF (image-text-to-text)

Introduces a new image-text-to-text model format for faster inference. · for ml engineers

Faster prompt lookup drafting in llama.cpp

Optimized prompt lookup in llama.cpp speeds up LLM interactions for developers. · for AI engineers

Can a 26B MoE Run on an 8 GB GPU? Gemma 4 26B A4B on an RTX 3050

Gemma 26B runs on an RTX 3050 via llama.cpp and CUDA, showing consumer-grade inference is feasible. · for ML engineers, AI researchers

A Dense 31B on an 8 GB GPU: Gemma 4 31B on an RTX 3050

Running a 31B Gemma model on consumer GPUs shows feasibility of low-cost inference. · for ml engineers, hardware developers

Apple Silicon and macOS VMs: Faster LLM Inference with llama.cpp

Faster LLM inference · for ai devs

orcarouter/DeepSeek-V4.1-Flash-1017-abliterated-GGUF (text-generation)

DeepSeek V4.1 model released in GGUF format for local inference with llama.cpp. · for AI engineers, ML ops

orcarouter/OrcaSAQ-2-Cyber-27B-Uncensored-GGUF (text-generation)

OrcaSAQ-2-Cyber-27B uncensored model released in GGUF format for llama.cpp. · for ml researchers