llama.cpp: High-Efficiency LLM Inference in Pure C/C++

What is llama.cpp?

llama.cpp is a lightweight, zero-dependency C/C++ implementation of transformer architectures created by Georgi Gerganov. It serves as the core inference engine powering downstream tools like Ollama, LM Studio, and Jan.

By implementing custom tensor math in pure C and leveraging low-level SIMD instructions (AVX2, AVX-512, ARM NEON), llama.cpp enables models to execute with exceptional performance on consumer hardware.

Core Technical Advantages

• Zero Third-Party Dependencies: Compiles into lightweight static binaries without Python, PyTorch, or large CUDA SDK dependencies.
• Hybrid Offloading: Split model layers dynamically between system RAM and GPU VRAM via the -ngl (number of GPU layers) flag.
• State-of-the-Art Quantization: Supports GGUF quantization levels from full 16-bit float down to 1.5-bit integer quants (IQ1_S, IQ2_XXS, Q4_K_M).
• Apple Silicon Native: Delivers near 100% memory bandwidth saturation on M-series unified memory via optimized Metal shaders.

Compilation & Installation

Option 1: Build with CMake

# Clone the repository
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# For CPU only (with AVX2)
cmake -B build
cmake --build build --config Release -j

# For NVIDIA CUDA GPU acceleration
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Option 2: Pre-Compiled Binaries

Download pre-built release packages matching your operating system from the official GitHub Releases page.

Battle-Tested CLI Recipes

1. Interactive Terminal Prompt with GPU Offload

# Offload 33 layers to GPU, set context length to 4096
./build/bin/llama-cli \
  -m models/llama-3.1-8b-instruct.Q4_K_M.gguf \
  -p "Explain the CAP theorem with distributed database examples:" \
  -n 512 \
  -c 4096 \
  -ngl 33 \
  -fa

2. Standalone OpenAI-Compatible Server

llama.cpp includes a high-performance web server (llama-server) with built-in chat UI and OpenAI endpoints:

./build/bin/llama-server \
  -m models/llama-3.1-8b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  -c 8192 \
  -ngl 99 \
  -fa

Once running, send requests to http://localhost:8080/v1/chat/completions:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.1-8b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'