Open Source

While the internet was busy losing its mind over the next ChatGPT update, llama.cpp quietly crossed 100,000 GitHub stars. It is now one of the most-used AI repositories on the planet and most developers have never opened the source.

The Setup

llama.cpp is Georgi Gerganov's single-purpose C++ project: run LLMs as fast as possible on whatever silicon you have lying around. CPU, CUDA, Metal, Vulkan, ROCm — it runs on all of it. It invented the GGUF file format that every local AI tool now uses.

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_METAL=1 -j8

# Run a quantized model
./main -m models/llama-3-8b-q4.gguf -p "Why is C++ better than you think?"

The Money Pattern

Ollama, LM Studio, Jan, KoboldCpp, GPT4All, and basically every "run AI locally" app you've ever heard of are wrappers around llama.cpp. When you pull a GGUF model from Hugging Face, that's llama.cpp's format. The whole local AI ecosystem stands on this one repo.

# Run llama.cpp's built-in OpenAI-compatible server
./server -m models/llama-3-8b-q4.gguf -c 4096 --host 0.0.0.0 --port 8080

# Then point any OpenAI SDK at http://localhost:8080/v1

The Catch

The build flags are a maze. Every backend has its own combination of CMake options, env vars, and gotchas. Breaking changes happen — GGUF v2 broke models, the new server flags broke scripts, and the README assumes you already know what a KV cache is.

It's also extremely C++. If you wanted Python ergonomics, you're in the wrong repo.

The Verdict

llama.cpp is the closest thing local AI has to Linux: invisible infrastructure that everything else runs on. 100K stars is the moment to acknowledge that one person in a basement is doing more for AI accessibility than the entire combined output of San Francisco's AI hype machine. Throw ggerganov a sponsorship.