Local AI

If you've been living under a rock, text-generation-webui — affectionately known as Oobabooga — was the original Swiss Army knife for running local LLMs. Plot twist: v2 quietly fixed the UI and now it's actually pleasant.

The Setup

v2 ships a redesigned interface, multi-backend support out of the box — llama.cpp, ExLlamaV2, Transformers, AutoGPTQ, AutoAWQ — and an OpenAI-compatible API that you can hit from anything. One launcher, every quant format, zero excuses.

{`git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui

# One-click installer, picks GPU/CPU automatically
./start_macos.sh   # or start_linux.sh / start_windows.bat

# Drop a GGUF into ./models/ and load it from the UI
# API server runs on http://localhost:5000`}

The Money Pattern

The OpenAI-compatible endpoint is the real unlock. Swap your base URL and your existing code just works — including streaming, function-calling shims, and embeddings if you load an embedding model.

{`curl http://localhost:5000/v1/chat/completions \\
  -H "Content-Type: application/json" \\
  -d '{
    "model": "llama-3-70b-instruct.Q4_K_M.gguf",
    "messages": [
      {"role": "system", "content": "You are concise."},
      {"role": "user", "content": "explain Supabase RLS in 5 bullets"}
    ],
    "temperature": 0.3,
    "max_tokens": 400
  }'`}

The Catch

It still feels like a power-user tool. The settings page has more sliders than a Eurorack synth, the model loader has six dropdowns, and the "characters" / "instruct templates" system is great if you know what it does and bewildering if you don't. The default theme is fine but not Jan-level pretty.

The Verdict

For tinkerers, prompt engineers, and anyone who wants every quant format and backend in one app, text-generation-webui v2 is still the king. Jan is prettier, LM Studio is friendlier, but neither has Ooba's depth. If you're running anything weird on a local M4 or a 4090, this is the cockpit you want.