Medical practices can't send patient notes to the cloud. Legal firms can't ship case files to an API. Financial advisors can't trust a third-party with client portfolios. Velocity X handles this: flip an env var and Brain routes to Ollama running Llama 3 70B on your laptop or internal server instead of Anthropic's API. No cloud calls. No data leaving your walls. Trade-off is real — you lose ~30% of Claude's nuance — but for compliance-first teams, it's the only move that works.
Why On-Device AI Matters
Cloud LLMs are cheap and powerful, but they have one non-negotiable cost: your data goes up to a third-party server. For regulated industries, that's a legal blocker. GDPR doesn't care that OpenAI says "we don't train on your data" — the moment data leaves your server, you've broken the control plane. Medical data under HIPAA, legal work under attorney-client privilege, financial advice under fiduciary rules — all of it demands local processing.
Ollama solves this. Run Llama 3 70B locally (or on your internal infrastructure), call it via HTTP from Velocity X, and data never touches the internet. Your compliance audit says "yes, approved" instead of "maybe, with a BAA".
Ollama Setup — 5 Minutes
Download Ollama from `ollama.ai` for macOS, Windows, or Linux. Pull the 70B model (45GB, first-time only):
ollama pull llama2:70bStart the server on port 11434 (default, can change):
ollama serveFor production, run it in a Docker container or systemd service so it stays running. For on-prem teams, deploy to an internal server (GPU-accelerated, ideally — M3 Max or RTX 4080 for under 100ms latency per token). Llama 3 70B requires ~48GB RAM; 8B variant works on 16GB laptops, 13B on 24GB.
Llama 3 70B vs Claude Sonnet — The Trade-Off
Llama 3 70B is Meta's state-of-the-art before Llama 3.1. Benchmarks show ~70% of Claude Sonnet's performance on reasoning, summarization, and coding. Where it shines: instruction following, zero-shot classification, and creative writing. Where it stumbles: nuanced interpretation, edge-case handling, and multi-step logic. For medical intake forms and legal document summaries, it's solid. For high-stakes financial advice and complex risk analysis, you'd want oversight.
Speed-wise, Llama 3 70B on an RTX 4080 generates ~40 tokens/second. Claude Sonnet is instant (cloud latency). If your reps need responses in sub-second time, on-device isn't viable. If "ready by the time you finish typing" is fine, local works.
Wiring It Into Velocity X Brain
Velocity X's Brain accepts a `MODEL_PROVIDER` env var. Set `PROVIDER=ollama` and `OLLAMA_URL=http://localhost:11434`, and all API calls route locally instead of to Anthropic:
# .env.local
MODEL_PROVIDER=ollama
OLLAMA_URL=http://localhost:11434
OLLAMA_MODEL=llama2:70bBrain's system prompt, caching logic, and output parsing all stay the same. The only difference is the backend. Swap the env var at runtime and you can A/B test or fallback gracefully if Ollama goes down.
Frequently Asked Questions
Do I need a GPU?
No, but you'll suffer. Llama 3 70B on CPU runs at ~3 tokens/second. With an RTX 4090 or M3 Max (with unified memory), you hit 40+ tokens/sec. For a team waiting for drafts, GPU is non-negotiable. For batch jobs overnight, CPU is fine.
What about Llama 3.1?
Llama 3.1 405B exists but needs 880GB VRAM. For local, stick with 70B. If you need more power, either pay for cloud (defeats the privacy goal) or run a smaller quant (8-bit quantized 70B fits on 24GB GPUs, loses ~5% accuracy).
Can I use other models?
Yes. Ollama supports Mistral, Phi, Orca, Code Llama. Performance varies wildly. Llama 3 70B is the sweet spot for general-purpose work. For specialized domains (medical, legal, code), try fine-tuned variants.
What if data still leaks on my internal network?
This is paranoia, but justified. Run Ollama on an airgapped machine with no internet connection. If your team needs to access it, use a secure tunnel (wireguard VPN, not the public internet). Your compliance officer will sleep better.
Cost-wise, how does this compare?
One-time GPU cost (~$2k for a solid RTX 4080) plus electricity (~$1/day). Claude Sonnet at 1,000 requests/day costs ~$18/day. Break-even is ~110 days. After that, local is essentially free. For teams that'll use this for 2+ years, on-device is cheaper.
Can I use this for production SaaS?
Not if you have customers in different jurisdictions. But for internal tools and regulated B2B services, absolutely. Velocity X's Brain supports this natively — you choose the model, not us.
The Bottom Line
Cloud LLMs are the default because they're easy. But "easy" doesn't work for regulated industries. Ollama + Llama 3 70B removes that blocker. You lose some quality and add latency, but you gain complete control and zero compliance friction. For medical, legal, and financial teams, that trade-off is a win. Set up Ollama, point Velocity X at it, and your data never leaves your walls. See the pricing page for Brain's model-agnostic architecture, or read how Velocity X handles GDPR + Australian Privacy Act compliance to understand the full privacy story.