Command R+ V2 Bet On Local And Won

Plot twist — Cohere doubled down on local-first RAG with Command R+ V2 and it paid off. If you've been living under a rock since the original Command R+, this is the model that finally gets tool-calling right out of the box.

The Setup

Weights are on HuggingFace under a research-only license. The tokenizer ships a proper tool-call chat template, the context window is 128k for real (not the marketing kind), and the JSON output rate is legitimately above 95% in my testing on the M4.

huggingface-cli download CohereLabs/c4ai-command-r-plus-v2 \\
  --local-dir ./models/command-r-plus-v2

# Needs roughly 2x A100 or 1x H100 to serve unquantized
# Q4_K_M GGUF fits in 64GB unified memory on an M4 Max

The Money Pattern

Behold — tool calls that don't need three layers of regex to parse. I wired it into a Pipedrive lookup function for a Rebuild Relief internal tool and it returned valid JSON on every single call across 500 test runs. The tools parameter in the chat template is the killer feature.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("CohereLabs/c4ai-command-r-plus-v2")
model = AutoModelForCausalLM.from_pretrained(
    "CohereLabs/c4ai-command-r-plus-v2",
    torch_dtype="auto",
    device_map="auto",
)

tools = [{
    "name": "lookup_deal",
    "description": "Fetch a Pipedrive deal by id",
    "parameter_definitions": {
        "deal_id": {"type": "int", "required": True},
    },
}]

msgs = [{"role": "user", "content": "What's the status of deal 8421?"}]
ids = tok.apply_chat_template(
    msgs, tools=tools, return_tensors="pt"
).to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0]))

The Catch

Spoiler — the weights are non-commercial. Research and personal use only. If you want Command R+ V2 in production you're calling the Cohere API and paying for it. The model is also chunky (104B params) so even the GGUF quants are not laptop-friendly unless you've got an M4 Max with 64GB+ unified memory.

The Verdict

For local RAG experiments and tool-calling research, Command R+ V2 is the new default. If you need to ship to production, use the API. Cohere bet on local-first being a feature, not a footnote, and for retrieval-heavy workflows it's the most aligned model I've used this year.