LLM observability without selling your prompts to a vendor
If you've been living under a rock, Langfuse just rolled past 10k stars and it's the open-source LLM observability tool I keep recommending. Self-host on Netlify-adjacent infra or use their cloud. Either way, you get traces, evals, and prompt versioning for free.
The Setup
I bolted this onto a Pipedrive-driven LLM workflow at Rebuild Relief in about 20 minutes. The decorator pattern means you instrument once and forget.
{`pip install langfuse openai
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_HOST="https://cloud.langfuse.com" # or your self-host URL`}The Money Pattern
The @observe() decorator is the killer feature. Wrap any function and you get the full call tree in the UI — inputs, outputs, latency, token cost, error traces. Plug in the OpenAI wrapper and your model calls light up automatically.
{`from langfuse.decorators import observe
from langfuse.openai import openai # drop-in wrapper
@observe()
def classify_claim(transcript: str) -> str:
res = openai.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "Classify the hail damage severity."},
{"role": "user", "content": transcript},
],
)
return res.choices[0].message.content
@observe()
def claim_pipeline(transcript: str) -> dict:
label = classify_claim(transcript)
return {"label": label}`}The Catch
Self-hosting needs Postgres plus Clickhouse plus a Redis. It's not a single-binary affair like Plausible. If you don't have someone on the team who likes Docker Compose, just pay for the cloud tier — the free hobby tier is generous.
Their SDKs occasionally lag the latest OpenAI client. Pin versions or you'll get a 30-minute surprise.
The Verdict
If you're shipping anything with LLM calls in production and you don't have observability, you're flying blind. Langfuse is the one I'd pick today over Helicone, LangSmith, or PostHog's AI feature. Open source, sane UI, fair pricing on cloud. Install it before the next bug ships.