Your SaaS is live. A customer reports: "Orders are timing out." You log in to see what happened. Error logs say "Stripe timeout". Okay. But which endpoint timed out? Was it your code waiting on Stripe, or Stripe waiting on you? Which database query was slow? How many users hit this? Your logs don't know. You dig through three systems (error tracker, database logs, APM) and lose 15 minutes finding the answer. By then the customer's frustrated and tweeting.
This is the log trap. Logs are events — they're snapshots. They don't connect. "User 5023 hit error" exists in isolation. "Database query took 2 seconds" lives in a separate log file. "Stripe API call failed" is a different system entirely. Observability is the practice of connecting these dots. When an order times out, a single trace follows the entire request: from your API endpoint → database → Stripe → response. Every function call, database query, and API call is annotated. You see where time was spent and where the failure happened. One view. One answer. Five minutes instead of fifteen.
Logs alone won't scale a SaaS. You need distributed tracing, metrics, and logs unified. OpenTelemetry is the open standard. Honeycomb is the SaaS that makes it actually useful. And if Honeycomb's pricing doesn't fit your runway, Axiom and Grafana Tempo exist. Here's the pattern Velocity X uses and how to pick the tool for your stage.
What Is Observability, Really?
Observability = three types of data working together: (1) logs = discrete events ("user signed in", "payment processed"), (2) metrics = aggregates over time ("99th percentile latency", "error rate", "database pool size"), (3) traces = request flows across services ("where did this request spend its time?"). You can have all three and still be blind if they don't talk to each other. Observability means they're correlated: a spike in error rate traces back to specific log entries and metric changes. You see the full story, not fragments.
The core problem observability solves: in production, you can't reproduce the bug. A user's request to your API timed out. It'll never time out in your dev environment. You can't step through it. You need to reconstruct what happened using the data that flowed through your system. Logs, metrics, and traces are the breadcrumbs. Observability tools read the breadcrumbs and tell you the story.
Why Logs Alone Fail (and When Traces Matter)
The log ceiling
Logs scale linearly with request volume. High-traffic SaaS = millions of log lines per day. You pay to store them. You pay to index them. You pay to query them. And every log line is disconnected. A request that hits three services generates three separate log entries. To understand what happened, you manually correlate them by user ID or request ID. At 1000 RPS, that's thousands of requests to correlate every minute. Logs don't scale here because they're not designed to. They're designed for humans to read, not for systems to connect.
Traces: one request, full visibility
A trace is one request's journey. It starts at your API handler, flows through middleware, database, external API call, cache check, response. Each step is a "span" — a named unit of work with a start time, end time, and attributes (user ID, database table, error message). Spans are hierarchical: the main request span contains child spans for each operation. You don't need to correlate manually. The trace structure IS the correlation. One request ID ties them all together.
When traces pay for themselves
Latency debugging: User says "checkout is slow." Look at the trace for that checkout request. You see: database query (200ms) → Stripe API (3000ms) → response (100ms). Stripe is slow. Not your code. You tell support "Stripe's rate limiting you today", show the trace, customer feels heard. Fixed in one minute instead of investigating false leads for an hour.
Distributed systems: logs fail completely
Your API calls a payments service. Payments service calls Stripe. Stripe takes 5 seconds. Log trail: API logs "timeout", payments service logs "Stripe timeout", Stripe doesn't log anything (it's external). Three systems, three disconnected events. A trace? Single path: API span → payments span → Stripe span, all linked. You see the full chain instantly.
OpenTelemetry: The Open Standard
What it is
OpenTelemetry (OTel) is a set of SDKs and standards for collecting traces, metrics, and logs from your application. You import the OTel library for your language (Node, Python, Go, Java, etc), instrument your code to create spans, and export them to a backend (Honeycomb, Axiom, Jaeger, Tempo, etc). OTel is not a storage backend — it's the tool you use to generate the data. Think of it as "the Sentry for traces" — vendor-agnostic, open-source, maintained by the Linux Foundation.
OTel setup: Node.js example
npm install @opentelemetry/api @opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http
// tracing.js — load this FIRST, before any other imports
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: 'https://api.honeycomb.io/v1/traces',
headers: {
'x-honeycomb-team': process.env.HONEYCOMB_API_KEY,
},
}),
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();Call this module first (before Express, database, anything else). OTel auto-instruments: HTTP requests, database queries (Postgres, MySQL, MongoDB), external API calls, and middleware. Every operation becomes a span. You export to Honeycomb's OTLP endpoint with your API key. That's it.
Add custom spans for business logic
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('my-app');
app.post('/checkout', async (req, res) => {
const span = tracer.startSpan('checkout-flow');
try {
// Your business logic
const span2 = tracer.startSpan('validate-cart', { parent: span });
// ... validation code
span2.end();
const span3 = tracer.startSpan('create-payment', { parent: span });
const payment = await stripe.payments.create({ ... });
span3.addEvent('payment_created', { paymentId: payment.id });
span3.end();
span.addEvent('checkout_complete', { orderId: order.id });
span.setStatus({ code: SpanStatusCode.OK });
res.json({ success: true });
} catch (error) {
span.recordException(error);
span.setStatus({ code: SpanStatusCode.ERROR });
res.status(500).json({ error: error.message });
} finally {
span.end();
}
});Wrap your business logic in named spans. Log events ("payment created"). On error, record the exception. Honeycomb will show the full flow: checkout-flow → validate-cart, create-payment, both children of the main span. If payment creation fails, you see it instantly in the trace waterfall.
Three Dashboards You'll Actually Use
Dashboard 1: Latency by Endpoint
Query: group spans by http.target (URL endpoint), plot p50/p95/p99 latency. Add a threshold line at "acceptable" (200ms for API, 1000ms for reporting). Instantly see which endpoint is slow. Filter by time range, service, environment. If checkout is slow, you drill into that endpoint's traces and see the 10 slowest requests. Dashboard alerts if p99 latency exceeds threshold.
Dashboard 2: Error Rate by Service
Query: count spans where status.code == ERROR, group by service.name. Stack chart with time on x-axis. Spike = outage. Hover to see which service. Drill in to see the error traces. This is your at-a-glance health check. If payments service has an error spike, alert goes to Slack and you've got a link to the traces.
Dashboard 3: Slow Database Queries
Query: filter spans where db.system == postgres, sort by duration descending. Show top 20 slowest queries with their counts. If you see SELECT * FROM users WHERE id = $1 taking 5 seconds, that's a missing index. The traces show you exactly which queries hurt performance. Fix the index, check the dashboard, latency drops. Proof-of-impact in one view.
Honeycomb vs Axiom vs Grafana Tempo: Pick Your Tool
Honeycomb: The SaaS Gold Standard
Price: $50-500/month depending on span volume. Free tier: 20 million spans/month. Best for: Velocity X and mature SaaS. Honeycomb's UI is purpose-built for traces. Click a spike in the latency graph, it auto-drills into the traces that caused it. Their "Derived Columns" feature lets you extract values from spans and group by them (e.g., extract user_id from span attributes, then count errors per user). The UX is built for ops teams that live in observability dashboards. Trade-off: Most expensive, but the time saved debugging is worth it for revenue-generating products.
Axiom: The Budget Pick
Price: $25-200/month for similar span volume. Best for: Smaller SaaS or early-stage products. Axiom ingests traces (OTel format), stores them, and lets you query via their UI or API. Less polished than Honeycomb's UX, but 70% of the functionality at 50% the cost. You can still see latency, errors, and drill into traces. Trade-off: Fewer built-in alerting features, no auto-drilling. You manually construct queries more often.
Grafana Tempo: The DIY Route (if you run Prometheus)
Price: Free (self-hosted) or ~$150/month (Grafana Cloud). Best for: Teams that already run Prometheus and Grafana for metrics. Tempo integrates with Grafana, so traces live next to your metrics dashboards. Trade-off: You manage the infrastructure. Scaling Tempo requires operational overhead. Start with Honeycomb or Axiom unless you're already deep in the Grafana ecosystem.
Side-by-side comparison
| Feature | Honeycomb | Axiom | Grafana Tempo |
|---|---|---|---|
| Base Price | $50/month | $25/month | Free (self-hosted) |
| OTel Support | Yes (native) | Yes (native) | Yes (native) |
| UI/UX Quality | Excellent | Good | Good (if you know Grafana) |
| Auto-drill Latency | Yes | No | No |
| Alerting | Built-in | Minimal | Yes (Grafana alerts) |
| Best For | Revenue-Stage SaaS | Startups, Budget-Conscious | Existing Prometheus Users |
The Aidxn Pattern: Velocity X Observability Stack
Early stage (pre-revenue, <100K requests/day): Start with Axiom. Ingest OTel traces, build your three dashboards, and learn what observability looks like. Cost is $25-50/month. When your traffic justifies it, migrate to Honeycomb (they have a migration tool).
Revenue stage (100K-1M requests/day): Switch to Honeycomb. The $200-400/month cost is negligible against a $50K MRR product. You'll save that in ops time. Use Honeycomb's Derived Columns to track business metrics (errors per user, slow transactions by customer tier). Pair with Sentry for client-side error monitoring — they're complementary. Sentry catches React errors, Honeycomb gives you the backend story.
Mature SaaS (1M+ requests/day): Stick with Honeycomb for traces. Add Prometheus for metrics (CPU, memory, request counts). Keep your three dashboards in Honeycomb, add a metrics dashboard in Prometheus/Grafana. Logs go to Better Stack (or your chosen log aggregator). You now have a three-tier observability stack: traces (Honeycomb) → metrics (Prometheus) → logs (Better Stack). Each tool does one thing well.
Six FAQs
Q: Do I need traces if I already have error monitoring (Sentry)?
A: Yes. Sentry tells you "error happened". Traces tell you "here's the full request that caused it, with latency for every step." They're complementary. Sentry catches exceptions; traces catch slowness, timeouts, and chains of events. A timeout might not be an exception (your code didn't error, the request just took too long). Sentry won't see it. A trace will.
Q: How much data will I generate?
A: ~100 bytes per span on average. If you generate 1000 requests/second and each request creates 10 spans, that's 10K spans/sec = 1 billion spans/day = 100GB/month of raw data. Honeycomb compresses this to ~5-10GB/month stored. Axiom similar. Budget for 50 million spans/month per $100 of your observability bill and you'll be close.
Q: What if Honeycomb's slow to query?
A: Honeycomb's queries are fast (sub-second for time ranges <7 days). If you're running complex queries on large time ranges, they'll take 10-30 seconds. That's acceptable for on-demand analysis. If you need real-time alerting, use their SLO product (pre-computed) or drop to a sampled dashboard (show 1-in-100 traces by default, drill into full data on demand). Sampling can hurt accuracy. Honeycomb's "Intelligent Sampling" feature uses ML to keep the important traces. Axiom and Tempo have similar sampling strategies.
Q: Do I need to instrument every function in my code?
A: No. OTel's auto-instrumentation covers HTTP, databases, and popular libraries. You only add custom spans for business logic that matters: checkout flow, payment processing, user signup. Anything that directly impacts revenue or UX. Auto-instrumentation gives you 80% of the insights for 0% of the code.
Q: Can I use traces for feature flags or A/B testing?
A: Indirectly. Add a span attribute experiment=control or experiment=variant. Query traces filtered by experiment. Compare latency and error rates between groups. Honeycomb calls this "Experiment Tracking" (paid feature). For DIY, you can do it manually with good dashboards. Better approach: use PostHog or Statsig for A/B testing and let them integrate with your traces for context.
Q: What if I'm using a managed backend (Firebase, Supabase, Lambda)?
A: You still need observability. Firebase has its own Monitoring product (limited). Supabase integrates with Honeycomb. Lambda works with OTel if you use Node/Python runtimes. The managed service traces are often limited — add OTel instrumentation to your application code to supplement. For example, OTel traces your API code and database calls; Lambda Insights traces the Lambda runtime overhead. Together you get the full picture.
The Bottom Line
Logs told you what happened. Traces tell you why and how long. Start with OpenTelemetry auto-instrumentation and Axiom at $25/month. Build your three dashboards (latency, errors, slow queries). Once you're profitable, migrate to Honeycomb for the superior UX. The time you save debugging is worth the upgrade. Pair with Sentry for client-side errors and you've got a complete observability stack. Your customers will thank you — their issues get fixed faster and you'll catch problems before they complain.
Ready to instrument your SaaS? Aidxn Design can help you set up OTel, build dashboards, and scale your observability stack from day one, or dive deeper with our error monitoring guide.