Streaming LLM Responses — Why You Need Server-Sent Events Right Now

If you're building a chat interface, a code-generation tool, or anything that talks to Claude, GPT, or Gemini, users are staring at a spinner while the API churns. 8 seconds to get a full LLM response from Anthropic or OpenAI feels like forever on the internet. But there's a silent revolution happening: streaming. Server-Sent Events (SSE) pipes tokens from the LLM to the browser in real time. First token lands in ~200ms. Users see the response building in front of them. You lose the dead UX. Here's how Aidxn ships it.

Why Streaming Matters — It's Not Just Speed

An 8-second spinner doesn't feel fast; it feels broken. Streaming flips the narrative. Users see text appearing token-by-token — a 2000-token response feels snappy because they're reading it as it arrives. Psychology matters: perceived speed doubles with streaming because the feedback loop is constant.

Technically, streaming cuts latency-to-first-interaction from 8000ms to ~200ms. That's a 40x improvement. Your product feels responsive. Secondly, streaming reduces bandwidth and memory on the server: instead of buffering a full 4kb response, you chunk it in 100-byte bites. Thirdly, users can abort midway if the model is going off the rails. And fourthly — the hidden win — streaming works with any LLM provider that supports it (Anthropic, OpenAI, Google Gemini, Hugging Face). Build once, use anywhere.

The Architecture: Netlify Function + EventSource

Here's the pattern Aidxn uses: Netlify Edge Function returns a ReadableStream. Client opens an EventSource connection. Tokens arrive as text/event-stream. Done.

Server: Netlify Function Streaming

Your Netlify function (or any Node backend) receives the request, calls the LLM API, and pipes the response directly back. If you're using Anthropic's SDK:

export default async (req, context) => {
  const { prompt } = await req.json();

  const stream = await fetch("https://api.anthropic.com/v1/messages/streaming", {
    method: "POST",
    headers: {
      "anthropic-version": "2023-06-01",
      "content-type": "application/json",
      "x-api-key": process.env.ANTHROPIC_API_KEY,
    },
    body: JSON.stringify({
      model: "claude-3-5-sonnet-20241022",
      max_tokens: 1024,
      messages: [{ role: "user", content: prompt }],
    }),
  });

  // Return the stream directly; Netlify handles SSE
  return new Response(stream.body, {
    headers: {
      "content-type": "text/event-stream",
      "cache-control": "no-cache",
      "connection": "keep-alive",
    },
  });
};

Netlify's runtime unwraps the ReadableStream and pipes it back to the client as HTTP chunks. The browser receives `content-type: text/event-stream`, and JavaScript can consume it. That's it — no middleware, no buffering.

Client: EventSource Consumer

On the frontend, open an EventSource and listen for message events:

const eventSource = new EventSource(
  `/api/stream?prompt=${encodeURIComponent(userPrompt)}`
);

let fullResponse = "";

eventSource.onmessage = (event) => {
  const data = JSON.parse(event.data);

  // Anthropic sends `type: "content_block_delta"` with text delta
  if (data.type === "content_block_delta") {
    fullResponse += data.delta.text;
    // Update UI in real time
    document.getElementById("response").innerText = fullResponse;
  }

  // End of stream
  if (data.type === "message_stop") {
    eventSource.close();
  }
};

eventSource.onerror = () => {
  console.error("Stream failed");
  eventSource.close();
};

Every message from the server is JSON — you parse it, extract the text delta, and append to your UI. The browser does the rendering; you just feed it tokens.

Real Example: Claude Streaming with Abort

Here's the production-ready pattern with abort signal support (so users can stop the response mid-generation):

const abortController = new AbortController();

const handleStreamResponse = async (prompt) => {
  const response = await fetch("/api/stream", {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ prompt }),
    signal: abortController.signal,
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let fullText = "";

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value);
    const lines = chunk.split("\n");

    for (const line of lines) {
      if (line.startsWith("data: ")) {
        const json = JSON.parse(line.slice(6));
        if (json.type === "content_block_delta") {
          fullText += json.delta.text;
          // Update UI
          setResponse(fullText);
        }
      }
    }
  }
};

// User hits "stop" button
const handleStop = () => {
  abortController.abort(); // Network request terminates immediately
};

EventSource doesn't support abort signals natively, so we use fetch + ReadableStream instead. When the user clicks stop, the abort controller terminates the network connection and the stream stops. No dangling requests.

Error Handling & Streaming Edge Cases

If the LLM API fails mid-stream, the client sees an incomplete response. Handle it gracefully: wrap the stream consumer in a try-catch, show an error state if the stream closes unexpectedly, and let users retry. Some providers send error events as JSON in the stream itself — parse the `type` field and check for `error` or `message_stop` to distinguish success from failure.

Also: streaming changes rate-limiting semantics. A single request now spans 2–10 seconds (the time it takes to receive all tokens). If you're rate-limiting by request count, you might accidentally allow ten concurrent streams when you meant five. Rate-limit by token count instead: track `total_input_tokens` + `total_output_tokens` from the LLM response metadata and deduct from a user's quota.

Six FAQs

Why not WebSockets?

WebSockets are bidirectional, which is useful for chat applications where users send multiple messages back-to-back. For single-request streaming (send prompt once, get response), SSE is simpler: it's plain HTTP, CORS works as expected, and you don't need a persistent socket. If you're building a real-time collaborative editor with LLM, use WebSockets. For 90% of AI products, SSE wins.

Does streaming work with all LLM providers?

Anthropic: yes, native SSE support. OpenAI: yes, supports streaming via response format. Google Gemini: yes, supports streaming. Ollama (local LLMs): yes. If your provider has a "stream" parameter in the API, you can stream. Older models that don't support streaming will return the full response at the end — no benefit, but no extra cost either.

What's the overhead of streaming vs buffered responses?

Negligible. Streaming adds ~50ms of I/O overhead per chunk (the time to parse and transmit), but saves 8 seconds of wall-clock time waiting for completion. Bandwidth is identical — you're sending the same tokens either way. Memory is lower on streaming (no buffer). Pick streaming unless you have a good reason not to.

Can I stream multiple responses in parallel?

Yes. Open multiple EventSource or fetch connections simultaneously. Each gets its own stream. Rate-limiting is per-API-key, so if you hit your provider's quota, both streams backpressure equally. In production, queue parallel requests rather than fire them all at once — gives you a bit more graceful degradation if the API is slow.

How do I know when streaming is complete?

The server sends a message with `type: "message_stop"` (Anthropic) or similar terminal event. Some providers send `[DONE]` for OpenAI. Read your provider's streaming format docs. If the connection closes without a terminal event, treat it as an error and show "response incomplete" to the user.

Does streaming work with server-side rendering (SSR)?

Streaming is client-only: the browser opens the connection, receives chunks, updates the DOM. SSR doesn't apply — your Astro/Next.js page loads, renders a UI shell, and then JavaScript takes over to populate it with streamed LLM responses. The pattern is transparent to your SSR framework. Works great.

The Bottom Line

Stop making users wait for LLM responses. Streaming flips UX from "slow and silent" to "responsive and alive". Build a Netlify function that pipes the LLM API's response directly back, consume it on the client via fetch + ReadableStream, and update the DOM as tokens arrive. Add abort signals so users can stop early. Handle errors gracefully. Done — your product now feels 40x faster than the competition.

Streaming is table stakes for AI products shipped in 2026. Implement it first week, not last. For deeper dives on LLM API selection (which provider supports streaming best?), see OpenAI vs Anthropic vs Gemini API 2026 — Pick the LLM API That Fits Your Stack. Building a multi-model LLM product? Check Aidxn Design services for guidance on production streaming patterns.