Streaming LLM API responses: events, tool calls and error handling

Streaming lets a language model API send its answer in small pieces while it is still generating, instead of making your app wait for the whole response. Both the Claude API and OpenAI’s Responses API do it with server-sent events (SSE): you turn on a stream flag and read a series of typed events. This guide explains the event flow, how to handle streamed tool calls and what can go wrong mid-stream, so you can build interfaces that feel fast and clients that fail gracefully.

Why stream at all

A long answer can take many seconds to finish. Without streaming, the user sees nothing until the last token is ready. With streaming, the first words appear almost at once, which improves perceived speed even though total generation time is the same. Streaming also helps with long outputs that might hit a request timeout, because bytes keep arriving on the connection.

The trade-off is complexity. You handle a sequence of events, you may need to stitch together partial data, and an error can arrive after you have already shown part of the answer.

How server-sent events work

SSE is a simple HTTP pattern: the server keeps the response open and writes lines such as event: message_stop followed by a data: line containing JSON. Browsers have a built-in EventSource API for it (see the MDN reference), but for model APIs you normally use the provider’s SDK, which parses the stream into typed objects. In most SDKs you enable streaming with stream=True (Python) or stream: true (JavaScript) and then loop over the events.

The event flow in the Claude API

According to Anthropic’s documentation, each stream follows the same order:

1. message_start carries a Message object with empty content.
2. One or more content blocks follow. Each has a content_block_start, one or more content_block_delta events and a content_block_stop. The index matches the block’s position in the final message.
3. One or more message_delta events report top-level changes, such as the stop reason. The token counts in usage are cumulative.
4. A final message_stop ends the stream.

The stream may also include ping events at any point. Treat them as keep-alives and ignore them. The documentation notes that new event types may be added over time, so your code should skip unknown types instead of crashing.

A text block delivers text_delta events with the next fragment of text. Appending these in order rebuilds the answer.

The event flow in OpenAI’s Responses API

OpenAI describes semantic, typed events. Some fire once per response and others repeat. For text, the common ones are response.created (once, at the start), response.output_text.delta (repeatedly, each with a new text fragment), response.completed (once, at the end) and error. Other events cover function-call arguments, refusals and built-in tool activity such as file search and code interpreter; the API reference lists them all. A typical loop checks each event’s type, prints text deltas, logs completion and reports errors.

Streaming tool calls

Tool use is where streaming gets trickier. In the Claude API, a tool_use block streams its input as input_json_delta events whose partial_json values are fragments of a JSON string, not valid JSON by themselves. The final tool_use.input is always an object. The documentation suggests accumulating the string and parsing it when the content_block_stop event arrives, using a partial-JSON parser, or using the SDK’s streaming helpers, which can also expose the parsed input incrementally in the Python and TypeScript SDKs.

The rule of thumb: never execute a tool call from a partial argument string. Wait until the block is complete, validate the arguments against your schema (see our guide to structured outputs and function calling) and only then run the tool.

Handling errors mid-stream

Errors can arrive inside an open stream. Anthropic gives the example of an overloaded_error sent as an error event during periods of high usage, which in a non-streaming call would be an HTTP 529. Because the HTTP status was already 200 when the stream began, you must check for error events in the loop, not just the status code.

A practical approach:

• Keep the text received so far, so you can show it or save it.
• Decide per case whether to retry the whole request or give the user a retry button. Retrying silently can duplicate text you already displayed.
• Use exponential backoff with jitter, as described in our guide to reliable LLM API clients.
• Set timeouts on idle connections, not only on total duration.

Other caveats

OpenAI warns that moderating partial completions is harder, which may affect approved usage, and that moderation scores requested with a generation arrive only after the full output is available, not in the deltas. If your product filters output, plan for checks on the assembled text as well as on fragments.

If you only need the finished message and not the live display, Anthropic’s SDKs also offer a way to stream under the hood and return the final message.

A production checklist

• Use the official SDK to parse events instead of splitting lines by hand.
• Handle ping and unknown event types without failing.
• Accumulate tool-call JSON and parse only at the end of the block.
• Check for error events inside the loop.
• Log the request ID and the final usage numbers.
• Test cancellation: what happens if the user closes the tab mid-answer.

New to API calls? Start with our first LLM API call guide.

What to watch next

• New event types added to provider streams; follow each provider’s changelog.
• SDK helpers that parse partial tool arguments for you.
• Provider support for persistent connections as an alternative to SSE; OpenAI’s docs mention a WebSocket mode.
• Better ways to moderate and evaluate output as it streams.

Key takeaways

• Streaming sends output as typed server-sent events so users see text immediately.
• Claude streams follow message_start, content blocks, message_delta and message_stop.
• OpenAI streams use events such as response.output_text.delta and response.completed.
• Tool-call arguments arrive as partial JSON; parse them only after the block ends.
• Errors can occur after a 200 status, so handle error events inside the loop.

Frequently asked questions

Does streaming make the model faster?

No. Total generation time is similar, but users see the first words sooner, which makes the app feel faster.

Why is the tool-call JSON invalid while streaming?

Each delta carries only a fragment of the argument string. Accumulate the fragments and parse the JSON when the content block finishes.

Can I get the final message without handling every event?

Yes. Anthropic’s SDKs provide helpers that stream internally and return the completed message, which is useful for long outputs.