Investigating Streaming LLM Timeouts and Retry Boundaries

When an agent such as OpenCode calls local vLLM or a slow inference API, the first chunk may take a long time to arrive.

Treating streaming as independent requests per token leads to incorrect assumptions about timeouts and retries.

In reality, one HTTP request opens the connection and the response body arrives as multiple chunks.

Verify the communication unit

LLM streaming generally follows this path:

client
  │ POST /v1/chat/completions
  ▼
API server
  │ response headers
  │ data: token chunk
  │ data: token chunk
  │ data: [DONE]
  ▼
client

Each chunk is part of a response flowing through one HTTP connection.

After a mid-stream disconnect, the client cannot necessarily ask the server to continue from token N.

Both server and client must implement a protocol for resume position, generation state, and request identity.

Without that protocol, the client must rerun the original request from the beginning or treat it as failed.

Separate the overall timeout from the chunk interval

OpenCode’s current configuration documentation has timeout for the whole request and chunkTimeout for the interval between chunks.

The official documentation describes a default timeout of 300,000 milliseconds and false as the value that disables it.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "local-vllm": {
      "options": {
        "timeout": 600000,
        "chunkTimeout": 120000
      }
    }
  }
}

The two values do not need to match.

For a model with a slow first chunk, both the overall timeout and the chunkTimeout before the first chunk must be long enough.

In the current OpenCode source, each SSE read is monitored by chunkTimeout.

That includes the read waiting for the first chunk.

With chunkTimeout at 120 seconds, the request is interrupted after 120 seconds without a first chunk even when the overall timeout is 600 seconds.

A model that pauses during generation needs a chunkTimeout that allows the silent interval.

Increasing timeout also delays detection of a communication failure.

Making chunkTimeout too small can close a connection while the server is calculating normally.

Choose values from measured time to first chunk and the longest silent interval during generation.

Separate transport retry from semantic retry

Retry safety depends on when the connection failed.

With no observed response, the client cannot know whether the server started processing.

After a partial response, it cannot know whether the server finished or only the final chunk was lost.

Sending the same request again can therefore execute the operation twice.

For text generation, the main effects are compute time and cost.

Tool calls and external API writes can duplicate side effects as well.

At minimum, separate these two kinds of retry:

• Transport retry: resend the same input after a transient failure before connection or response observation.
• Semantic retry: start a new attempt with changed instructions because the received JSON or content is invalid.

The OpenCode LLM package design describes transient transport errors and rate limits as retry candidates.

It also describes avoiding implicit retries after observable output or tool side effects.

This behavior is version- and provider-SDK-dependent.

Check the configuration and logs for the version in use before relying on a retry boundary.

Isolate a local vLLM layer directly

When OpenCode calls local vLLM, send a request to the API server directly before blaming the harness.

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "MODEL_ID",
    "messages": [{"role": "user", "content": "Return a short response."}],
    "stream": true,
    "max_tokens": 32
  }'

If the direct stream is stable, inspect OpenCode’s provider configuration and startup logs next.

Separate the symptom into four categories:

1. The first chunk is slow
2. The gap between chunks is long
3. The connection itself drops after a fixed period
4. JSON validation or tool execution fails after the response

Cases 1 and 2 are candidates for timeout adjustment.

Case 3 requires finding whether a proxy, HTTP client, server, or network closes the connection.

Case 4 belongs to output validation and semantic retry, not streaming itself.

Track retries with identifiers

A client that repeats one logical operation should log at least three identifiers.

run_id=20260819-001
operation_id=12
transport_attempt=2
semantic_attempt=1

This distinguishes a resend caused by a disconnect from regeneration caused by invalid content.

For external writes, consider treating operation_id as a server-side idempotency key.

Do not log access keys, tokens, or personal data included in prompts without a deliberate policy.

Choose values from measurements

Do not begin with extremely large timeout values; measure the actual responses.

1. Send a short non-streaming request
2. Enable streaming and measure time to the first chunk
3. Measure the maximum chunk interval during generation
4. Set timeout above total processing time with margin
5. Set chunkTimeout above the maximum chunk interval with margin
6. Check connection drops, 429, 5xx, and output-format errors separately
7. Confirm retry count and boundary in logs

To reduce disconnects, adjust input length, maximum output length, and concurrency as well as timeout values.

Harness settings do not make server computation faster.

They define the boundary between normal computation and a stalled connection, and the unit to rerun after failure.

References

• OpenCode Config
• OpenCode LLM package design