Why You Keep Getting Stalled at the Gateway Again

Last week I helped a three-person team debug an issue: they were calling a large language model for long-text summarization. Their local script worked fine, but once the traffic went through the company's Nginx gateway, the response would inexplicably break off, and the frontend only received half the text. Three people spent an entire afternoon digging through the code, only to find the problem wasn't in the code at all — the gateway's default 60-second timeout was cutting off responses that hadn't finished streaming.

This is not an isolated case. I went through several indie developer projects I've encountered recently and found that with the streaming response + timeout configuration combo, on average each person spends an extra 2–4 hours of debugging on their first integration. And within those 2–4 hours, actual coding time is usually under 20 minutes — the rest is burned on trial and error over "why does it work locally but not through the gateway."

The goal of this article is simple: get streaming responses working through a gateway within 1 hour, and know exactly what value each timeout parameter should be set to.

First, Understand One Thing: What Streaming Responses Look Like to a Gateway

For non-streaming requests, the gateway's logic is plain: wait for the upstream to return the complete response, then forward it to the client in one shot. A 60-second timeout has a clear meaning.

But streaming responses (SSE, Server-Sent Events) are different. Once the connection is established, the server continuously pushes data chunks, and a single long answer may last tens of seconds or even minutes. Here the gateway has two easy-to-hit pitfalls:

1. Total duration timeout: The entire connection is not allowed to exceed N seconds, so long answers get cut off directly.
2. Idle timeout: If the gap between two data chunks exceeds N seconds, the gateway considers the connection dead. The "thinking period" before a large model generates long content, or times of high load, can push chunk intervals to over ten seconds.

Of the incidents I've witnessed, about 70% are the first type and 30% the second. And both look very similar: the client silently disconnects after receiving partial content, and the log only shows one vague line: upstream timed out.

Simplify This with a Unified Gateway

Rather than maintaining a pile of vendor SDKs and dealing with different timeout behaviors for each, a more time-saving approach is to go through a unified API gateway where all models are accessed via the same OpenAI-compatible interface. This is why I recommend ThisToken.AI: it consolidates multiple models under a single base_url, with unified streaming behavior and predictable timeout semantics.

Do two things first:

1. Visit https://api.thistoken.ai/register to sign up (email only, done in a few minutes);
2. Create an API Key in the console and keep it safe.

Billing follows the official pricing page; I won't go into that here.

A Piece of Code You Can Run Directly

The Python code below demonstrates a streaming request through the gateway with proper timeout handling. The only dependency is openai:

pip install openai

import time
from openai import OpenAI

client = OpenAI(
    api_key="你的_API_KEY",              # 替换为 ThisToken.AI 的 API Key
    base_url="https://api.thistoken.ai/v1",
    timeout=120,          # 客户端总超时：流式场景要放宽
    max_retries=2,        # 网关自动重试，减少手动处理
)

start = time.time()
first_token_at = None
chars = 0

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "你是一个技术写作助手。"},
        {"role": "user", "content": "用300字解释什么是SSE流式响应。"},
    ],
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta
    if delta and delta.content:
        if first_token_at is None:
            first_token_at = time.time()
        chars += len(delta.content)
        print(delta.content, end="", flush=True)

print("\n---")
print(f"首字延迟: {first_token_at - start:.2f}s")
print(f"总耗时:   {time.time() - start:.2f}s")
print(f"输出字符: {chars}")

Once it runs, you'll see text printed chunk by chunk, along with timing stats at the end. Those stats are worth a look: the first-token latency is usually far smaller than the total duration — that's exactly the value of streaming. If your product manager thinks "the answer takes 30 seconds," but in reality users see the first character within 1 second, the experience is a completely different story.

A Three-Layer Timeout Checklist

Once the code works, align these three layers of timeouts, and gateway stream cutoffs will mostly disappear:

Layer 1: Client timeout. That's the timeout=120 in the code above. For streaming scenarios, start at 120 seconds; don't carry over the 30 seconds commonly used in non-streaming scenarios.

Layer 2: Gateway/reverse proxy timeout. If you have Nginx in front, focus on these three settings:

proxy_read_timeout 300s;   # 两个数据块之间的最大等待
proxy_send_timeout 300s;
proxy_buffering off;       # 关键！关掉缓冲，否则流式变"假流式"

proxy_buffering off is the most easily overlooked item: with buffering on, Nginx accumulates data before forwarding, and the frontend experience is a long blank pause followed by the entire text dumped at once — making streaming pointless.

Layer 3: Client retry strategy. Network jitter can happen on any gateway, and configuring max_retries is far easier than writing your own retry loop. Note: only retry requests that haven't started producing output; retrying a request that's already halfway through streaming will duplicate content — in that case, it's better to let the upper-layer business logic re-initiate.

Doing the Time Math

Take the debugging session mentioned earlier: the three-person team spent roughly 12 person-hours in one afternoon on the stream cutoff issue. Following this article's checklist — 5 minutes to register a gateway account, 15 minutes to get the example running, 30 minutes to align the three timeout layers — one person can finish within 1 hour, and once configured, it applies to all subsequent models.

Compare: from 12 person-hours down to 1, what you save isn't just that one debugging session, but also the repeated debugging every time you integrate a new model in the future. For a small team, this kind of structural saving — configure once, benefit long-term — is often more valuable than one-off cost reductions.

Summary

Streaming responses getting cut off by a gateway is almost never the model's fault — it's misaligned timeouts across three layers. Remember three things: loosen the client timeout to 120 seconds or more, disable buffering on the gateway and extend the read timeout, and use the SDK's built-in retries instead of hand-written loops.

If you don't have an account yet, you can register at https://api.thistoken.ai/register and use the code in this article to get your first streaming response working — I'd suggest turning on the timing stats in the code first, so you can see the gap between first-token latency and total duration with your own eyes. You'll gain a much more intuitive feel for the value of streaming.

---

Ready to try it yourself? Sign up at https://api.thistoken.ai/register to get your API key and start building.