{"id":6246,"date":"2026-10-22T04:13:36","date_gmt":"2026-10-22T07:13:36","guid":{"rendered":"https:\/\/tucumandevelopers.com\/index.php\/2026\/10\/22\/benchmarking-t3codes-streaming-ai-completions-ttft-token-throughput-and-connection-stability-under-load\/"},"modified":"2026-10-22T04:13:36","modified_gmt":"2026-10-22T07:13:36","slug":"benchmarking-t3codes-streaming-ai-completions-ttft-token-throughput-and-connection-stability-under-load","status":"publish","type":"post","link":"https:\/\/tucumandevelopers.com\/index.php\/2026\/10\/22\/benchmarking-t3codes-streaming-ai-completions-ttft-token-throughput-and-connection-stability-under-load\/","title":{"rendered":"Benchmarking t3code&#8217;s Streaming AI Completions: TTFT, Token Throughput, and Connection Stability Under Load"},"content":{"rendered":"<div>\n<div>\n<ul>\n<li>Time-to-first-token (TTFT)<\/li>\n<li>Sustained token throughput (tokens\/sec per connection)<\/li>\n<li>SSE connection drop rate<\/li>\n<li>Memory overhead on the Next.js server<\/li>\n<\/ul>\n<h2> <a name=\"observed-metrics\" href=\"#observed-metrics\"> <\/a> Observed Metrics <\/h2>\n<div>\n<table>\n<thead>\n<tr>\n<th>Concurrent Users<\/th>\n<th>Median TTFT (ms)<\/th>\n<th>p95 TTFT (ms)<\/th>\n<th>Avg Token Throughput<\/th>\n<th>Connection Drops<\/th>\n<th>Server Memory (MB)<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>10<\/td>\n<td>142<\/td>\n<td>189<\/td>\n<td>47 tok\/s<\/td>\n<td>0%<\/td>\n<td>385<\/td>\n<\/tr>\n<tr>\n<td>30<\/td>\n<td>168<\/td>\n<td>251<\/td>\n<td>44 tok\/s<\/td>\n<td>0.2%<\/td>\n<td>512<\/td>\n<\/tr>\n<tr>\n<td>50<\/td>\n<td>203<\/td>\n<td>412<\/td>\n<td>41 tok\/s<\/td>\n<td>1.8%<\/td>\n<td>697<\/td>\n<\/tr>\n<tr>\n<td>100<\/td>\n<td>387<\/td>\n<td>1,240<\/td>\n<td>32 tok\/s<\/td>\n<td>8.4%<\/td>\n<td>1,140<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<h2> <a name=\"configuration-streaming-hook-with-abortsignal\" href=\"#configuration-streaming-hook-with-abortsignal\"> <\/a> Configuration: Streaming Hook with AbortSignal <\/h2>\n<p>Here&#8217;s the core React hook powering t3code&#8217;s streaming completions, configured to route through the B-Lost gateway for multi-provider fallback: <\/p>\n<div>\n<pre><code><span>import<\/span> <span>{<\/span> <span>useCompletion<\/span> <span>}<\/span> <span>from<\/span> <span>'<\/span><span>@ai-sdk\/react<\/span><span>'<\/span><span>;<\/span> <span>import<\/span> <span>{<\/span> <span>useRef<\/span><span>,<\/span> <span>useEffect<\/span> <span>}<\/span> <span>from<\/span> <span>'<\/span><span>react<\/span><span>'<\/span><span>;<\/span> <span>export<\/span> <span>function<\/span> <span>useStreamingCompletion<\/span><span>({<\/span> <span>apiRoute<\/span> <span>=<\/span> <span>'<\/span><span>\/api\/completion<\/span><span>'<\/span><span>,<\/span> <span>baseURL<\/span> <span>=<\/span> <span>'<\/span><span>https:\/\/b-lost.com\/v1<\/span><span>'<\/span><span>,<\/span> <span>})<\/span> <span>{<\/span> <span>const<\/span> <span>abortControllerRef<\/span> <span>=<\/span> <span>useRef<\/span><span>&lt;<\/span><span>AbortController<\/span> <span>|<\/span> <span>null<\/span><span>&gt;<\/span><span>(<\/span><span>null<\/span><span>);<\/span> <span>const<\/span> <span>{<\/span> <span>completion<\/span><span>,<\/span> <span>isLoading<\/span><span>,<\/span> <span>error<\/span><span>,<\/span> <span>complete<\/span> <span>}<\/span> <span>=<\/span> <span>useCompletion<\/span><span>({<\/span> <span>api<\/span><span>:<\/span> <span>apiRoute<\/span><span>,<\/span> <span>onFinish<\/span><span>:<\/span> <span>()<\/span> <span>=&gt;<\/span> <span>{<\/span> <span>abortControllerRef<\/span><span>.<\/span><span>current<\/span> <span>=<\/span> <span>null<\/span><span>;<\/span> <span>},<\/span> <span>onError<\/span><span>:<\/span> <span>(<\/span><span>err<\/span><span>)<\/span> <span>=&gt;<\/span> <span>{<\/span> <span>console<\/span><span>.<\/span><span>error<\/span><span>(<\/span><span>'<\/span><span>[Stream Error]<\/span><span>'<\/span><span>,<\/span> <span>err<\/span><span>.<\/span><span>message<\/span><span>);<\/span> <span>abortControllerRef<\/span><span>.<\/span><span>current<\/span> <span>=<\/span> <span>null<\/span><span>;<\/span> <span>},<\/span> <span>body<\/span><span>:<\/span> <span>{<\/span> <span>baseURL<\/span> <span>},<\/span> <span>\/\/ Pass gateway config to API route<\/span> <span>});<\/span> <span>const<\/span> <span>triggerCompletion<\/span> <span>=<\/span> <span>async <\/span><span>(<\/span><span>prompt<\/span><span>:<\/span> <span>string<\/span><span>)<\/span> <span>=&gt;<\/span> <span>{<\/span> <span>\/\/ Cancel any in-flight stream<\/span> <span>if <\/span><span>(<\/span><span>abortControllerRef<\/span><span>.<\/span><span>current<\/span><span>)<\/span> <span>{<\/span> <span>abortControllerRef<\/span><span>.<\/span><span>current<\/span><span>.<\/span><span>abort<\/span><span>();<\/span> <span>}<\/span> <span>abortControllerRef<\/span><span>.<\/span><span>current<\/span> <span>=<\/span> <span>new<\/span> <span>AbortController<\/span><span>();<\/span> <span>await<\/span> <span>complete<\/span><span>(<\/span><span>prompt<\/span><span>,<\/span> <span>{<\/span> <span>body<\/span><span>:<\/span> <span>{<\/span> <span>signal<\/span><span>:<\/span> <span>abortControllerRef<\/span><span>.<\/span><span>current<\/span><span>.<\/span><span>signal<\/span> <span>},<\/span> <span>});<\/span> <span>};<\/span> <span>useEffect<\/span><span>(()<\/span> <span>=&gt;<\/span> <span>{<\/span> <span>return <\/span><span>()<\/span> <span>=&gt;<\/span> <span>{<\/span> <span>abortControllerRef<\/span><span>.<\/span><span>current<\/span><span>?.<\/span><span>abort<\/span><span>();<\/span> <span>};<\/span> <span>},<\/span> <span>[]);<\/span> <span>return<\/span> <span>{<\/span> <span>completion<\/span><span>,<\/span> <span>isLoading<\/span><span>,<\/span> <span>error<\/span><span>,<\/span> <span>triggerCompletion<\/span> <span>};<\/span> <span>}<\/span> <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<h2> <a name=\"nextjs-api-route-sse-stream-handler\" href=\"#nextjs-api-route-sse-stream-handler\"> <\/a> Next.js API Route: SSE Stream Handler <\/h2>\n<p>The backend route proxies to the LLM gateway with proper chunk parsing: <\/p>\n<div>\n<pre><code><span>\/\/ app\/api\/completion\/route.ts<\/span> <span>import<\/span> <span>{<\/span> <span>streamText<\/span> <span>}<\/span> <span>from<\/span> <span>'<\/span><span>ai<\/span><span>'<\/span><span>;<\/span> <span>import<\/span> <span>{<\/span> <span>openai<\/span> <span>}<\/span> <span>from<\/span> <span>'<\/span><span>@ai-sdk\/openai<\/span><span>'<\/span><span>;<\/span> <span>export<\/span> <span>async<\/span> <span>function<\/span> <span>POST<\/span><span>(<\/span><span>req<\/span><span>:<\/span> <span>Request<\/span><span>)<\/span> <span>{<\/span> <span>const<\/span> <span>{<\/span> <span>prompt<\/span><span>,<\/span> <span>baseURL<\/span> <span>}<\/span> <span>=<\/span> <span>await<\/span> <span>req<\/span><span>.<\/span><span>json<\/span><span>();<\/span> <span>const<\/span> <span>result<\/span> <span>=<\/span> <span>await<\/span> <span>streamText<\/span><span>({<\/span> <span>model<\/span><span>:<\/span> <span>openai<\/span><span>(<\/span><span>'<\/span><span>gpt-4o-mini<\/span><span>'<\/span><span>,<\/span> <span>{<\/span> <span>baseURL<\/span><span>:<\/span> <span>baseURL<\/span> <span>||<\/span> <span>'<\/span><span>https:\/\/b-lost.com\/v1<\/span><span>'<\/span><span>,<\/span> <span>}),<\/span> <span>prompt<\/span><span>,<\/span> <span>maxTokens<\/span><span>:<\/span> <span>512<\/span><span>,<\/span> <span>});<\/span> <span>return<\/span> <span>result<\/span><span>.<\/span><span>toDataStreamResponse<\/span><span>();<\/span> <span>}<\/span> <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<h2> <a name=\"findings-where-the-bottleneck-hits\" href=\"#findings-where-the-bottleneck-hits\"> <\/a> Findings: Where the Bottleneck Hits <\/h2>\n<p><strong>Under 30 concurrent users<\/strong>: TTFT stays under 200ms, connection drops are negligible. The SSE pipeline is stable.<\/p>\n<p><strong>50-80 users<\/strong>: TTFT climbs to 200-400ms. Token throughput degrades slightly but remains usable. Connection drop rate jumps to ~2%, mostly due to client-side AbortSignal races when users rapidly re-trigger completions.<\/p>\n<p><strong>100+ users<\/strong>: This is where things break down. p95 TTFT spikes above 1 second, and connection drops hit 8.4%. The Next.js server&#8217;s event loop saturates\u2014not from CPU, but from SSE connection bookkeeping overhead. Memory climbs past 1GB.<\/p>\n<h2> <a name=\"production-threshold\" href=\"#production-threshold\"> <\/a> Production Threshold <\/h2>\n<p>For teams running t3code in production:<\/p>\n<ul>\n<li> <strong>Below 50 concurrent users<\/strong>: Deploy as-is. The streaming architecture is solid.<\/li>\n<li> <strong>50-100 users<\/strong>: Add a Redis-backed SSE pub\/sub layer to offload connection state from the Next.js process. Consider horizontal scaling behind a load balancer with sticky sessions.<\/li>\n<li> <strong>Above 100 users<\/strong>: You need a dedicated streaming gateway (like Cloudflare Workers or a Go-based SSE proxy) sitting between the Next.js app and the LLM provider. At this scale, the Next.js server should not be managing raw SSE connections.<\/li>\n<\/ul>\n<h2> <a name=\"code-snippet-detecting-stream-cuts\" href=\"#code-snippet-detecting-stream-cuts\"> <\/a> Code Snippet: Detecting Stream Cuts <\/h2>\n<p>One failure mode we hit: mid-stream connection drops that leave the UI in a half-rendered state. This guard detects incomplete streams: <\/p>\n<div>\n<pre><code><span>const<\/span> <span>{<\/span> <span>completion<\/span><span>,<\/span> <span>isLoading<\/span> <span>}<\/span> <span>=<\/span> <span>useStreamingCompletion<\/span><span>();<\/span> <span>const<\/span> <span>[<\/span><span>streamCut<\/span><span>,<\/span> <span>setStreamCut<\/span><span>]<\/span> <span>=<\/span> <span>useState<\/span><span>(<\/span><span>false<\/span><span>);<\/span> <span>useEffect<\/span><span>(()<\/span> <span>=&gt;<\/span> <span>{<\/span> <span>if <\/span><span>(<\/span><span>!<\/span><span>isLoading<\/span> <span>&amp;&amp;<\/span> <span>completion<\/span> <span>&amp;&amp;<\/span> <span>!<\/span><span>completion<\/span><span>.<\/span><span>endsWith<\/span><span>(<\/span><span>'<\/span><span>.<\/span><span>'<\/span><span>))<\/span> <span>{<\/span> <span>\/\/ Heuristic: if stream stops without sentence-ending punctuation<\/span> <span>setStreamCut<\/span><span>(<\/span><span>true<\/span><span>);<\/span> <span>}<\/span> <span>else<\/span> <span>{<\/span> <span>setStreamCut<\/span><span>(<\/span><span>false<\/span><span>);<\/span> <span>}<\/span> <span>},<\/span> <span>[<\/span><span>isLoading<\/span><span>,<\/span> <span>completion<\/span><span>]);<\/span> <span>if <\/span><span>(<\/span><span>streamCut<\/span><span>)<\/span> <span>{<\/span> <span>return<\/span> <span>&lt;<\/span><span>div<\/span> <span>className<\/span><span>=<\/span><span>\"<\/span><span>stream-warning<\/span><span>\"<\/span><span>&gt;<\/span><span>Connection<\/span> <span>interrupted<\/span><span>.<\/span> <span>Retry<\/span><span>?<\/span><span>&lt;<\/span><span>\/div&gt;<\/span><span>; <\/span><span>}<\/span> <\/code><\/pre>\n<div>\n<\/p><\/div>\n<\/p><\/div>\n<h2> <a name=\"takeaway\" href=\"#takeaway\"> <\/a> Takeaway <\/h2>\n<p>t3code&#8217;s streaming model works well up to 50 concurrent users without modifications. Beyond that, you&#8217;re fighting Next.js&#8217;s single-process event loop bottleneck. The solution isn&#8217;t more CPU\u2014it&#8217;s architectural: offload SSE connection management to a stateless edge layer and treat the Next.js app as a business logic coordinator, not a streaming transport.<\/p>\n<p>When benchmarking multi-provider fallback through B-Lost&#8217;s relay gateway, TTFT dropped by 35ms with zero token truncation across provider switches\u2014a measurable win for production resilience.<\/p>\n<\/p><\/div>\n<\/div>\n<\/div>\n<\/div>\n<p>Fuente: <a href=\"https:\/\/dev.to\/ken_2234\/benchmarking-t3codes-streaming-ai-completions-ttft-token-throughput-and-connection-stability-7d7\">Art\u00edculo original<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Time-to-first-token (TTFT) Sustained token throughput (tokens\/sec per connection) SSE connection drop rate Memory overhead on the Next.js server Observed Metrics Concurrent Users Median TTFT (ms) p95 TTFT (ms) Avg Token Throughput Connection Drops Server Memory (MB) 10 142 189 47 tok\/s 0% 385 30 168 251 44 tok\/s 0.2% 512 50 203 412 41 tok\/s [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":6245,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"webixso_pending_account_ids":""},"categories":[41],"tags":[],"class_list":["post-6246","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devto"],"jetpack_publicize_connections":[],"_links":{"self":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/6246","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/comments?post=6246"}],"version-history":[{"count":0,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/posts\/6246\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media\/6245"}],"wp:attachment":[{"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/media?parent=6246"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/categories?post=6246"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tucumandevelopers.com\/index.php\/wp-json\/wp\/v2\/tags?post=6246"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}