Engineering

Streaming AI answers through your server

Streaming makes an AI rewrite feel fast, but the provider key belongs on your server. What the browser sends, what the endpoint must check, and how to test it.

6 min read

An AI rewrite that takes eight seconds feels broken if nothing appears until the end. Streaming fixes the wait: the first words arrive in under a second and the rest follow. The easy way to get streaming is to call the AI provider straight from the browser, and that is the way to avoid. This article explains why the request should pass through your server, and what the editor and the server each do.

Why the browser should not call the provider

The shape of a streamed request

The browser sends one request to an endpoint on your server. Your server calls the provider with its own key, and passes each piece of the answer back as it arrives. Server-Sent Events are the usual transport: one HTTP response that stays open, with a small frame for each piece of text, then a final frame with the complete structured result, and a closing frame.

In the editor, the AI Toolkit has a helper for the browser side. It reads the stream, calls you back for each piece, and can be cancelled:

Code example
var stream = editor.aiToolkit.streamRequest({
    url: "/richtextbox/ai/stream",
    body: { mode: "rewrite", selectionText: "..." },
    onDelta:    function (chunk, full) { /* show the text so far */ },
    onResponse: function (payload)     { /* the final structured result */ },
    onDone:     function (fullText)    { },
    onError:    function (err)         { }
});
// stream.abort()  cancels mid-response

The helper uses fetch with a readable stream, and falls back to a normal request in a browser that cannot stream. The exact frame format, and the ASP.NET Core endpoint that produces it, are in the AI providers and streaming documentation.

What the server endpoint must do

  1. Authenticate the user with your normal session, exactly as for any other write request.
  2. Limit the input. Cap the size of the text you forward, and cap the answer length. Both are what you pay for.
  3. Apply a rate limit per user, not only per IP address.
  4. Stop when the user stops. When the browser cancels, cancel the provider call too. Otherwise you pay for an answer nobody reads.
  5. Turn off buffering. A proxy or a compression layer that buffers the response delivers the whole answer at once, which defeats streaming.
  6. Record usage for each request: who, which mode, and the token counts the provider reports.

Do not write streamed text straight into the document

Show the streamed text in a preview, and change the document only when the user accepts it. A stream can fail halfway, and half a rewritten paragraph is worse than none. Treat the final text like any other untrusted input: sanitize it before it becomes part of the stored document. The stored XSS article covers the checks.

A short test list

For how the editor keeps provider keys on your server, see the bring-your-own-key AI editor page, and for the costs, see AI usage and pricing.


RichTextEditor is a perpetual-licence JavaScript editor — one purchase, self-hosted, no metered editor loads. Download the evaluation or see how it compares.