STT for Agents

This page describes the WebSocket protocol for Krisp streaming speech-to-text (STT) with the krisp-realtime-stt-agents-v1 model, built for voice AI agents. It returns fast partial results and predicts the end of the user's turn.

Endpoint

• WebSocket URL: wss://streaming.krisp.ai/stt
• Auth: temporary session token. See Authentication.

Use the session token as the API key when opening the WebSocket. Do not send the master token on the WebSocket.

Authorization: api-key <session-token>

If your client cannot set headers, pass the same value as the authorization query parameter. See the API overview.

Connection and handshake flow

1) Client initiates connection + sends handshake message

After opening the WebSocket connection with the session token, the client must send an initial JSON handshake that defines the session.

Handshake message (client → server, JSON):

{
  "model": "krisp-realtime-stt-agents-v1",
  "vocabulary": ["word1", "word2"],
  "sample_rate": 16000,
  "channels": 1,
  "audio_format": "pcm_s16le",
  "turn": {
    "prediction": true,
    "sensitivity": 0.9,
    "max_silence": 2000,
    "silence_validation": 100
  },
  "metadata": {}
}

2) Server acknowledges readiness (or returns an error)

Once the server validates the handshake and is ready to receive audio, it responds with a ready message.

Ready response (server → client, JSON):

{
  "type": "ready",
  "request_id": "9f2c...",
  "tracks_ms": [2400, 1200, 480],
  "metadata": {}
}

• request_id: unique identifier for the streaming session. Include it if you contact Krisp support.
• tracks_ms: delays in ms of the decoders used for partial results. Each partial word's src is one of these values.
• metadata: echo of the metadata object from the handshake.

Do not send audio before this message. If no handshake is received within 15 seconds, the server closes the session.

Error response

If the server cannot accept the session, it responds with an error message and closes the connection. Besides the errors below, the common errors apply.

Error response (server → client, JSON):

{
  "type": "error",
  "request_id": "9f2c...",
  "error": {
    "code": "invalid_api_key",
    "status_code": 401,
    "message": "Invalid Authorization header.",
    "details": {
      "hint": "Check authorization header."
    }
  }
}

Audio streaming

3) Client streams audio frames (raw bytes)

After receiving type: "ready", the client sends raw audio bytes as binary WebSocket messages.

• Message type: binary (raw bytes)
• Encoding: according to audio_format in the handshake: pcm_s16le (PCM signed 16-bit little-endian) or opus
• Sample rate: 16,000 Hz
• Channels: 1 (mono)
• Recommended chunk duration: 80 ms, or multiples of 80 ms
• Maximum message size: 65,536 bytes

Stream audio in realtime. Do not wrap audio in JSON.

Server frames

Every server message is a JSON text frame, and all frames carry a type. Finals and partials are both transcript frames and differ in transcript.isFinal.

Times are milliseconds of audio from the session's first sample, except in turn frames, where they are seconds.

4) Final transcript: isFinal: true

{
  "type": "transcript",
  "metadata": {},
  "transcript": {
    "words": [
      {
        "text": " weather",
        "start": 12480,
        "end": 12880,
        "language": "en",
        "language_confidence": 0.9921
      }
    ],
    "isFinal": true
  }
}

• words holds one entry: a whole word with turn prediction, a word piece without it.
• text keeps its leading space, so joining text in arrival order gives the transcript.
• start and end bound the word. language and language_confidence are present when the server runs language ID.
• metadata is always empty.
• A final never changes and is never repeated.
• Finals trail speech by about 2.4 s, except at a turn end, when the turn's pending words arrive at once.

5) Partial transcript: isFinal: false

{
  "type": "transcript",
  "metadata": {},
  "transcript": {
    "words": [
      { "text": " is",    "start": 13040, "end": 13200, "src": 1200 },
      { "text": " nice",  "start": 13200, "end": 13600, "src": 480 },
      { "text": " today", "start": 13600, "end": 14000, "src": 480 }
    ],
    "isFinal": false
  }
}

• words is the whole partial region: everything after the last final, one entry per word.
• src is the delay in ms of the decoder that produced the word, one of ready.tracks_ms: 480 for the newest words, 1200 for words revised once.
• Each partial frame replaces all earlier partial content, and words: [] clears it.
• Show the finals followed by the latest partial frame.
• Partial words may change or disappear in the next frame.
• A partial frame is sent only when the words change.

6) Turn: type: "turn"

{
  "type": "turn",
  "confidence": 0.934,
  "reason": "model",
  "detected_at": 14560,
  "finalized_until": 14640,
  "processing_duration": 95
}

7) Force finalization

Force the end of a turn at any point during the stream:

{"type": "finalize"}

8) Stream end

To end the stream, the client sends one text frame after its last audio message:

{"type": "end_of_audio"}

The server then finishes the session and closes the socket itself. The client shouldn't close it; it should read frames until end arrives.

After end_of_audio, the last frame the server sends is end, and it closes the socket right after:

{
  "type": "end",
  "language": {
    "overall_language": "en",
    "overall_confidence": 0.982,
    "current_language": "en",
    "current_confidence": 0.991,
    "per_language": {
      "de": 0.003,
      "en": 0.982,
      "es": 0.004,
      "fr": 0.005,
      "ru": 0.003,
      "hy-AM": 0.003
    }
  }
}

The last frame of a session. language is present when the server runs language ID:

• overall_* averages the whole session.
• current_* covers the last 0.5 s.
• per_language holds each language's mean probability.

Example (Python)

Streams a 16 kHz, mono, 16-bit PCM WAV file at realtime pace and prints finals, partials and turn events. Requires pip install "websockets>=14".

import asyncio
import json
import sys
import wave

import websockets

SESSION_TOKEN = "YOUR_SESSION_TOKEN"  # see the Authentication page
URL = "wss://streaming.krisp.ai/stt"
CHUNK_MS = 80


async def main(path: str) -> None:
    headers = {"Authorization": f"api-key {SESSION_TOKEN}"}
    async with websockets.connect(URL, additional_headers=headers) as ws:
        await ws.send(json.dumps({
            "model": "krisp-realtime-stt-agents-v1",
            "sample_rate": 16000,
            "channels": 1,
            "audio_format": "pcm_s16le",
            "turn": {"prediction": True, "sensitivity": 0.9, "max_silence": 2000},
        }))
        hello = json.loads(await ws.recv())
        if hello["type"] != "ready":
            raise RuntimeError(hello)

        async def send_audio() -> None:
            with wave.open(path, "rb") as wav:
                frames = 16000 * CHUNK_MS // 1000
                while chunk := wav.readframes(frames):
                    await ws.send(chunk)
                    await asyncio.sleep(CHUNK_MS / 1000)
            await ws.send(json.dumps({"type": "end_of_audio"}))

        sender = asyncio.create_task(send_audio())
        async for message in ws:
            msg = json.loads(message)
            if msg["type"] == "transcript":
                kind = "final" if msg["transcript"]["isFinal"] else "partial"
                text = "".join(w["text"] for w in msg["transcript"]["words"])
                print(kind, text)
            elif msg["type"] == "turn":
                print("end of turn", msg["reason"], msg["confidence"])
            elif msg["type"] == "end":
                break
        await sender


asyncio.run(main(sys.argv[1]))

Changelog

• 2026-10-01: Page created.