> ## Documentation Index
> Fetch the complete documentation index at: https://60db.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# STT WebSocket

> Real-time Speech-to-Text WebSocket API for streaming audio transcription

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# STT WebSocket API

Real-time Speech-to-Text transcription via WebSocket streaming with support for 99+ languages and telephony integration.

## 🚀 Quick Start (Copy & Paste)

```javascript theme={null}
const WebSocket = require('ws');

// 1. Your API key
const API_KEY = 'sk_live_your_api_key';

// 2. Connect
const ws = new WebSocket(`wss://api.60db.ai/ws/stt?apiKey=${API_KEY}`);

// 3. Handle messages
ws.on('message', (data) => {
  const msg = JSON.parse(data);

  // Authenticated? Start session!
  if (msg.connection_established) {
    console.log('✅ Authenticated');
    ws.send(JSON.stringify({
      type: 'start',
      languages: ['en'],
      config: { encoding: 'mulaw', sample_rate: 8000, continuous_mode: true }
    }));
  }

  // Session ready? Send audio!
  if (msg.type === 'connected') {
    console.log('✅ Ready! Send audio now');

    // Send dummy audio (480 bytes every 60ms)
    let count = 0;
    const interval = setInterval(() => {
      ws.send(Buffer.alloc(480, 0xff));
      if (++count >= 83) {  // 5 seconds
        clearInterval(interval);
        ws.send(JSON.stringify({ type: 'stop' }));
      }
    }, 60);
  }

  // Got text!
  if (msg.type === 'transcription' && msg.is_final) {
    console.log('📝', msg.text);
  }

  // Done!
  if (msg.type === 'session_stopped') {
    console.log('✅ Complete! Cost:', msg.billing_summary.total_cost);
    ws.close();
  }
});
```

**That's it!** You'll see:

* ✅ Authenticated
* ✅ Ready! Send audio now
* 📝 Hello world (transcribed text)
* ✅ Complete! Cost: \$0.000043

***

## 📖 How It Works (5 Simple Steps)

1. **Connect** with your API key
2. **Send** `{ type: "start", ... }` to begin session
3. **Stream** audio data (binary chunks)
4. **Receive** text transcriptions in real-time
5. **Stop** with `{ type: "stop" }` when done

***

## Endpoint

<ParamField name="url" type="string">
  `ws://api.60db.ai/ws/stt` or `wss://api.60db.ai/ws/stt`
</ParamField>

## Authentication

Query parameter authentication:

<ParamField name="apiKey" type="string" required>
  Your API key for authentication
</ParamField>

Example:

```
ws://api.60db.ai/ws/stt?apiKey=sk_live_your_api_key
```

## Connection Details

| Property       | Value                                                |
| -------------- | ---------------------------------------------------- |
| Protocol       | WebSocket (RFC 6455)                                 |
| Frame types    | Binary (telephony) or Text/JSON (browser)            |
| Ping/keepalive | Server sends WebSocket pings every 30s (timeout 10s) |

## Session Lifecycle

```
Client                          Server
  │                               │
  │──── TCP/TLS connect ─────────►│
  │◄─── {"connecting": true} ─────│  authenticating...
  │◄─── connection_established ──│  authenticated
  │                               │
  │──── {"type":"start", ...} ───►│
  │◄─── {"type":"connected"} ─────│  session ready
  │                               │
  │──── audio frames / messages ─►│
  │◄─── {"type":"speech_started"} │  VAD detected voice
  │◄─── {"type":"transcription"}  │  is_final=true, speech_final=true
  │                               │
  │──── {"type":"stop"} ─────────►│
  │◄─── {"type":"session_stopped"}│
```

## Client → Server Messages

### `start` — Begin session

Sent once after connection is established. Must be sent before any audio.

<RequestExample>
  ```json Request theme={null}
  {
    "type": "start",
    "languages": ["en", "hi"],
    "config": {
      "encoding": "mulaw",
      "sample_rate": 8000,
      "utterance_end_ms": 500,
      "continuous_mode": true,
      "interim_results_frequency": 300,
      "no_speech_threshold": 0.60
    }
  }
  ```
</RequestExample>

**Parameters:**

<ParamField name="languages" type="array">
  Array of Whisper language codes e.g. `["en", "hi"]`. Omit or send `null` for auto-detect across all 99+ languages. Default: `null` (auto)
</ParamField>

<ParamField name="config.encoding" type="string" default="mulaw">
  Audio encoding format. Use `"mulaw"` for telephony/Twilio, `"linear"` for browser PCM. Options: `"mulaw"`, `"linear"`
</ParamField>

<ParamField name="config.sample_rate" type="integer" default="8000">
  Actual sample rate of the audio being sent. Server resamples to 16kHz internally. Options: `8000`, `16000`, `24000`, `44100`, `48000`
</ParamField>

<ParamField name="config.utterance_end_ms" type="integer" default="500">
  Silence duration (ms) after last speech chunk before finalizing the utterance. **Minimum 300ms**. Recommended 500-1000ms for voicebots. Range: `≥ 300`
</ParamField>

<ParamField name="config.continuous_mode" type="boolean" default="false">
  Keep session alive between utterances instead of stopping after first transcription. Required for voicebot / phone call use cases.
</ParamField>

<ParamField name="config.interim_results_frequency" type="integer">
  How often (ms) to emit interim (partial) transcription results during speech. Use 300ms for barge-in, 500ms otherwise. Disabled by default. Range: `≥ 300`
</ParamField>

<ParamField name="config.no_speech_threshold" type="float" default="0.60">
  Whisper no-speech probability threshold. Higher = more aggressive noise rejection. Range: `0.0–1.0`
</ParamField>

### `audio` — JSON audio chunk (browser mode)

<RequestExample>
  ```json Request theme={null}
  {
    "type": "audio",
    "audio": "<base64-encoded Int16 PCM or μ-law bytes>",
    "encoding": "linear",
    "sample_rate": 48000,
    "timestamp": 1700000000000
  }
  ```
</RequestExample>

**Fields:**

<ParamField name="type" type="string" required>
  Must be `"audio"`
</ParamField>

<ParamField name="audio" type="string" required>
  Base64-encoded audio bytes (Int16 PCM or μ-law)
</ParamField>

<ParamField name="encoding" type="string" required>
  `"linear"` or `"mulaw"`
</ParamField>

<ParamField name="sample_rate" type="integer" required>
  Actual sample rate of the audio
</ParamField>

<ParamField name="timestamp" type="integer">
  Unix ms timestamp — useful for latency measurement
</ParamField>

### Binary frame — raw μ-law audio (telephony mode)

Send a raw WebSocket binary frame with μ-law bytes, no JSON wrapper.
The server auto-detects this as telephony mode on the first binary frame.

```
Recommended chunk size: 480 bytes = 60ms at 8kHz
Twilio default: 160 bytes = 20ms — batch 3 chunks into 60ms before sending
```

### `config` — Change language mid-session

<RequestExample>
  ```json Request theme={null}
  {
    "type": "config",
    "languages": ["hi"],
    "continuous_mode": true
  }
  ```
</RequestExample>

Both `languages` and `continuous_mode` are optional; include only fields you want to change.
Send `"languages": null` to revert to auto-detect.

### `stop` — End session

<RequestExample>
  ```json Request theme={null}
  {
    "type": "stop"
  }
  ```
</RequestExample>

Server processes any remaining audio buffer, sends `session_stopped`, then closes.

### `test` — Ping / latency check

<RequestExample>
  ```json Request theme={null}
  {
    "type": "test",
    "message": "ping",
    "timestamp": 1700000000000
  }
  ```
</RequestExample>

Server echoes `test_response` with the same `timestamp` for round-trip measurement.

## Server → Client Messages

### `connecting` — Authentication in progress

<ResponseExample>
  ```json Response theme={null}
  {
    "connecting": true,
    "message": "Authenticating...",
    "timestamp": 1775465918269
  }
  ```
</ResponseExample>

### `connection_established` — Authentication successful

<ResponseExample>
  ```json Response theme={null}
  {
    "connection_established": {
      "service": "stt",
      "user_id": 43,
      "credit_balance": 9.97,
      "workspace": "default"
    }
  }
  ```
</ResponseExample>

**Fields:**

<ResponseField name="service" type="string">
  Service name: `"stt"`
</ResponseField>

<ResponseField name="user_id" type="integer">
  Your user ID
</ResponseField>

<ResponseField name="credit_balance" type="number">
  Available credits
</ResponseField>

<ResponseField name="workspace" type="string">
  Workspace name
</ResponseField>

### `connected` — After `start` message is processed

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "connected",
    "server_info": {
      "server_type": "Modular Sentence-based STT",
      "device": "cuda",
      "model": "large-v3",
      "processing_mode": "sentence_based_modular",
      "supported_languages": { "en": "English", "hi": "Hindi" },
      "total_languages": 99,
      "features": {
        "sentence_based_processing": true,
        "real_time_streaming": true,
        "telephony_support": true,
        "unicode_support": true,
        "mixed_language_support": true
      }
    }
  }
  ```
</ResponseExample>

### `speech_started` — VAD detected voice activity

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "speech_started",
    "timestamp": 1700000000.123
  }
  ```
</ResponseExample>

Use this for barge-in: interrupt TTS playback when this arrives.
Fired after 2 consecutive VAD-positive chunks (\~64ms of confirmed speech).

### `transcription` — Transcription result

All results (interim and final) share the same `transcription` type — differentiate with flags.

**Final result** (`is_final=true`, `speech_final=true`):

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "transcription",
    "text": "Hello, how are you?",
    "confidence": 0.87,
    "language": "en",
    "language_name": "EN",
    "is_final": true,
    "speech_final": true,
    "is_partial": false,
    "sentence_id": 3,
    "processing_mode": "sentence_complete",
    "duration": 1.82,
    "latency": 0.43,
    "timestamp": 1700000000.456,
    "words": [
      { "word": "Hello", "start": 0.0, "end": 0.32, "probability": 0.98 },
      { "word": "how", "start": 0.35, "end": 0.52, "probability": 0.94 }
    ],
    "utterance_end_ms": 1820
  }
  ```
</ResponseExample>

**Empty speech\_final signal** (`text=""`, `is_final=true`, `speech_final=true`):
Sent when audio was detected but transcription was rejected (silence, hallucination, low confidence, wrong language).
Client should reset its state on this message and not treat it as an error.

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "transcription",
    "text": "",
    "confidence": 0.0,
    "is_final": true,
    "speech_final": true,
    "processing_mode": "speech_end_no_result",
    "timestamp": 1700000000.789
  }
  ```
</ResponseExample>

**Interim result** (`is_final=false`, `speech_final=false`) — only sent when `interim_results_frequency` is set:

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "transcription",
    "text": "Hello how",
    "confidence": 0.72,
    "language": "en",
    "is_final": false,
    "speech_final": false,
    "is_partial": true
  }
  ```
</ResponseExample>

Use interims only for barge-in word-count checks. Never send interim text to the LLM — a final with `is_final=true, speech_final=true` will follow.

**Response Fields:**

<ResponseField name="text" type="string">
  Transcribed text. Empty string = speech-end-no-result signal.
</ResponseField>

<ResponseField name="confidence" type="number">
  0.0–1.0. Telephony typically 0.35–0.75; browser 0.55–0.95.
</ResponseField>

<ResponseField name="language" type="string">
  Detected language code e.g. `"en"`.
</ResponseField>

<ResponseField name="language_name" type="string">
  Uppercase language code e.g. `"EN"`.
</ResponseField>

<ResponseField name="is_final" type="boolean">
  `true` = complete utterance transcription or end-of-speech signal.
</ResponseField>

<ResponseField name="speech_final" type="boolean">
  `true` when speech segment fully processed. Always matches `is_final` for finals.
</ResponseField>

<ResponseField name="is_partial" type="boolean">
  `true` for interim results only.
</ResponseField>

<ResponseField name="sentence_id" type="integer">
  Monotonically increasing counter per session.
</ResponseField>

<ResponseField name="duration" type="number">
  Duration (seconds) of the audio segment transcribed.
</ResponseField>

<ResponseField name="latency" type="number">
  Seconds from processing start to result ready (excludes queue time).
</ResponseField>

<ResponseField name="words" type="array">
  Word-level timestamps `[{word, start, end, probability}]`. Present on finals when available.
</ResponseField>

<ResponseField name="utterance_end_ms" type="integer">
  Timestamp (ms) of last word in the utterance.
</ResponseField>

<ResponseField name="processing_mode" type="string">
  Internal mode string — useful for debugging.
</ResponseField>

### `language_changed` — After `config` message changes language

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "language_changed",
    "language": "Multi-language: HI",
    "language_code": ["hi"]
  }
  ```
</ResponseExample>

### `mode_changed` — After `config` message changes `continuous_mode`

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "mode_changed",
    "continuous_mode": true,
    "mode_name": "continuous",
    "silence_threshold": 0.5
  }
  ```
</ResponseExample>

### `session_stopped` — After `stop` is processed

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "session_stopped",
    "billing_summary": {
      "total_duration_seconds": 5.2,
      "total_cost": 0.000043,
      "characters_transcribed": 42
    }
  }
  ```
</ResponseExample>

### `error` — Processing error

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "error",
    "error": "Audio processing error: ...",
    "timestamp": 1700000000.0
  }
  ```
</ResponseExample>

### `test_response` — Reply to `test` ping

<ResponseExample>
  ```json Response theme={null}
  {
    "type": "test_response",
    "message": "pong - Sentence-based STT ready",
    "timestamp": 1700000000000,
    "processing_mode": "sentence_based"
  }
  ```
</ResponseExample>

## Complete Example

<Tabs>
  <TabItem value="javascript" label="JavaScript (Node.js)">
    ```javascript theme={null}
    const WebSocket = require('ws');

    const API_KEY = 'sk_live_your_key';
    const ws = new WebSocket(`ws://api.60db.ai/ws/stt?apiKey=${API_KEY}`);

    ws.on('open', () => {
      console.log('✓ Connected');
    });

    ws.on('message', (data) => {
      const msg = JSON.parse(data);
      console.log('←', msg.type || Object.keys(msg)[0]);

      if (msg.connection_established) {
        console.log('  User ID:', msg.connection_established.user_id);
        console.log('  Credits:', msg.connection_established.credit_balance);

        // Start session
        ws.send(JSON.stringify({
          type: 'start',
          languages: ['en', 'hi'],
          config: {
            encoding: 'mulaw',
            sample_rate: 8000,
            continuous_mode: true,
            utterance_end_ms: 500,
            interim_results_frequency: 300
          }
        }));

      } else if (msg.type === 'connected') {
        console.log('✓ Session started! Send audio now...');

        // Send audio chunks (480 bytes = ~60ms at 8kHz)
        const audioInterval = setInterval(() => {
          const audioChunk = getAudioChunk();
          ws.send(audioChunk);
        }, 60);

        // Stop after 5 seconds
        setTimeout(() => {
          clearInterval(audioInterval);
          ws.send(JSON.stringify({ type: 'stop' }));
        }, 5000);

      } else if (msg.type === 'speech_started') {
        console.log('🎤 Speech detected - barge-in opportunity');

      } else if (msg.type === 'transcription') {
        if (msg.is_final) {
          console.log('✓ Final:', msg.text, `(confidence: ${msg.confidence})`);
        } else {
          console.log('  Partial:', msg.text);
        }

      } else if (msg.type === 'session_stopped') {
        console.log('✓ Session stopped');
        console.log('  Duration:', msg.billing_summary.total_duration_seconds, 's');
        console.log('  Cost: $', msg.billing_summary.total_cost);
        ws.close();
      }
    });

    ws.on('error', (error) => {
      console.error('Error:', error);
    });

    ws.on('close', () => {
      console.log('Connection closed');
    });
    ```
  </TabItem>

  <TabItem value="python" label="Python">
    ```python theme={null}
    import asyncio
    import json
    import websockets

    async def stt_websocket():
        API_KEY = "sk_live_your_key"
        url = f"ws://api.60db.ai/ws/stt?apiKey={API_KEY}"

        async with websockets.connect(url) as ws:
            # Wait for connection
            msg = json.loads(await ws.recv())
            if msg.get('connection_established'):
                print(f"✓ Connected (User: {msg['connection_established']['user_id']})")
                print(f"  Credits: ${msg['connection_established']['credit_balance']}")

            # Start session
            await ws.send(json.dumps({
                "type": "start",
                "languages": ["en", "hi"],
                "config": {
                    "encoding": "mulaw",
                    "sample_rate": 8000,
                    "continuous_mode": True,
                    "utterance_end_ms": 500,
                    "interim_results_frequency": 300
                }
            }))

            # Wait for connected
            msg = json.loads(await ws.recv())
            assert msg["type"] == "connected"
            print("✓ Session started!")

            # Send audio for 5 seconds
            for _ in range(83):  # ~5000ms / 60ms
                audio_chunk = get_audio_chunk()
                await ws.send(audio_chunk)  # Send as binary
                await asyncio.sleep(0.06)

            # Stop session
            await ws.send(json.dumps({"type": "stop"}))

            # Process remaining messages
            while True:
                msg = json.loads(await ws.recv())
                msg_type = msg.get("type")

                if msg_type == "speech_started":
                    print("🎤 Speech detected")

                elif msg_type == "transcription":
                    if msg.get("is_final"):
                        print(f"✓ {msg['text']} (confidence: {msg['confidence']})")
                    else:
                        print(f"  {msg['text']}...")

                elif msg_type == "session_stopped":
                    print(f"✓ Session stopped")
                    print(f"  Duration: {msg['billing_summary']['total_duration_seconds']}s")
                    print(f"  Cost: ${msg['billing_summary']['total_cost']}")
                    break

    asyncio.run(stt_websocket())
    ```
  </TabItem>

  <TabItem value="browser" label="Browser">
    ```javascript theme={null}
    const ws = new WebSocket('ws://api.60db.ai/ws/stt?apiKey=sk_live_your_key');
    let mediaRecorder;
    let audioContext;

    ws.onopen = () => {
      console.log('✓ Connected');
    };

    ws.onmessage = (event) => {
      const msg = JSON.parse(event.data);

      if (msg.connection_established) {
        console.log('✓ Authenticated!');

        // Start session
        ws.send(JSON.stringify({
          type: 'start',
          languages: ['en'],
          config: {
            encoding: 'linear',
            sample_rate: 48000,
            continuous_mode: true,
            interim_results_frequency: 300
          }
        }));

      } else if (msg.type === 'connected') {
        console.log('✓ Session started! Start speaking...');
        startAudioCapture();

      } else if (msg.type === 'speech_started') {
        console.log('🎤 Speech detected');

      } else if (msg.type === 'transcription') {
        if (msg.is_final) {
          console.log('✓', msg.text);
          updateTranscriptDisplay(msg.text);
        } else {
          console.log('...', msg.text);
          updateInterimDisplay(msg.text);
        }

      } else if (msg.type === 'session_stopped') {
        console.log('✓ Session stopped');
        stopAudioCapture();
        ws.close();
      }
    };

    async function startAudioCapture() {
      const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
      audioContext = new AudioContext({ sampleRate: 48000 });
      const source = audioContext.createMediaStreamSource(stream);
      const processor = audioContext.createScriptProcessor(4096, 1, 1);

      processor.onaudioprocess = (e) => {
        const inputData = e.inputBuffer.getChannelData(0);
        const pcm16 = new Int16Array(inputData.length);
        for (let i = 0; i < inputData.length; i++) {
          pcm16[i] = Math.max(-32768, Math.min(32767, inputData[i] * 32768));
        }

        ws.send(JSON.stringify({
          type: 'audio',
          audio: btoa(String.fromCharCode(...new Uint8Array(pcm16.buffer))),
          encoding: 'linear',
          sample_rate: 48000
        }));
      };

      source.connect(processor);
      processor.connect(audioContext.destination);
      mediaRecorder = { stream, processor, source };
    }

    function stopAudioCapture() {
      if (mediaRecorder) {
        mediaRecorder.stream.getTracks().forEach(track => track.stop());
        mediaRecorder.processor.disconnect();
        audioContext.close();
      }
    }
    ```
  </TabItem>
</Tabs>

## Audio Requirements

| Property    | Telephony (μ-law) | Browser (PCM)                 |
| ----------- | ----------------- | ----------------------------- |
| Encoding    | `mulaw` (8-bit)   | `linear` (16-bit)             |
| Sample Rate | 8000 Hz           | 16000, 24000, 44100, 48000 Hz |
| Chunk Size  | 480 bytes (60ms)  | 960-1920 bytes (60-120ms)     |
| Channels    | Mono (1 channel)  | Mono (1 channel)              |

## Supported Languages

The service supports 99+ languages via Whisper auto-detection. Common languages include:

| Code | Language  | Code | Language |
| ---- | --------- | ---- | -------- |
| `en` | English   | `hi` | Hindi    |
| `bn` | Bengali   | `es` | Spanish  |
| `fr` | French    | `de` | German   |
| `gu` | Gujarati  | `ta` | Tamil    |
| `te` | Telugu    | `kn` | Kannada  |
| `ml` | Malayalam | `mr` | Marathi  |
| `pa` | Punjabi   | `ar` | Arabic   |

## Pricing

* **Rate**: \$0.00000833 per second
* **Minimum**: \$0.01 per session
* **Billing**: Per second of audio processed

## Related

* [TTS WebSocket](/websocket-api/tts) - Text-to-Speech endpoint
* [WebSocket Quick Start](/websocket-api/quickstart) - Get started guide
* [WebSocket Playground](/websocket-playground) - Test in browser