STT WebSocket

Real-time Speech-to-Text transcription via WebSocket streaming with support for 99+ languages and telephony integration.

Endpoint

Authentication

Query parameter authentication: Example:

ws://api.60db.ai/ws/stt?apiKey=sk_live_your_api_key

Connection Details

Property	Value
Protocol	WebSocket (RFC 6455)
Frame types	Binary (telephony) or Text/JSON (browser)
Ping/keepalive	Server sends WebSocket pings every 30s (timeout 10s)

The same port accepts plain HTTP GET requests and responds 200 ok — safe for load-balancer health checks.

Session Lifecycle

Client                          Server
  │                               │
  │──── TCP/TLS connect ─────────►│
  │◄─── {"connecting": true} ─────│  authenticating...
  │◄─── connection_established ──│  authenticated
  │                               │
  │──── {"type":"start", ...} ───►│
  │◄─── {"type":"connected"} ─────│  session ready
  │                               │
  │──── audio frames / messages ─►│
  │◄─── {"type":"speech_started"} │  VAD detected voice
  │◄─── {"type":"transcription"}  │  is_final=true, speech_final=true
  │                               │
  │──── {"type":"stop"} ─────────►│
  │◄─── {"type":"session_stopped"}│

Client → Server Messages

`start` — Begin session

Sent once after connection is established. Must be sent before any audio.

{
  "type": "start",
  "languages": ["en", "hi"],
  "config": {
    "encoding": "mulaw",
    "sample_rate": 8000,
    "utterance_end_ms": 500,
    "continuous_mode": true,
    "interim_results_frequency": 300,
    "no_speech_threshold": 0.60
  }
}

Parameters:

`audio` — JSON audio chunk (browser mode)

{
  "type": "audio",
  "audio": "<base64-encoded Int16 PCM or μ-law bytes>",
  "encoding": "linear",
  "sample_rate": 48000,
  "timestamp": 1700000000000
}

Fields:

Binary frame — raw μ-law audio (telephony mode)

Send a raw WebSocket binary frame with μ-law bytes, no JSON wrapper. The server auto-detects this as telephony mode on the first binary frame.

Recommended chunk size: 480 bytes = 60ms at 8kHz
Twilio default: 160 bytes = 20ms — batch 3 chunks into 60ms before sending

`config` — Change language mid-session

{
  "type": "config",
  "languages": ["hi"],
  "continuous_mode": true
}

Both languages and continuous_mode are optional; include only fields you want to change. Send "languages": null to revert to auto-detect.

`stop` — End session

{
  "type": "stop"
}

Server processes any remaining audio buffer, sends session_stopped, then closes.

`test` — Ping / latency check

{
  "type": "test",
  "message": "ping",
  "timestamp": 1700000000000
}

Server echoes test_response with the same timestamp for round-trip measurement.

Server → Client Messages

`connecting` — Authentication in progress

{
  "connecting": true,
  "message": "Authenticating...",
  "timestamp": 1775465918269
}

`connection_established` — Authentication successful

{
  "connection_established": {
    "service": "stt",
    "user_id": 43,
    "credit_balance": 9.97,
    "workspace": "default"
  }
}

Fields:

string

Service name: "stt"

integer

Your user ID

number

Available credits

string

Workspace name

`connected` — After `start` message is processed

{
  "type": "connected",
  "server_info": {
    "server_type": "Modular Sentence-based STT",
    "device": "cuda",
    "model": "large-v3",
    "processing_mode": "sentence_based_modular",
    "supported_languages": { "en": "English", "hi": "Hindi" },
    "total_languages": 99,
    "features": {
      "sentence_based_processing": true,
      "real_time_streaming": true,
      "telephony_support": true,
      "unicode_support": true,
      "mixed_language_support": true
    }
  }
}

`speech_started` — VAD detected voice activity

{
  "type": "speech_started",
  "timestamp": 1700000000.123
}

Use this for barge-in: interrupt TTS playback when this arrives. Fired after 2 consecutive VAD-positive chunks (~64ms of confirmed speech).

`transcription` — Transcription result

All results (interim and final) share the same transcription type — differentiate with flags. Final result (is_final=true, speech_final=true):

{
  "type": "transcription",
  "text": "Hello, how are you?",
  "confidence": 0.87,
  "language": "en",
  "language_name": "EN",
  "is_final": true,
  "speech_final": true,
  "is_partial": false,
  "sentence_id": 3,
  "processing_mode": "sentence_complete",
  "duration": 1.82,
  "latency": 0.43,
  "timestamp": 1700000000.456,
  "words": [
    { "word": "Hello", "start": 0.0, "end": 0.32, "probability": 0.98 },
    { "word": "how", "start": 0.35, "end": 0.52, "probability": 0.94 }
  ],
  "utterance_end_ms": 1820
}

Empty speech_final signal (text="", is_final=true, speech_final=true): Sent when audio was detected but transcription was rejected (silence, hallucination, low confidence, wrong language). Client should reset its state on this message and not treat it as an error.

{
  "type": "transcription",
  "text": "",
  "confidence": 0.0,
  "is_final": true,
  "speech_final": true,
  "processing_mode": "speech_end_no_result",
  "timestamp": 1700000000.789
}

Interim result (is_final=false, speech_final=false) — only sent when interim_results_frequency is set:

{
  "type": "transcription",
  "text": "Hello how",
  "confidence": 0.72,
  "language": "en",
  "is_final": false,
  "speech_final": false,
  "is_partial": true
}

Use interims only for barge-in word-count checks. Never send interim text to the LLM — a final with is_final=true, speech_final=true will follow. Response Fields:

string

Transcribed text. Empty string = speech-end-no-result signal.

number

0.0–1.0. Telephony typically 0.35–0.75; browser 0.55–0.95.

string

Detected language code e.g. "en".

string

Uppercase language code e.g. "EN".

boolean

true = complete utterance transcription or end-of-speech signal.

boolean

true when speech segment fully processed. Always matches is_final for finals.

boolean

true for interim results only.

integer

Monotonically increasing counter per session.

number

Duration (seconds) of the audio segment transcribed.

number

Seconds from processing start to result ready (excludes queue time).

array

Word-level timestamps [{word, start, end, probability}]. Present on finals when available.

integer

Timestamp (ms) of last word in the utterance.

string

Internal mode string — useful for debugging.

`language_changed` — After `config` message changes language

{
  "type": "language_changed",
  "language": "Multi-language: HI",
  "language_code": ["hi"]
}

`mode_changed` — After `config` message changes `continuous_mode`

{
  "type": "mode_changed",
  "continuous_mode": true,
  "mode_name": "continuous",
  "silence_threshold": 0.5
}

`session_stopped` — After `stop` is processed

{
  "type": "session_stopped",
  "billing_summary": {
    "total_duration_seconds": 5.2,
    "total_cost": 0.000043,
    "characters_transcribed": 42
  }
}

`error` — Processing error

{
  "type": "error",
  "error": "Audio processing error: ...",
  "timestamp": 1700000000.0
}

`test_response` — Reply to `test` ping

{
  "type": "test_response",
  "message": "pong - Sentence-based STT ready",
  "timestamp": 1700000000000,
  "processing_mode": "sentence_based"
}

Complete Example

Audio Requirements

Property	Telephony (μ-law)	Browser (PCM)
Encoding	`mulaw` (8-bit)	`linear` (16-bit)
Sample Rate	8000 Hz	16000, 24000, 44100, 48000 Hz
Chunk Size	480 bytes (60ms)	960-1920 bytes (60-120ms)
Channels	Mono (1 channel)	Mono (1 channel)

Supported Languages

The service supports 99+ languages via Whisper auto-detection. Common languages include:

Code	Language	Code	Language
`en`	English	`hi`	Hindi
`bn`	Bengali	`es`	Spanish
`fr`	French	`de`	German
`it`	Italian	`pt`	Portuguese
`ar`	Arabic	`ja`	Japanese
`ko`	Korean	`zh`	Chinese
`gu`	Gujarati	`ta`	Tamil
`te`	Telugu	`kn`	Kannada
`ml`	Malayalam	`mr`	Marathi
`pa`	Punjabi	`bn`	Bengali

Mixed-language text is supported. Language aliasing: Urdu (ur) is treated as Hindi (hi), Malay (ms) as Indonesian (id).

Limitations and Best Practices

utterance_end_ms Values below 300ms are silently clamped to 300ms. Recommended 500ms for voicebots. Lower values cause premature sentence cuts and fragmented transcriptions. Language count Max 2 languages recommended. More languages = proportionally more latency per utterance. Telephony confidence 8kHz μ-law → 16kHz resampling reduces Whisper confidence by ~0.10–0.15 compared to browser audio. Apply a client-side threshold of 0.35 for telephony (vs 0.45–0.55 for browser). Buffer limits

Minimum buffer to trigger transcription: 0.8s of audio
Maximum buffer before force-processing: 5.0s
Pre-speech ring buffer: 1.0s (captures first word before VAD fires)

VAD thresholds

Speech start: Silero probability > 0.50 (requires 2 consecutive frames, ~64ms)
Speech continuation: probability > 0.40
Grace period: 6 chunks (192ms) of sub-threshold audio before speech_end is declared

Model Whisper large-v3, int8_float16 quantization, CTranslate2 backend. Expected latency: 300–600ms from speech_end to final result (GPU), 1–3s (CPU).

Pricing

Rate: $0.00000833 per second
Minimum: $0.01 per session
Billing: Per second of audio processed

Error Codes

Code	Description
1008	Authentication failed
1008	Insufficient credits
1011	Internal server error
1006	Connection lost

Testing

# Install wscat
npm install -g wscat

# Test STT
wscat -c "ws://api.60db.ai/ws/stt?apiKey=sk_live_your_key"

Then send:

{"type":"start","languages":["en"],"config":{"encoding":"mulaw","sample_rate":8000,"continuous_mode":true}}

TTS WebSocket - Text-to-Speech endpoint
WebSocket API Reference - Complete documentation
Voices API - Get available voices

​STT WebSocket

​Endpoint

​Authentication

​Connection Details

​Session Lifecycle

​Client → Server Messages

​start — Begin session

​audio — JSON audio chunk (browser mode)

​Binary frame — raw μ-law audio (telephony mode)

​config — Change language mid-session

​stop — End session

​test — Ping / latency check

​Server → Client Messages

​connecting — Authentication in progress

​connection_established — Authentication successful

​connected — After start message is processed

​speech_started — VAD detected voice activity

​transcription — Transcription result

​language_changed — After config message changes language

​mode_changed — After config message changes continuous_mode

​session_stopped — After stop is processed

​error — Processing error

​test_response — Reply to test ping

​Complete Example

​Audio Requirements

​Supported Languages

​Limitations and Best Practices

​Pricing

​Error Codes

​Testing

​Related

STT WebSocket

Endpoint

Authentication

Connection Details

Session Lifecycle

Client → Server Messages

`start` — Begin session

`audio` — JSON audio chunk (browser mode)

Binary frame — raw μ-law audio (telephony mode)

`config` — Change language mid-session

`stop` — End session

`test` — Ping / latency check

Server → Client Messages

`connecting` — Authentication in progress

`connection_established` — Authentication successful

`connected` — After `start` message is processed

`speech_started` — VAD detected voice activity

`transcription` — Transcription result

`language_changed` — After `config` message changes language

`mode_changed` — After `config` message changes `continuous_mode`

`session_stopped` — After `stop` is processed

`error` — Processing error

`test_response` — Reply to `test` ping

Complete Example

Audio Requirements

Supported Languages

Limitations and Best Practices

Pricing

Error Codes

Testing

Related