Streaming

Real-time audio streaming for low-latency applications

Overview

Streaming allows you to receive audio chunks in real-time as they’re generated, enabling immediate playback without waiting for the entire audio file to be created. This is essential for interactive applications like voice assistants and chatbots.

Benefits

Low Latency

Start playing audio within milliseconds

Memory Efficient

Process chunks instead of loading entire file

Better UX

Progressive audio playback feels more responsive

Long Content

Handle unlimited text length efficiently

How It Works

1. Send Request: Submit text to the streaming endpoint
2. Receive Chunks: Get audio chunks as they’re generated
3. Play Immediately: Start playing the first chunk
4. Continue Streaming: Receive and play subsequent chunks
5. Complete: Receive completion notification

Implementation

JavaScript

import { SixtyDBClient } from '60db';

const client = new SixtyDBClient('your-api-key');

// Audio player setup
const audioContext = new AudioContext();
const audioQueue = [];

await client.textToSpeechStream(
  {
    text: 'This is a longer text that will be streamed in real-time for immediate playback.',
    voice_id: 'default-voice'
  },
  {
    onChunk: async (chunk) => {
      // Convert chunk to audio buffer
      const audioBuffer = await audioContext.decodeAudioData(chunk.buffer);
      
      // Add to queue and play
      audioQueue.push(audioBuffer);
      if (audioQueue.length === 1) {
        playNextChunk();
      }
    },
    onComplete: () => {
      console.log('Streaming complete');
    },
    onError: (error) => {
      console.error('Streaming error:', error);
    }
  }
);

function playNextChunk() {
  if (audioQueue.length === 0) return;
  
  const buffer = audioQueue[0];
  const source = audioContext.createBufferSource();
  source.buffer = buffer;
  source.connect(audioContext.destination);
  
  source.onended = () => {
    audioQueue.shift();
    playNextChunk();
  };
  
  source.start();
}

Python

from sixtydb import SixtyDBClient
import pyaudio

client = SixtyDBClient('your-api-key')

# Audio player setup
p = pyaudio.PyAudio()
stream = p.open(
    format=pyaudio.paInt16,
    channels=1,
    rate=24000,
    output=True
)

def handle_chunk(chunk):
    # Play audio chunk immediately
    stream.write(chunk)

def handle_complete():
    print("Streaming complete")
    stream.stop_stream()
    stream.close()
    p.terminate()

def handle_error(error):
    print(f"Error: {error}")

# Stream text to speech
client.text_to_speech_stream(
    text='This is a longer text that will be streamed in real-time.',
    on_chunk=handle_chunk,
    on_complete=handle_complete,
    on_error=handle_error,
    voice_id='default-voice'
)

Advanced Usage

React Component

import { useState } from 'react';
import { SixtyDBClient } from '60db';

function StreamingTTS() {
  const [isStreaming, setIsStreaming] = useState(false);
  const [text, setText] = useState('');

  const client = new SixtyDBClient(process.env.REACT_APP_API_KEY);

  const streamAudio = async () => {
    setIsStreaming(true);

    await client.textToSpeechStream(
      { text, voice_id: 'default-voice' },
      {
        onChunk: (chunk) => {
          // Play chunk
          playAudioChunk(chunk);
        },
        onComplete: () => {
          setIsStreaming(false);
        },
        onError: (error) => {
          console.error(error);
          setIsStreaming(false);
        }
      }
    );
  };

  return (
    <div>
      <textarea
        value={text}
        onChange={(e) => setText(e.target.value)}
        placeholder="Enter text to speak..."
      />
      <button onClick={streamAudio} disabled={isStreaming}>
        {isStreaming ? 'Streaming...' : 'Speak'}
      </button>
    </div>
  );
}

Voice Assistant

class VoiceAssistant {
  constructor(apiKey) {
    this.client = new SixtyDBClient(apiKey);
    this.audioContext = new AudioContext();
  }

  async speak(text) {
    return new Promise((resolve, reject) => {
      this.client.textToSpeechStream(
        { text, voice_id: 'assistant-voice' },
        {
          onChunk: (chunk) => this.playChunk(chunk),
          onComplete: () => resolve(),
          onError: (error) => reject(error)
        }
      );
    });
  }

  async playChunk(chunk) {
    const audioBuffer = await this.audioContext.decodeAudioData(chunk.buffer);
    const source = this.audioContext.createBufferSource();
    source.buffer = audioBuffer;
    source.connect(this.audioContext.destination);
    source.start();
  }
}

// Usage
const assistant = new VoiceAssistant('your-api-key');
await assistant.speak('Hello! How can I help you today?');

Performance Optimization

Buffering Strategy

class AudioStreamer {
  constructor() {
    this.chunks = [];
    this.isPlaying = false;
    this.minBufferSize = 3; // Wait for 3 chunks before playing
  }

  addChunk(chunk) {
    this.chunks.push(chunk);
    
    if (!this.isPlaying && this.chunks.length >= this.minBufferSize) {
      this.startPlayback();
    }
  }

  async startPlayback() {
    this.isPlaying = true;
    
    while (this.chunks.length > 0) {
      const chunk = this.chunks.shift();
      await this.playChunk(chunk);
    }
    
    this.isPlaying = false;
  }

  async playChunk(chunk) {
    // Play audio chunk
    return new Promise((resolve) => {
      // Audio playback logic
      setTimeout(resolve, chunkDuration);
    });
  }
}

Best Practices

Buffer Management

• Maintain a small buffer (2-3 chunks) for smooth playback
• Handle network interruptions gracefully
• Implement retry logic for failed chunks

Error Handling

• Always implement onError callback
• Provide user feedback during streaming
• Have fallback for streaming failures

Performance

• Reuse AudioContext instances
• Clean up resources after playback
• Monitor memory usage for long streams

User Experience

• Show loading indicator before first chunk
• Allow users to stop streaming
• Provide playback controls

Use Cases

Real-time Chat

async function sendMessage(message) {
  // Display user message
  displayMessage('user', message);
  
  // Get AI response
  const response = await getAIResponse(message);
  displayMessage('assistant', response);
  
  // Stream audio response
  await client.textToSpeechStream(
    { text: response },
    {
      onChunk: (chunk) => playAudioChunk(chunk),
      onComplete: () => enableInput()
    }
  );
}

Audiobook Player

async function playChapter(chapterText) {
  let currentPosition = 0;
  
  await client.textToSpeechStream(
    { text: chapterText },
    {
      onChunk: (chunk) => {
        playChunk(chunk);
        currentPosition += chunk.duration;
        updateProgress(currentPosition);
      },
      onComplete: () => {
        moveToNextChapter();
      }
    }
  );
}

API Reference

Streaming API

View complete streaming API documentation