Skip to main content

Lyria RealTime Integration

The Lyria RealTime integration provides direct access to real-time music generation, enabling interactive music creation with continuous steering and real-time parameter adjustments.

Architecture Overview

The Lyria integration operates through a persistent WebSocket connection with multiple coordination layers:

Technical Specifications

Audio Format

Lyria RealTime generates audio with specific characteristics:
  • Output Format: Raw 16-bit PCM Audio
  • Sample Rate: 48kHz
  • Channels: 2 (stereo)
  • Encoding: Base64 (for WebSocket transmission)
  • Streaming: Real-time generation via WebSocket

API Connection

The system establishes a persistent bidirectional WebSocket connection:

Music Generation Control

Weighted Prompts

The system uses weighted prompts for continuous music steering:

Configuration Parameters

Music generation is controlled through comprehensive configuration:

Playback Control

Session Management

The system provides comprehensive playback control:

Real-time Music Steering

Dynamic Prompt Updates

The system supports real-time prompt modifications for smooth transitions:

Parameter Transitions

Configuration changes can be applied without stopping the stream:

Connection Management

Initialization and Retry Logic

The system implements robust connection management:

Automatic Reconnection

Audio Chunk Processing

Message Handling

Lyria sends audio data through WebSocket messages:

Integration with Orchestrator

Sensor-Driven Generation

The system integrates sensor data with Lyria generation:

Crossfade Detection

The system determines when crossfades are needed:

Safety and Limitations

Content Filtering

Lyria implements safety filters for prompts:
  • Filtered Prompts: Prompts triggering safety filters are ignored
  • Explanation Field: Filtered prompts include explanation in output
  • Fallback Behavior: System continues with previous valid prompts

Watermarking

All generated audio includes watermarking:
  • Identification: Output audio is watermarked for identification
  • Responsible AI: Follows Google’s Responsible AI principles
  • Transparency: Watermarks enable content attribution

Model Limitations

  • Instrumental Only: Lyria generates instrumental music exclusively
  • Real-time Generation: Optimized for streaming, not offline processing
  • Experimental Status: Lyria RealTime is currently experimental

Configuration

Environment Variables

Default Parameters

The system uses optimized defaults:

Best Practices

Effective Prompting

  1. Be Descriptive: Use adjectives for mood, genre, and instrumentation
  2. Gradual Steering: Modify elements incrementally for smooth transitions
  3. Weight Experimentation: Use prompt weights to influence generation strength
  4. Context Management: Reset context only for dramatic changes (BPM/scale)

Performance Optimization

  1. Buffer Management: Implement robust audio buffering for smooth playback
  2. Rate Limiting: Respect API quotas with appropriate request spacing
  3. Error Handling: Implement fallback systems for connection failures
  4. Memory Management: Clean up audio resources properly