Skip to main content

Vibestreams

Vibestreams are real-time music sessions generated from device sensor data. They combine motion detection, AI interpretation, and music generation to create unique audio experiences that respond to user movement.

Vibestream Modes

Solo Mode

Solo mode creates individual vibestreams for single-user experiences:
// Solo mode configuration in VibestreamModal.tsx
const rtaConfig = {
  mode: 'solo',
  store_to_filecoin: true,
  creator: accountIdToUse,
  created_at: Date.now(),
  // Group-specific fields are undefined for solo mode
  distance: undefined,
  ticket_amount: undefined,
  ticket_price: undefined,
  pay_per_stream: false,
  stream_price: '0'
};
Solo Mode Characteristics:
  • Single creator experience
  • No participant limits
  • No geographic constraints
  • Optional PPM monetization
  • Direct sensor-to-music mapping

Group Mode

Group mode enables collaborative vibestreams with participant management:
// Group mode configuration with validation
const rtaConfig = {
  mode: 'group',
  store_to_filecoin: true,
  // Required group parameters
  distance: parseInt(distance),      // 1-10 meters per contract
  ticket_amount: parseInt(ticketAmount), // Must be >= 1
  ticket_price: freeTickets ? undefined : ticketPrice,
  pay_per_stream: payPerStream,
  stream_price: payPerStream ? streamPrice : '0',
  creator: accountIdToUse,
  created_at: Date.now()
};
Group Mode Parameters:
ParameterTypeRangeDescription
distancenumber1-10 metersGeographic proximity constraint
ticket_amountnumber>= 1Maximum participants
ticket_pricestring>= 0Cost per ticket (optional)
pay_per_streamboolean-Enable per-minute payments
stream_pricestring>= 0Rate per minute (if PPM enabled)

Creation Flow

Parameter Validation

Group Mode Validation

The system validates group mode parameters before creation:
// Validation logic in VibestreamModal.tsx
const isLaunchDisabled = () => {
  if (mode === 'group') {
    // Distance validation (1-10 meters per contract)
    const distanceNum = parseInt(distance);
    if (!distance || distanceNum < 1 || distanceNum > 10) return true;
    
    // Ticket amount validation (must be at least 1)
    const ticketAmountNum = parseInt(ticketAmount);
    if (!ticketAmount || ticketAmountNum < 1) return true;
    
    // PPM validation
    if (payPerStream && (!streamPrice || streamPrice === '0')) return true;
    
    // Ticket price validation
    if (!freeTickets && (!ticketPrice || ticketPrice === '0')) return true;
  }
  return false;
};

Network-Specific Creation

NEAR Network

// NEAR vibestream creation
console.log(`🔥 Creating NEAR vibestream with config:`, rtaConfig);
const fullTokenId = await createRTANFT(rawRtaId, rtaConfig);
// Returns format: "rta_1234567890_abcdef"

Metis Hyperion

// Metis vibestream with integrated PPM
console.log(`🔥 Creating Metis vibestream with config:`, rtaConfig);
const fullTokenId = await createRTANFT(rawRtaId, rtaConfig);
// Returns format: "metis_vibe_1234567890_abcdef"
// Automatically registers with PPM contract if payPerStream enabled

Audio Processing Pipeline

Creator Mode (Full Processing)

// Full orchestration for creators in VibePlayer.tsx
console.log('🎛️ CREATOR MODE: Creating new vibestream with full orchestration');

// Start audio chunk service
audioChunkService.startCollecting(rtaId, config.creator);

// Initialize orchestration with wallet
await orchestrationIntegration.initializeWithWallet({ account: { accountId: config.creator } });

// Start Lyria orchestration
const success = await orchestrationIntegration.startOrchestration();

// Start vibestream session (sensors + chunks)
orchestrationIntegration.startVibestreamSession(rtaId, audioChunkService);

Participant Mode (Streaming Only)

// Simplified processing for participants
console.log('👥 PARTICIPANT MODE: Joining existing vibestream');

// Create audio element for LIVE HTTP-FLV stream
const liveAudio = new Audio();
liveAudio.src = participantOptions.streamingUrl;
liveAudio.autoplay = true;

// Handle streaming events
liveAudio.oncanplay = () => console.log('✅ Live stream ready to play');
liveAudio.onerror = () => {
  // Try HLS fallback
  if (participantOptions?.hlsUrl) {
    liveAudio.src = participantOptions.hlsUrl;
  }
};

Sensor Data Integration

Real-Time Processing

// Sensor data processing in VibePlayer.tsx
const processRealTimeSensorData = useCallback(async (sensorData) => {
  try {
    // Accurate waveform representation
    const magnitude = Math.sqrt(sensorData.x ** 2 + sensorData.y ** 2 + sensorData.z ** 2);
    
    // Sensitive amplitude calculation
    const amplitudeResponse = Math.min(magnitude * 2.5 + 0.1, 1);
    setCurrentAmplitude(amplitudeResponse);
    
    // Multi-axis frequency mapping
    const frequencyBase = 1.0;
    const xContribution = Math.abs(sensorData.x) * 4;
    const yContribution = Math.abs(sensorData.y) * 2;
    const zContribution = Math.abs(sensorData.z) * 1.5;
    setCurrentFrequency(frequencyBase + xContribution + yContribution + zContribution);
    
  } catch (error) {
    console.warn('Real-time processing error:', error);
  }
}, []);

Activity Detection

// Enhanced activity tracking in chunks.tsx
private updateMusicActivityDetection(now, timeSinceLastActivity) {
  // Adaptive threshold based on recent patterns
  let adaptiveThreshold = this.activityDetectionWindow;
  
  if (this.musicActivityHistory.length > 5) {
    const recentIntervals = [];
    for (let i = 1; i < this.musicActivityHistory.length; i++) {
      recentIntervals.push(this.musicActivityHistory[i] - this.musicActivityHistory[i - 1]);
    }
    
    if (recentIntervals.length > 0) {
      const avgInterval = recentIntervals.reduce((a, b) => a + b, 0) / recentIntervals.length;
      adaptiveThreshold = Math.min(Math.max(avgInterval * 0.6, this.activityDetectionWindow), this.activityDetectionWindow * 2);
    }
  }
  
  this.isLiveMusicActive = timeSinceLastActivity < adaptiveThreshold;
}

Storage and Persistence

Chunk Generation

// 60-second chunk timer in chunks.tsx
private startOptimizedChunkTimer() {
  this.chunkTimer = setInterval(async () => {
    if (this.isCollecting && this.audioBuffer.length > 0) {
      // Create chunk in background
      const chunkData = {
        chunkId: this.generateChunkId(false),
        buffer: this.combineAudioBuffers(),
        metadata: this.createCurrentMetadata(),
        isFinal: false,
        priority: this.isLiveMusicActive ? 'low' : 'normal'
      };
      
      // Add to background queue
      this.backgroundQueue.push(chunkData);
      
      // Clear buffer and reset timing
      this.audioBuffer = [];
      this.bufferStartTime = Date.now();
      this.chunkSequence++;
    }
  }, this.chunkDuration); // 60000ms = 60 seconds
}

Metadata Structure

// Chunk metadata in chunks.tsx
private createCurrentMetadata() {
  return {
    rtaId: this.currentRtaId,
    chunkId: this.generateChunkId(false),
    timestamp: Date.now(),
    creator: this.currentCreator,
    startTime: this.startTime.toString(),
    chunkTimestamp: this.bufferStartTime,
    sampleRate: 48000,
    channels: 2,
    bitDepth: 16,
    participantCount: this.currentParticipantCount,
    audioFormat: 'webm-opus'
  };
}

User Interface Components

Mode Selection

// Mode selection UI in VibestreamModal.tsx
<View style={styles.modeContainer}>
  <TouchableOpacity
    style={[styles.modeButton, mode === 'solo' && styles.selectedMode]}
    onPress={() => handleModeSelect('solo')}
  >
    <Text style={styles.modeText}>SOLO</Text>
  </TouchableOpacity>
  
  <TouchableOpacity
    style={[styles.modeButton, mode === 'group' && styles.selectedMode]}
    onPress={() => handleModeSelect('group')}
  >
    <Text style={styles.modeText}>GROUP</Text>
  </TouchableOpacity>
</View>

Group Configuration

// Group mode parameter inputs
{mode === 'group' && (
  <>
    <View style={styles.inputGroup}>
      <Text style={styles.inputLabel}>DISTANCE (1-10 METERS)</Text>
      <TextInput
        style={styles.textInput}
        value={distance}
        onChangeText={(text) => {
          const num = parseInt(text);
          if (text === '' || (num >= 1 && num <= 10)) {
            setDistance(text);
          }
        }}
        keyboardType="numeric"
        maxLength={2}
      />
    </View>
    
    <View style={styles.inputGroup}>
      <Text style={styles.inputLabel}>TICKETS</Text>
      <TextInput
        style={styles.textInput}
        value={ticketAmount}
        onChangeText={setTicketAmount}
        keyboardType="numeric"
      />
    </View>
  </>
)}

Error Handling

Fallback Modes

// Fallback handling in VibestreamModal.tsx
try {
  const fullTokenId = await createRTANFT(rawRtaId, rtaConfig);
  creationSucceeded = true;
} catch (error) {
  console.warn(`⚠️ ${networkInfo.type} vibestream creation failed:`, error);
  console.log('🔄 Using fallback mode - vibestream will continue with mock data');
  
  // Generate fallback data
  const networkPrefix = networkInfo.type === 'metis-hyperion' ? 'metis_vibe' : 'rta';
  fullTokenId = `${networkPrefix}_${rawRtaId}`;
  creationSucceeded = false;
  
  // Add fallback indicators
  rtaConfig.fallback_mode = true;
  rtaConfig.fallback_reason = error.message.includes('timeout') ? 
    `${networkInfo.type}_timeout` : `${networkInfo.type}_error`;
}

Next Steps