Audio frame not converting to ndarray

Question

I am trying to run a colab file training openAI’s jukebox, however when I try to run the function code which loads the audio, I am getting an error:

File "/content/jukebox/jukebox/data/files_dataset.py", line 82, in get_song_chunk
data, sr = load_audio(filename, sr=self.sr, offset=offset, duration=self.sample_length)
File "/content/jukebox/jukebox/utils/io.py", line 48, in load_audio
frame = frame.to_ndarray(format=’fltp’) # Convert to floats and not int16
AttributeError: ‘list’ object has no attribute ‘to_ndarray’

It seems to be interpreting the frame input as a list, which when printed looks like this:

[<av.AudioFrame 0, pts=None, 778 samples at 22050Hz, stereo, fltp at
0x7fd03dd64150>]

When I try to change to frame = resampler.resample(frame) I get this error:

TypeError: ‘av.audio.frame.AudioFrame’ object cannot be interpreted as
an integer

I don’t really know much about audio files so i’m not sure how to debug and would appreciate help here.

the full code to load the audio is below.

def load_audio(file, sr, offset, duration, resample=True, approx=False, time_base='samples', check_duration=True):
    if time_base == 'sec':
        offset = offset * sr
        duration = duration * sr
    # Loads at target sr, stereo channels, seeks from offset, and stops after duration
    container = av.open(file)
    audio = container.streams.get(audio=0)[0] # Only first audio stream
    audio_duration = audio.duration * float(audio.time_base)
    if approx:
        if offset + duration > audio_duration*sr:
            # Move back one window. Cap at audio_duration
            offset = np.min(audio_duration*sr - duration, offset - duration)
    else:
        if check_duration:
            assert offset + duration <= audio_duration*sr, f'End {offset + duration} beyond duration {audio_duration*sr}'
    if resample:
        resampler = av.AudioResampler(format='fltp',layout='stereo', rate=sr)
    else:
        assert sr == audio.sample_rate
    offset = int(offset / sr / float(audio.time_base)) #int(offset / float(audio.time_base)) # Use units of time_base for seeking
    duration = int(duration) #duration = int(duration * sr) # Use units of time_out ie 1/sr for returning
    sig = np.zeros((2, duration), dtype=np.float32)
    container.seek(offset, stream=audio)
    total_read = 0
    for frame in container.decode(audio=0): # Only first audio stream
        if resample:
            frame.pts = None
            frame = resampler.resample(frame)
        frame = frame.to_ndarray(format='fltp') # Convert to floats and not int16
        read = frame.shape[-1]
        if total_read + read > duration:
            read = duration - total_read
        sig[:, total_read:total_read + read] = frame[:, :read]
        total_read += read
        if total_read == duration:
            break
    assert total_read <= duration, f'Expected {duration} frames, got {total_read}'
    return sig, sr

Asked By: walker_4

||

Source

Answer 1

Try replacing frame = frame.to_ndarray(format='fltp') by a direct assignation of the variable frame:

import numpy as np

#frame = frame.to_ndarray(format='fltp') #Original line
frame = np.ndarray(frame)

If you want it to be a specific data type, you can change the dtype argument of the ndarray function:

frame = np.ndarray(frame, dtype=np.float32)

Answered By: Cardstdani

Answer 2

If your variable frame is interpreted as a list, you could replace frame = resampler.resample(frame) with frame = resampler.resample(frame)[0]. Your code ran without errors once I made this edit.

Answered By: jylls

Answer 3

Try: frame = frame[0].to_ndarray(format='fltp')

Answered By: Neeraj Jain

Audio frame not converting to ndarray

Question:

Answers: