This addresses #27 and implements a new ffmpeg backend. I choose ffmpeg-python for reading and writing. Here the audio is piped directly to stdin instead of writing temporarly files with pysoundfile and converting them in a separate process call.

Part of the code was copied from spleeters audio backend. First benchmarks of the input piping indicate that this method is twice as fast as my previous "tmpfile based method".

Saving stems still requires to save temporarly files since the complex filter cannot be carried out using python-ffmpeg. This enabled a new API. Here the idea was to not come up with presets and do all the checks to cover all use cases but instead let users have to do this themselves. This means more errors for users, but its way easier to maintain. E.g. if a user wants to write multistream audio as .wav files, an error will be thrown, since this container does not support multiple streams. The user would instead have to use streams_as_multichannel.

This PR furthermore introduces a significant number of new features:

Audio Loading

• Loading audio now uses the same API as in spleeters audio loading backend
• A target samplerate can be specified to resample audio on-the-fly and return the resampled audio
• An option stems_from_multichannel was added to load stems that are aggregated into multichannel audio (concatenation of pairs of stereo channels), see more info on audio writing
• substream titles can be read from the Info object.

Audio Writing

• stems can now be saved as substreams, aggregated into channels or saved as multiple files.
• titles for each substream can now be embedded into metadata
• in addition to write_stems (which is a preset to achieve compatibility with NI stems), we also have write_streams (supports writing as multichannel or multiple files). And, in case, stempeg is used for just stereo files, write_audio can be used (Again this is API compatible to spleeter).

The procedure for writing stream files may be quite complex as it varies depending of the specified output container format. Basically there are two possible stream saving options:

1.) container supports multiple streams (mp4/m4a, opus, mka) 2.) container does not support multiple streams (wav, mp3, flac)

For 1.) we provide two options:

1a.) streams will be saved as substreams aka when streams_as_multichannel=False (default) 1b.) streams will be aggregated into channels and saved as multichannel file. Here the audio tensor of shape=(streams, samples, 2) will be converted to a single-stream multichannel audio (samples, streams*2). This option is activated using streams_as_multichannel=True 1c.) streams will be saved as multiple files when streams_as_files is active

For 2.), when the container does not support multiple streams there are also two options:

2a) streams_as_multichannel has to be set to True (See 1b) otherwise an error will be raised. Note that this only works for wav and flac). * file ending of path determines the container (but not the codec!). 2b) streams_as_files so that multiple files will be created when streams_as_files is active

Example / Use Cases

"""Opens a stem file and saves (re-encodes) back to a stem file
"""
import argparse
import stempeg
import subprocess as sp
import numpy as np
from os import path as op


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument(
        'input',
    )
    args = parser.parse_args()

    # load stems
    stems, rate = stempeg.read_stems(args.input)

    # load stems,
    # resample to 96000 Hz,
    # use multiprocessing
    stems, rate = stempeg.read_stems(
        args.input,
        sample_rate=96000,
        multiprocess=True
    )

    # --> stems now has `shape=(stem x samples x channels)``

    # save stems from tensor as multi-stream mp4
    stempeg.write_stems(
        "test.stem.m4a",
        stems,
        sample_rate=96000
    )

    # save stems as dict for convenience
    stems = {
        "mix": stems[0],
        "drums": stems[1],
        "bass": stems[2],
        "other": stems[3],
        "vocals": stems[4],
    }
    # keys will be automatically used

    # from dict as files
    stempeg.write_stems(
        "test.stem.m4a",
        data=stems,
        sample_rate=96000
    )

    # `write_stems` is a preset for the following settings
    # here the output signal is resampled to 44100 Hz and AAC codec is used
    stempeg.write_stems(
        "test.stem.m4a",
        stems,
        sample_rate=96000,
        writer=stempeg.StreamsWriter(
            codec="aac",
            output_sample_rate=44100,
            bitrate="256000",
            stem_names=['mix', 'drums', 'bass', 'other', 'vocals']
        )
    )

    # Native Instruments compatible stems
    stempeg.write_stems(
        "test_traktor.stem.m4a",
        stems,
        sample_rate=96000,
        writer=stempeg.NIStemsWriter(
            stems_metadata=[
                {"color": "#009E73", "name": "Drums"},
                {"color": "#D55E00", "name": "Bass"},
                {"color": "#CC79A7", "name": "Other"},
                {"color": "#56B4E9", "name": "Vocals"}
            ]
        )
    )

    # lets write as multistream opus (supports only 48000 khz)
    stempeg.write_stems(
        "test.stem.opus",
        stems,
        sample_rate=96000,
        writer=stempeg.StreamsWriter(
            output_sample_rate=48000,
            codec="opus"
        )
    )

    # writing to wav requires to convert streams to multichannel
    stempeg.write_stems(
        "test.wav",
        stems,
        sample_rate=96000,
        writer=stempeg.ChannelsWriter(
            output_sample_rate=48000
        )
    )

    # # stempeg also supports to load merged-multichannel streams using
    stems, rate = stempeg.read_stems(
        "test.wav",
        reader=stempeg.ChannelsReader(nb_channels=2)
    )

    # mp3 does not support multiple channels,
    # therefore we have to use `stempeg.FilesWriter`
    # outputs are named ["output/0.mp3", "output/1.mp3"]
    # for named files, provide a dict or use `stem_names`
    # also apply multiprocessing
    stempeg.write_stems(
        ("output", ".mp3"),
        stems,
        sample_rate=rate,
        writer=stempeg.FilesWriter(
            multiprocess=True,
            output_sample_rate=48000,
            stem_names=["mix", "drums", "bass", "other", "vocals"]
        )
    )

import glob, os
import stempeg
import os.path

train_path = "path_to_train/"
os.chdir(train_path)
for file in glob.glob("*.stem.mp4"):
    file_path = train_path + file
    print(os.path.isfile(file_path))
    S, rate = stempeg.read_stems(file_path)

Even isfile returns true, read_stems throws 'FileNotFoundError: [WinError 2] '

I am running it on anaconda. It seems to work perfectly on colab. However on anaconda it fails.

The behavior is weird as well. I ran the command on bash and it runs correctly.

I have a loop which runs through all the stem files and it breaks after executing random iterations giving the error stated below. I believe this could be an multiprocessing issue. Could it be that that file is already being used by another process?

File "", line 1, in runfile('C:/Users/w1572032/.spyder-py3/temp.py', wdir='C:/Users/w1572032/.spyder-py3')

File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile execfile(filename, namespace)

File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/w1572032/.spyder-py3/temp.py", line 28, in t = np.copy(track.targets['vocals'].audio.T)

File "C:\ProgramData\Anaconda3\lib\site-packages\musdb\audio_classes.py", line 113, in audio audio = source.audio

File "C:\ProgramData\Anaconda3\lib\site-packages\musdb\audio_classes.py", line 47, in audio filename=self.path, stem_id=self.stem_id

File "C:\ProgramData\Anaconda3\lib\site-packages\stempeg\read.py", line 90, in read_stems FFinfo = FFMPEGInfo(filename)

File "C:\ProgramData\Anaconda3\lib\site-packages\stempeg\read.py", line 19, in init self.json_info = read_info(self.filename)

File "C:\ProgramData\Anaconda3\lib\site-packages\stempeg\read.py", line 54, in read_info out = sp.check_output(cmd)

File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 336, in check_output **kwargs).stdout

File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 418, in run output=stdout, stderr=stderr)

CalledProcessError: Command '['ffprobe', 'C:\Users\w1572032\Desktop\musdb18\train\BigTroubles - Phantom.stem.mp4', '-v', 'error', '-print_format', 'json', '-show_format', '-show_streams']' returned non-zero exit status 3221225478.

🥳 osx seems to support stem files and has a UI to select the stem right from the quicklook window:

However, in seems that is uses some specific metadata to read the stem track name. Currently I don't know how to do that with ffmpeg, but it would be great to find out if there is way to support this.

Hello,

As you stated in the documentation the stems write doesn't always work well. I am using this command with ffmpeg to create a STEM file:

ffmpeg -i ~/mix.wav -i ~drums.wav -i ~/vocals.wav -map 0 -map 1 -map 2 -c:a libfdk_aac -metadata:s:0 title=mix -metadata:s:1 title=drums -metadata:s:2 title=vocals ~/output.stem.mp4

I then tried to read it back using the musdb library and it works well. I was wondering if this could be included in your library to finally make it work properly.

I unfortunately do not have much time to work more on this and ask for a pull request but I made a simple implementation if could be of any help. Also check this homebrew-ffmpeg if the right codecs are not installed properly in the official ffmpeg distribution.

I am using the musdb package and convert the mp4 files containing multiple audio sources to wave files, as shown here:

https://github.com/f90/Wave-U-Net/blob/master/Datasets.py#L132

But randomly during conversion (so with potentially any file), conversion just freezes forever. After interrupting the process I can read the following error:

Traceback (most recent call last): File "/opt/local/pycharm/helpers/pydev/pydevd.py", line 1668, in main() File "/opt/local/pycharm/helpers/pydev/pydevd.py", line 1662, in main globals = debugger.run(setup['file'], None, None, is_module) File "/opt/local/pycharm/helpers/pydev/pydevd.py", line 1072, in run pydev_imports.execfile(file, globals, locals) # execute the script File "/mnt/daten/PycharmProjects/Wave-U-Net/Training.py", line 326, in @ex.automain File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/sacred/experiment.py", line 137, in automain self.run_commandline() File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/sacred/experiment.py", line 260, in run_commandline return self.run(cmd_name, config_updates, named_configs, {}, args) File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/sacred/experiment.py", line 209, in run run() File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/sacred/run.py", line 221, in call self.result = self.main_function(*args) File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/sacred/config/captured_function.py", line 46, in captured_function result = wrapped(*args, **kwargs) File "/mnt/daten/PycharmProjects/Wave-U-Net/Training.py", line 348, in dsd_100_experiment dsd_train, dsd_test = Datasets.getMUSDB(model_config["musdb_path"]) # List of (mix, acc, bass, drums, other, vocal) tuples File "/mnt/daten/PycharmProjects/Wave-U-Net/Datasets.py", line 149, in getMUSDB vocal_audio = track.targets["vocals"].audio File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/musdb/audio_classes.py", line 113, in audio audio = source.audio File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/musdb/audio_classes.py", line 47, in audio filename=self.path, stem_id=self.stem_id File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/stempeg/read.py", line 91, in read_stems FFinfo = FFMPEGInfo(filename) File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/stempeg/read.py", line 19, in init self.json_info = read_info(self.filename) File "/home/daniel/tf-env-waveunet/local/lib/python2.7/site-packages/stempeg/read.py", line 55, in read_info out = sp.check_output(cmd) File "/usr/lib/python2.7/subprocess.py", line 567, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs) File "/usr/lib/python2.7/subprocess.py", line 711, in init errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1319, in _execute_child data = _eintr_retry_call(os.read, errpipe_read, 1048576) File "/usr/lib/python2.7/subprocess.py", line 476, in _eintr_retry_call return func(*args) KeyboardInterrupt

Process finished with exit code 1

It seems that the ffmpeg/ffprobe process that identifies the stems within the mp4 file never returns, or returns empty output, or sth of that sort, so that the stempeg library waits forever for a response at sp.check_output. It doesnt look like there is a timeout for waiting for the ffmpeg output either. Plus ffmpeg is called with -v error, maybe that is suppressing errors that we should react to?

Any idea of how to fix this?

In the last version of stempeg the method to load PCM through python-ffmpeg resulted in small differences to what I obtained with the previous version of stempeg.

This PR reverts the casting procedure for int16 to float32 + normalization [-1, 1]. This is important since dithering errors did significantly influence separation scores in regression tests.

With this PR, ffmpeg pipes int16 into the numpy buffer and is converted and normalized to float in numpy. This got the same results as I used to have before where I used temporarly wav files, converting to float32 using soundfile. Furthermore, when using int16 pipes, conversion is slightly faster.

Also ping @romi1502 and @mmoussallam since this function originally derived from spleeters code. You might want to change it there as well.

my data has a name format like 'xxxx - xxxx.stem.m64'. but stempeg cannot reconginize the blank space before "-". So it will raise a error said there is no file.

The dataset actually is MUSDB18-7 set

I solve it by following codes which actually deletes the front blank space. But I hope there is a better way to solve it.

index = track_name.index('-')
track_name = track_name[:index-1] + track_name[index:]

In some distributions (e.g. NixOS) ffmpeg is not necessarily in PATH. Unifying detection of the ffmpeg and ffprobe executables makes it easier for packagers to patch the package to accomodate such situations.

Hi author(s),

I'm trying to run the tests included in this package, but the assert statements on the shapes of the stems are failing. The tests expect a shape of (5, 265216, 2) but the file has a shape of (5, 267264, 2).

Is this a bug or have the files been updated without updating the tests?

Thanks!

carefully reviews the proposals made by @romi1502 in #39 and reverts the fix. To allow regression tests to pass for dependencies such as musdb or museval, the old behaviour can be used with

stempeg.read_stems(..., ffmpeg_format="s16le")

Bug Description: When using stempeg as part of musdb, I encountered the following error:

        stem_durations = np.array([t.shape[0] for t in stems])
        if not (stem_durations == stem_durations[0]).all():
>           warnings.warning("Stems differ in length and were shortend")
E           AttributeError: module 'warnings' has no attribute 'warning'

/usr/local/lib/python3.9/site-packages/stempeg/read.py:299: AttributeError

warning() does not exist after checking the warnings package.

Suggested Solution: warnings.warning() -> warnings.warn() since warn() exists.

Is there a way to convert the 4 stem output files from the new Open-Unmix UMX using Stempeg to output 16 bit flac files instead of the 24 bit flac files I am currently getting using it?

Thank, Rog

supporting file-like objects to read and decode in-memory data would be a useful enhancement. There may be problems, as suggested here, though: https://github.com/kkroening/ffmpeg-python/issues/292