SEP model BS-RoFormer source separation ( a voice extraction from the source )

[mirek190]

Summary

Model BS_roformer
https://github.com/lucidrains/BS-RoFormer

Adds native BS-RoFormer vocal source separation beside the existing
htdemucs and mel_band_roformer families. ( in my opinion BS-RoFormer is doing the best work to separate a voice from my experience )

The tested checkpoint is Viperx
model_bs_roformer_ep_368_sdr_12.9628.ckpt. It follows the Band-Split
RoFormer architecture from ByteDance AI Labs as implemented by
lucidrains/BS-RoFormer; audio-separator was used as the executable Python
parity reference.

Implementation

• registers a new bs_roformer offline sep family immediately beside
  HTDemucs and Mel-Band RoFormer in the framework registry
• reuses the shared RoFormer STFT/ISTFT, chunking, attention, and transformer
  infrastructure while adding BS-specific:
  • explicit non-overlapping freqs_per_bands
  • fused QKV projection
  • checkpoint-compatible transformer/final RMSNorm placement
  • arbitrary mask-estimator depth
• returns named vocals and instrumental stems; instrumental is derived as
  mixture minus vocals
• adds package-spec loading for converted SafeTensors and standalone GGUF
• embeds the package spec and config.json in a standalone GGUF
• adds checkpoint conversion and Python reference helpers under
  tests/bs_roformer
• adds BS-RoFormer to the WebUI Source separation model list directly between
  HTDemucs and Mel-Band RoFormer
• points the WebUI catalog at the standalone named GGUF file and wires the
  visible Separate action directly to the separation handler, avoiding a
  stalled two-stage Gradio callback chain
• updates the main model table, audio-tools guide, GGUF status table, and
  English/Chinese WebUI documentation

Reproduction

Reference checkpoint conversion

python tests\bs_roformer\convert_reference_ckpt.py `
  --ckpt model_bs_roformer_ep_368_sdr_12.9628.ckpt `
  --config-path model_bs_roformer_ep_368_sdr_12.9628.yaml `
  --output-dir models\BS-RoFormer-ep368

Q8 GGUF conversion

build\windows-cuda-release\bin\audiocpp_gguf.exe `
  --input models\BS-RoFormer-ep368\model.safetensors `
  --output models\BS-RoFormer-ep368_Q8\BS-RoFormer-ep368_Q8.gguf `
  --type q8_0 `
  --family bs_roformer `
  --model-spec model_specs\bs_roformer.json `
  --root models\BS-RoFormer-ep368

Generated local artifacts:

• models\BS-RoFormer-ep368\model.safetensors: 639,103,376 bytes
• models\BS-RoFormer-ep368_Q8\BS-RoFormer-ep368_Q8.gguf: 172,532,256 bytes
• CLI/server/WebUI outputs: vocals.wav and instrumental.wav

Windows CUDA build

$cuda = "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
$env:CMAKE_CUDA_COMPILER = "$cuda\bin\nvcc.exe"
$env:CUDACXX = $env:CMAKE_CUDA_COMPILER
$env:CUDA_PATH = $cuda
$env:CUDAToolkit_ROOT = $cuda
$env:NVCC_PREPEND_FLAGS = "-allow-unsupported-compiler"
.\scripts\build_windows.ps1 -Target audiocpp_cli -Jobs 16
.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16

CLI

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task sep `
  --model models\BS-RoFormer-ep368_Q8\BS-RoFormer-ep368_Q8.gguf `
  --backend cuda `
  --audio input_8s.wav `
  --out-dir outputs\bs-roformer-q8

The converted SafeTensors package was also tested:

build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task sep `
  --family bs_roformer `
  --model models\BS-RoFormer-ep368 `
  --backend cuda `
  --audio input_8s.wav `
  --out-dir outputs\bs-roformer-f32

Server

The server was started with a CUDA/offline sep model entry whose path was
models\BS-RoFormer-ep368_Q8\BS-RoFormer-ep368_Q8.gguf, then exercised with:

{
  "model": "bs-roformer-q8",
  "request": {
    "audio": "E:\\path\\to\\input_8s.wav"
  }
}

against POST /v1/tasks/run. It returned both named stems. Their SHA-256
hashes were exactly equal to the CLI-generated WAVs.

WebUI

$env:AUDIOCPP_BUNDLE = $PWD
$env:AUDIOCPP_BACKEND = "gpu"
venv\Scripts\python.exe webui\webui.py

The browser test selected Source separation → bs-roformer, uploaded the
same 8-second WAV, and used the visible Separate action. The WebUI-managed
server loaded the standalone GGUF and displayed two playable/downloadable
tracks. A full-page result screenshot was retained locally as
outputs\bs_roformer_webui_test.png.

Parity and performance

Test system:

• Windows, CUDA 12.4
• NVIDIA GeForce RTX 3090, 24 GB
• stereo 44.1 kHz inputs

20-second test

Comparison	Waveform cosine
audio.cpp Q8 vs audio.cpp F32	0.999985368
audio.cpp F32 vs audio-separator Python	0.996474981
audio.cpp Q8 vs audio-separator Python	0.996387362

Exact-frame 8-second test

Both paths produced exactly 352,800 frames.

Comparison/metric	Result
Q8 vs F32 waveform cosine	0.999994784
Q8 vs F32 log-mel cosine	0.999898629
Q8 vs F32 mean absolute sample error	0.0002294
Q8 vs F32 maximum absolute sample error	0.0044556
F32 vs Python waveform cosine	0.989149342
F32 vs Python log-mel cosine	0.998817655

The Q8 result is therefore numerically and perceptually nearly identical to
the native F32 result for the tested checkpoint while reducing the weight
file to about 27% of its original size.

The lower short-input Python waveform cosine comes from the different
edge/chunk-overlap policy of the complete audio-separator inference path;
the exact frame count and high log-mel similarity provide the more useful
parity evidence for this test.

Request timings

Path	Input	Result
Native server HTTP, including lazy load/base64/file decode	8.0 s	3,032.1 ms
WebUI visible action (session.wall_ms)	8.0 s	2,676.2 ms
WebUI reported end-to-end separation	8.0 s	2.7 s
WebUI standalone-GGUF model load	—	0.5 s

RSS and peak VRAM were not instrumented for this run.

Known limitations

• The validated checkpoint is a one-target vocals model. It produces vocals
  directly and derives instrumental from the mixture.
• Native CLI/server input is WAV; this checkpoint expects 44.1 kHz audio.
  The WebUI normalizes supported uploads before sending them to the server.
• CUDA on RTX 3090 is the validated backend. CPU, Vulkan, and Metal were not
  performance-tested in this contribution.
• Other BS-RoFormer checkpoints may use additional architectural variants and
  should be parity-tested before being declared compatible.

The GGUF model is here

https://huggingface.co/mirek190/audio.cpp/tree/main/vocal%20separation%20models

[0xShug0]

Thanks @mirek190! I’m going to hold off on merging new models until the model spec redesign and migration are complete #89. The current design is becoming too costly to maintain.

[mirek190]

Pushed follow-up commit f78b6d2 (Optimize BS-RoFormer CUDA inference).

What changed

• CUDA axial attention now uses the shared ScaledDotProductAttentionModule with FlashPreserveViews, non-causal attention, and F32 accumulation.
• CPU and non-CUDA backends retain the previous explicit F32 attention path.
• Attention gates now use GGML broadcasting directly instead of materializing a repeated gate tensor.
• First/last/single-chunk fade windows are cached once per session instead of copied and modified for every chunk.
• Added the session option bs_roformer.num_overlap. The package value (4 for ep368) remains the default; 2 and 1 are opt-in faster modes.
• A single-pass request now uses a correct two-sided edge window.
• Updated docs/audio_tools.md and tests/bs_roformer/README.md with the control, commands, parity, and performance results.

Performance

Backend: CUDA 12.4, NVIDIA GeForce RTX 3090, 24 GB. Input: 8.0-second stereo 44.1 kHz WAV, 352,800 output frames.

Route	session.wall_ms	Throughput
Before this commit, Q8, overlap 4	2,643.1	3.03x real time
Optimized Q8, overlap 4 (default)	1,440.4	5.55x real time
Optimized F32, overlap 4	1,874.3	4.27x real time
Optimized Q8, overlap 2	734.3	10.89x real time
Optimized Q8, overlap 1	384.0	20.84x real time

The quality-oriented default is therefore about 45.5% lower latency / 1.84x faster than the pre-change CUDA route.

Resident-server per-request results:

Request	Internal session.wall_ms	End-to-end HTTP
1	1,731.0 ms	2,235.7 ms
2	1,401.1 ms	1,496.0 ms
3	1,422.0 ms	1,520.0 ms

Both server-returned WAV artifacts (vocals, instrumental) were byte-identical to the CLI artifacts.

Parity and quality

Comparison	Waveform cosine	Log-mel cosine
Optimized default Q8 vs saved pre-change Q8	0.999996424	0.999937057
Optimized Q8 vs optimized F32	0.999993563	0.999881983
CUDA vs explicit CPU fallback, overlap 1	0.999995887	0.999922335
Overlap 1 vs quality default overlap 4	0.989162147	0.999110937

The last row is why lower overlap remains opt-in rather than becoming the default.

Commands used

Build:

.\scripts\build_windows.ps1 -Target audiocpp_cli -Jobs 16
.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16

Default quality route:

.\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task sep --family bs_roformer `
  --model ..\models\BS-RoFormer-ep368_Q8\BS-RoFormer-ep368_Q8.gguf `
  --backend cuda `
  --audio ..\outputs\bs_roformer_benchmark\input_8s.wav `
  --out-dir ..\outputs\bs_roformer_benchmark\optimized_final `
  --log

Optional single-pass route:

.\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task sep --family bs_roformer `
  --model ..\models\BS-RoFormer-ep368_Q8\BS-RoFormer-ep368_Q8.gguf `
  --backend cuda `
  --audio ..\outputs\bs_roformer_benchmark\input_8s.wav `
  --out-dir ..\outputs\bs_roformer_benchmark\overlap1_final `
  --session-option bs_roformer.num_overlap=1 `
  --log

CPU fallback:

.\build\windows-cuda-release\bin\audiocpp_cli.exe `
  --task sep --family bs_roformer `
  --model ..\models\BS-RoFormer-ep368_Q8\BS-RoFormer-ep368_Q8.gguf `
  --backend cpu --threads 16 `
  --audio ..\outputs\bs_roformer_benchmark\input_8s.wav `
  --out-dir ..\outputs\bs_roformer_benchmark\cpu_fallback `
  --session-option bs_roformer.num_overlap=1 `
  --log

The CPU fallback completed in 20,463.9 ms.

Experiments not retained

• Batching 2 or 4 overlap chunks did not improve latency on the RTX 3090 (~1.60-1.63 s versus ~1.58 s for the comparable serial Flash route) and required substantially more activation memory.
• Converting the Q8 source weights to F16 storage was slower (~1.58 s) and increased graph-build time.
• Lower Flash precision produced no speed improvement and reduced parity, so F32 accumulation is retained.

[mirek190]

Thanks @mirek190! I’m going to hold off on merging new models until the model spec redesign and migration are complete #89. The current design is becoming too costly to maintain.

Sure ;)

But that as you know is not a real model.
Rather a tool for models which is separating a vocal from music like already present htdemucs and mel_band_roformer in the audio.cpp

[mirek190]

Thanks @mirek190! I’m going to hold off on merging new models until the model spec redesign and migration are complete #89. The current design is becoming too costly to maintain.

Do you need a help with it?

[0xShug0]

@mirek190 I could definitely use a hand. I’ll let you know.

[0xShug0]

@mirek190 Would you be interested in taking a stab at the version of GPT_SoVITS suggested in #92?

[mirek190]

@mirek190 Would you be interested in taking a stab at the version of GPT_SoVITS suggested in #92?

I checked that already and think it is not worth it to add it currently to audio.cpp.
That tts cloning model is a quite old ( 2 years ) . We have much better models for it like nemotron , qwen , higgs and others even with emotions tags.

GPT_SoVITS has already built frameworks around that model all-in-one so is working ...more or less ;)

I think first we should make stable audio.cpp first then adding new functionalities like GPT_SoVITS

But is worth to add asr models which GPT_SoVITS is using as well.
https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512
https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512

[0xShug0]

@mirek190 I thought you were a fun of retro models :) No worries. GPT_SoVITS model is just for completeness.

If I remember correctly Fun-ASR has their official GGML impls.

Two models I’m planning to add to the core framework are WordVoice and the Zipformer-based models such as Kroko-ASR. WordVoice brings novel capabilities, while Kroko-ASR could introduce a new set of reusable building blocks. Let me know if you’re interested in working on either of them.

[mirek190]

OK I check that tomorrow