Kyutai Releases MuScriptor: An Open-Weight Decoder-Only Transformer for Multi-Instrument Music Transcription to MIDI

Automatic Music Transcription (AMT) converts an audio recording into symbolic notes, usually MIDI. Single-instrument transcription already works reasonably well. However, transcribing a full multi-instrument mix stays difficult. Kyutai and Mirelo team now release MuScriptor to close that gap. It is an open-weight model trained on real, multi-instrument recordings across many genres.

This article explains how MuScriptor works, what the benchmarks show, and how to run it.

At its core, MuScriptor is a decoder-only Transformer for music transcription. First, it reads a mel-spectrogram of a short audio segment. Then it autoregressively predicts MIDI-like tokens for pitch, timing, and instrument. In effect, transcription becomes a language-modeling task, following the MT3 tokenization scheme.

The release ships three weight variants on Hugging Face. Their sizes are small (103M), medium (307M, default), and large (1.4B). The inference code uses the MIT license. The weights use CC BY-NC 4.0, so commercial use is restricted.

MuScriptor’s main idea is data, not architecture. Accordingly, training moves through three stages, and each builds on the last.

For evaluation, the research team use DTest , 372 held-out tracks with accurate annotations. They report instrument-agnostic metrics from the mir_eval library. Among them, Multi F1 is strictest, since it also requires the correct instrument.

The table below traces each training stage against the YourMT3+ baseline, using the large (~1.3B) model.

Clearly, every stage improves results, and real data matters most. Synthetic-only training reaches competitive frame F1 but weak onset and multi scores. Adding DReal then lifts all metrics by roughly 20 points. Finally, RL post-training reduces false negatives and sharpens onset timing.

Cross-dataset tests point the same way. For example, frame F1 on Dagstuhl ChoirSet rises from 51.0 to 80.7. Even so, onset and offset stay lower on hard styles like chorals.

Installation takes one command, and inference streams note events directly.

For the released models, keep cfg_coef at 1, since they are already RL post-trained. Additionally, uvx muscriptor serve launches a browser web UI with a live piano roll.

Because the output is standard MIDI, many workflows open up:

Check out the Paper , GitHub Repo and Model Weights . Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter . Wait! are you on telegram? now you can join us on telegram as well.