Open Source Project #241: YuE2 — The Open Music Model That Uses Symbolic Scores as a 'White-Box' Interface, Beating Suno v5/v6 on SongBench

Introduction

"Compose in symbols. Create in sound."

This is article #241 in the "One Open Source Project a Day" series. Today's project is YuE2 — an open foundation model that unifies symbolic music generation and audio music generation, jointly developed by HKUST, M·A·P, Tokenwave.AI, NYU, Stanford, MBZUAI, NOIZ, and ACE Studio. It has 12,280 Stars, hit #1 on GitHub's daily trending across all languages, peaked at #3 on Hugging Face's site-wide trending models, and reached #1 in the Text-to-Audio category trend.

Most AI music generation tools today are black boxes: you give a style description and a lyric, and it spits out a finished audio clip — what happens in between, whether it can be edited, and where to edit it, is completely invisible and untouchable. YuE2's core thesis flips that around: first generate a melody-and-chord score that both humans and agents can read, inspect, and modify — a symbolic "composition plan" — then render that score into a complete song. This is the first time music generation has had an intermediate artifact that can actually be intervened on, rather than a one-shot end-to-end output.

What You'll Learn

• The "symbolic planning + audio rendering" two-stage architecture: why generate a score first instead of generating waveforms end-to-end
• How zero-shot covers work without any cover-specific fine-tuning
• An Agent Skill-driven multi-turn music editing workflow: how conversation can modify harmony, melody, tempo, or even an entire arrangement
• Quick start: from one pip command to your first generated song
• How the project measures up against Suno v5/v6, Mureka 9, and other commercial systems on WildSongBench

Prerequisites

• Basic familiarity with Python virtual environments (venv/pip)
• Some grasp of ABC notation or basic music theory (melody, chords, key) helps with understanding the symbolic plan output, though it's not required
• Running locally requires an NVIDIA GPU with BF16 support — a rough sense of VRAM requirements is useful

Project Background

Overview

YuE2 is the second-generation evolution of the YuE project (the original YuE's code, docs, and license remain preserved on the YuE-v1 branch). It uses an AR-NAR Mixture-of-Transformers backbone: it first autoregressively (AR) predicts a symbolic score and semantic tokens, then generates acoustic latents via flow matching (a non-autoregressive, NAR process), and finally decodes those latents to stereo audio through a VAE decoder. This single architecture supports three scenarios at once — composing from scratch, covering an existing song, and iteratively editing a previously generated piece — the only difference between them being where the score comes from (generated by YuE2 itself, transcribed from an audio recording, or edited by a human or agent).

The project also ships several companion models/datasets: MERT2 (a music understanding/representation model), SheetSage2 (an audio-to-score transcription model), and the WildSongBench evaluation dataset — together forming a fairly complete "understand → generate → evaluate" music AI toolchain, rather than an isolated generation model.

Author / Team

• Developed by: the multimodal-art-projection (M·A·P) organization, jointly with HKUST, Tokenwave.AI, NYU, Stanford, MBZUAI, NOIZ, ACE Studio, and other institutions
• Primary language: Python
• License: code under Apache 2.0; model weights under CC BY-NC 4.0 with additional creator permissions
• Created: 2025-01-23
• Status: actively developed; YuE2 and SheetSage2 technical reports both landed in a concentrated burst in September–October 2026

Project Stats

• ⭐ GitHub Stars: 12,280+
• 🍴 Forks: 1,409+
• 📄 License: code Apache 2.0 / model weights CC BY-NC 4.0 + additional terms
• 📅 Created: 2025-01-23
• 🏆 #1 on GitHub's cross-language daily trending (2026-09-14); peaked at #3 on Hugging Face's site-wide model trending (09-17); #1 in the Text-to-Audio category trend (09-20)
• 👥 Core contributors: led by a43992899 (143 commits), followed by hf-lin (24), Yiming-Liang, instr3, and others
• 🏷️ Topics: audio-generation, foundation-models, music-generation, style-transfers, voice-cloning, and more

Quick Start

Requirements

Linux · Python 3.12 · an NVIDIA GPU with BF16 support, 24GB VRAM. YuE2 outputs 48kHz stereo audio directly, with no lossy intermediate compression. Model files are downloaded automatically from Hugging Face on first use.

Generate from the Command Line

git clone https://github.com/multimodal-art-projection/YuE.git
cd YuE
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install .
python examples/generate.py --output outputs/first-song

Open outputs/first-song/audio.flac to hear the result. The output directory also preserves the score, semantic tokens, acoustic latents, generation parameters, and the exact model identifiers used along the way — a direct, artifact-level manifestation of the "white-box" design, where every intermediate step is kept inspectable.

Python API

import json
from pathlib import Path
from yue2 import YuE2Pipeline
 
request = json.loads(Path("examples/song.json").read_text(encoding="utf-8"))
with YuE2Pipeline.from_pretrained("m-a-p/YuE2-3B", device="cuda") as pipe:
    song = pipe(**request)
    song.save_artifacts("outputs/my-song")
    print(song.truncated)

The generation pipeline exposes a staged, four-step API: plan() → generate_semantic() → synthesize() → decode(), each of which can be called, inspected, or substituted independently. The cot (chain-of-thought, here specifically meaning symbolic planning) parameter controls planning depth: "full" produces an editable, full melody-and-chord plan (the default for new songs); "melody" produces a melody-only plan with free accompaniment (recommended for covers); "off" generates directly from the lyric and style description end-to-end, skipping the symbolic planning stage entirely.

Core Deep-Dive

Symbolic Planning: Turning "Black-Box Generation" Into a "White-Box, Editable" Interface

This is the single most central idea in YuE2's design. Traditional end-to-end music generation models — including most commercial products — jump directly from "lyric + style" to an audio waveform, with nothing in between visible or controllable: changing one note means regenerating the entire song, with no way to specify exactly where the change should go. YuE2 inserts an explicit intermediate artifact between "lyric + style" and the final audio: a melody-and-chord score written in ABC notation.

That score is a genuine white-box interface: a human or agent can read it directly, view it in an editor, or even preview it in music software to confirm the overall structure and direction before deciding whether to render it to audio at all. The README demonstrates this with a concrete multi-turn editing example — a reworking of "The Last Train" that went through 9 steps and 14 versions, moving from Mandarin pop all the way to an English jazz arrangement with a saxophone solo, with the full conversation log, score, prompts, and lyrics for every version preserved for playback and comparison. That kind of traceable, replayable editing history is structurally impossible on an end-to-end black-box model.

Zero-Shot Covers: Transforming a Song Without Any Cover-Specific Fine-Tuning

Cover generation usually requires scenario-specific fine-tuning to do well, but YuE2's approach reuses the same general-purpose generation model with no cover-specific training at all. The workflow: use the companion SheetSage2 transcription model to convert a source recording into a melody-only ABC score, review and edit it, pair it with new lyrics or a target style, and regenerate with cot="melody" (deliberately omitting chord symbols from the score so the accompaniment is free to adapt to the new style).

The quantitative result the project reports is striking: across 948 pieces, a YuE2 cover generated with a full score reached 0.647 CLEWS mAP (a metric measuring whether a cover preserves the identity of the source piece), while a control group generated without any score scored only 0.006 — nearly two orders of magnitude lower. This also makes a point about the value of the symbolic planning layer itself: the gap isn't about model scale, it's about whether an explicit score constraint exists at all.

Agent Skill: Making Conversational Editing the Default Music-Creation Workflow, Not a Gimmick

YuE2 ships a dedicated yue2-music Agent Skill, teaching an agent how to generate songs and instrumental music, transcribe and cover recordings, edit ABC scores, check musical invariants after an edit (e.g., whether pitches and onset times were unintentionally broken), and organize listening comparisons. The README explicitly recommends pairing it with GPT-6 Astra as the executing agent, while making clear that YuE2 itself still does the actual composing and audio generation — the Skill just packages "the correct way to call YuE2" for an agent, rather than replacing YuE2's generative capability.

One detail worth noting is how the instrumental-music workflow is actually implemented: the default path is "YuE2 generates a full score → vocal-part notes are moved to an instrumental part → render," rather than training a separate "instrumental-only model." When converting an audio cover into an instrumental version, the supporting tooling preserves the original vocal notes' pitch, onset time, and duration during score conversion, and flags any overlapping parts — but the README is also upfront that this step still requires a human listening check afterward, to confirm there's no vocal "bleed-through" and that melodic fidelity holds up. It doesn't dress this step up as a fully automatic, zero-risk black box — that kind of "the tool gets you this far, the rest is for a human ear to judge" boundary statement is more credible than vague marketing.

Side by Side With Commercial Systems — Honest Enough to Flag Its Own Headline Number as Possibly Not Significant

YuE2 doesn't shy away from direct comparison with Suno, Mureka, and other leading commercial systems, publishing a full leaderboard on WildSongBench (192 prompts, fully automated evaluation):

YuE2 (best-of-8, i.e., picking the best of 8 candidates) achieved the highest SongBench average among all 17 evaluated configurations in this benchmark, and the standard single-shot YuE2 ranks fourth, just behind Suno v5 and ahead of Mureka 9. But the sentence that immediately follows that result is worth noting: the README states directly that "rankings vary by metric, and the small gap between the highest means does not establish statistical significance" — it doesn't dodge that caveat just because its own number happens to top the chart. Flagging "don't take this too literally" right next to your own best-looking data is a more honest move than simply announcing a #1 ranking and leaving it there. Relative to the original YuE (SongBench average of just 4.9165), the improvement in under two years is also substantial.

Resources

• 🌟 GitHub: multimodal-art-projection/YuE
• 📄 Technical reports: YuE2 arXiv / SheetSage2 arXiv
• 🎧 Demos and samples: map-yue2.github.io
• 🚀 Free online demo: yue.noizai.net (hosted by NOIZ, no install required)
• 🗳️ Blind-vote arena: Music Arena
• 🤗 Models: YuE2-3B / MERT2 / SheetSage2
• 📊 Evaluation dataset: WildSongBench
• 🎛️ Ecosystem integrations: an official ComfyUI workflow, NOIZ's inference acceleration suite YuE2-Turbo, the Maestro local creation tool
• 💬 Community: Discord

Summary

YuE2 answers a question that's been largely overlooked in AI music generation: catching up to commercial systems on generation quality matters, but if the creation process stays a one-shot black box, users can only ever "reroll" rather than precisely edit — and that limitation alone keeps AI music tools from ever fitting into a real professional creative workflow.

Three things worth noting:

Using a symbolic score as an explicit intermediate artifact is the most fundamental design choice separating this project from most end-to-end music generation models. This isn't just a technical detail — it's exactly what determines whether "music generation" can graduate from one-shot rerolling into an iterative, traceable creative process. The 9-step, 14-version editing demo proves this is practical, not just a nice-sounding theory.

The 0.647-versus-0.006 gap on zero-shot covers proves the symbolic planning layer isn't a nice-to-have — it's decisive. Many projects struggle to produce a clean controlled experiment that actually proves an architectural choice matters; YuE2 answers that question with a near-two-orders-of-magnitude difference.

Flagging "this gap may not be statistically significant" right next to its own chart-topping result, and being upfront that the instrumental-conversion step "still needs a human listening check," reflects an unusual degree of engineering restraint. That kind of self-imposed caveat is rare among AI generation projects that lean heavily on benchmark rankings, and it's an important signal for judging whether a project's reported numbers can actually be trusted.

If you're interested in whether AI-generated music can be inspected, edited, and version-controlled the way code can — or you're looking for an open-source music model that genuinely supports editing rather than one-shot generation — YuE2's symbolic planning architecture and its companion Agent Skill workflow are well worth trying.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.