- MCP Server with audio fallback, sample management - Song generator with bus routing - Reference listener and audio resampler - Vector-based sample search - Master chain with limiter and calibration - Fix: Audio fallback now works without M4L - Fix: Full song detection in sample loader Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
23 KiB
AbletonMCP-AI Full Handoff
This file is the broadest handoff in the repo.
If another AI needs to retake the project with minimal context loss, this is the file to read first.
Project root:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI
Current local branch:
main
Last pushed commit at the moment this file was updated:
2a0d2f3dbf5f89b18690fee2a2659957f81b8191
Read Order
Read in this order:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\CLAUDE.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\README.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\AI_HANDOFF.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\ARCHITECTURE.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\MCP_TOOLS.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\REMOTE_PROTOCOL.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\PROJECT_CONTEXT.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\GPU_DIRECTML.md
Useful secondary docs:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\ABLETUNES_TEMPLATE_NOTES.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\SAMPLE_SYSTEM_README.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\ROADMAP_MASTER_GLM.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\automation\README.md
What This Project Is
AbletonMCP-AI is a hybrid system to control Ableton Live 12 from MCP, generate long editable arrangements, analyze reference tracks, retrieve similar material from a local sample library, and build original projects that feel closer to a real producer workflow than to a loop toy.
The desired output is:
- always Arrangement View
- editable tracks and clips
- many specialized roles
- buses and returns
- original output, not stems from the reference track
Core Design Rules That Must Not Be Lost
- Arrangement-first is mandatory.
- Reference audio is for analysis, not plagiarism.
- Do not use stems from the target song in the final output.
- The system must keep working even if the M4L path is incomplete.
- Stable fallback is better than a flashy broken feature.
- The local sample library is the primary sound source.
- Validation and diagnosis matter because the stack is large and brittle.
Important Paths
Main runtime:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\__init__.py
MCP server:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\server.py
Music generator:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\song_generator.py
Reference analysis:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\reference_listener.py
Sample selection:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\sample_selector.py
Audio resampling:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\audio_resampler.py
Socket smoke test:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\socket_smoke_test.py
Segment RAG builder:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\segment_rag_builder.py
Local library:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\librerias\all_tracks
Reference folder:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\sample
Ableton log:
C:\Users\ren\AppData\Roaming\Ableton\Live 12.0.15\Preferences\Log.txt
Recovery popup file:
C:\Users\ren\AppData\Roaming\Ableton\Live 12.0.15\Preferences\CrashRecoveryInfo.cfg
User library:
C:\Users\ren\Documents\Ableton\User Library
External Assets And References Used During Development
Reference tracks that shaped the direction:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\sample\Eli Brown x GeezLy - Me Gusta.mp3C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\sample\Mr. Pauer, Goyo - Quimica (Video Oficial).mp3
Producer template reference pack:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\Abletunes_Free_Templates_Pack
The reference tracks were used for:
- BPM and key estimation
- section and energy direction
- sample retrieval guidance
- stylistic remake goals
They were not supposed to be used as final stems.
The Story From The Beginning Until Now
Phase 0: Initial Goal
The original goal was not only to make sounds in Ableton, but to give the system the ability to:
- receive a prompt or a reference MP3
- understand the style and structure
- choose similar sounds from the local library
- generate a long arrangement that feels professionally produced
- keep the result editable inside Ableton
Very early it became clear that a plain Session View loop machine was not enough.
Phase 1: Make The Remote Script Actually Work
The first major work was stabilizing the remote layer between MCP and Ableton.
Main problems solved in that phase:
- command naming mismatches between MCP and the Remote Script
- parameter normalization like
track_index,clip_index,scene_index - note writing API mismatch in Live
- socket protocol mismatch
- stale or broken sample manager initialization
This phase made the project usable enough to:
- create tracks
- create clips
- write MIDI
- query session state
- build the first generated projects
Phase 2: Arrangement-First Pivot
At that point the output still behaved too much like Session clips and loops. The user explicitly wanted to see everything in Arrangement View.
That created the second major architectural pivot:
- generate blueprint in Session
- commit Session to Arrangement in ordered scene playback
- place audio fallback and overlays in Arrangement
This became the stable route.
Important lesson:
- precreating certain audio tracks before the Session to Arrangement commit produced silent or broken sets
- the stable route is still: Session blueprint first, Arrangement commit second, audio layers after that
Phase 3: Richer Project Shape
Once Arrangement-first worked, the next problem was musical complexity. The output felt like repeated loops.
The generator was expanded with:
- more track roles
- section-aware pattern generation
- richer drums, bass, melodic and FX layers
- scene naming and locators
- guide tracks
- more realistic arrangement structures
Important roles that became standard:
- kick
- clap
- snare fill
- hat closed
- hat open
- top loop
- percussion
- ride
- tom fill
- sub bass
- bass
- drone
- chords
- pluck
- vocal chop
- stab
- pad
- arp
- lead
- counter
- crash
- reverse FX
- riser FX
- impact FX
- atmos
- plus many
AUDIO ...tracks
Phase 4: Local Library As Primary Source
The next major issue was weak selection quality. At one point the code was pointed to a small mirror path and not to the real large library.
The real usable library was:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\librerias\all_tracks
That directory had hundreds of real audio assets, while the old fallback mirror only had a few dozen.
Fixing that changed the quality of:
- drum matching
- bass loop matching
- vocal loop matching
- FX and atmos selection
This was a major turning point because the generated output stopped sounding starved.
Phase 5: Reference-Led Reconstruction
After the library path was fixed, the project started using references more seriously.
The target workflow became:
- analyze the reference
- infer BPM, key, energy, style and section behavior
- find similar sounds in the local library
- reconstruct a new original track
There was a temporary detour where stems of a reference song were materialized for analysis, but that was explicitly rejected because the desired product is not a stem-based copy workflow.
The rule became strict:
- no final stems from the reference
- only original reconstruction using local assets and generated MIDI
Phase 6: Buses, Returns, Master, And Production Logic
The project then moved from "long loop" to "production-shaped session".
Major additions:
- real bus tracks
- return tracks
- track routing into buses
- role-based mixing
- return snapshots by section
- master chain blueprint
Typical bus layout now:
- drums
- bass
- music
- vocal
- FX
Typical return layout now:
- space
- echo
- heat
- glue
This phase made the result feel more like a produced project and less like independent loops.
Phase 7: Template Analysis
The project analyzed professional Abletunes templates to absorb real producer patterns.
Main conclusions:
- professional templates are Arrangement-heavy
- there are many specialized layers, not just one drum loop and one bass loop
- sidechain triggers, transitions, buses, returns and printed audio are common
- arrangement blocks are often 16, 32 or 64 bars
- heavy automation is normal
This analysis informed later changes in:
- section shape
- layer count
- transition behavior
- bus logic
Phase 8: Audio Fallback And Hybrid Output
The system learned to combine:
- MIDI and stock instruments
- local audio loops and hits
- special
AUDIO ...overlays
This was critical because:
- sometimes stock-device generation gives editability and musical logic
- sometimes local audio assets give the genre-specific realism that MIDI alone cannot deliver
So the stack became hybrid by design, not by accident.
Phase 9: Audio Resampling And Derived FX
Then came the derived transition layer.
The system added support for:
- reverse FX
- riser
- downlifter
- stutter
This area improved through:
audio_resampler.py- reference-driven placement
- later bugfixes for short clips and defensive rendering
One real bug that had to be solved:
AUDIO RESAMPLE STUTTERfailed until the resampler and short-clip FFT handling were hardened
That was fixed later and validated in real Live runs.
Phase 10: GPU And Deeper Reference Listening
The user specifically asked to use GPU if possible.
The system moved toward:
DirectML- more expensive analysis of the reference
- segmented analysis at multiple window sizes
- heavier similarity scoring
This became one of the most important changes for retrieval quality.
The reference listener started doing:
- segmenting the reference into short windows
- scoring by role
- deeper reranking
- using local metadata and cache
This was the start of an audio-retrieval-style workflow rather than simple filename matching.
Phase 11: Segment RAG Direction
To go further, the project started building a segment-level retrieval cache for the local library.
That work added:
- persistent per-segment cache files
- a segment builder CLI
- partial index expansion over the real library
This is not a text RAG in the usual sense. It is closer to:
- segmented audio retrieval
- coarse search plus rerank
- role-aware filtering
- diversity constraints
This remains one of the most promising long-term directions for better remake quality.
Phase 12: GLM Workflow
At some point token efficiency became a concern, so GLM-5 was introduced as a worker model.
The workflow that proved useful was:
- Codex writes a narrow
.mdtask - GLM edits only 1 to 3 related files
- Codex reviews the diff
- Codex corrects technical mistakes and validates the runtime
What GLM was good at:
- narrow feature implementation
- heuristic expansion
- helper tools
- reports and manifests
What GLM was bad at:
- declaring things complete too early
- runtime-sensitive work without supervision
- architectural judgment
- avoiding diff inflation
Practical verdict:
- useful as a worker
- not reliable enough as the sole closer of large features
Phase 13: Temporary WSL / n8n / Local Gitea Automation Detour
There was a detour into WSL orchestration, n8n, local Gitea and Telegram loops.
Scaffolding was generated for:
- WSL deployment
- Docker Compose
- n8n flows
- Telegram notifications
- GLM to Codex automation loops
That stack produced a lot of files under:
automation\automation\wsl\
But the key lesson was:
- the deployment summary was overstated
- the generated stack was not a truly finished deployment
- the main product value still lives in the music system, not in orchestration
Because of that, the project intentionally pivoted back to the flow that worked:
- Codex writes focused
.md - GLM does bounded work
- Codex reviews and fixes
Phase 14: Retrieval Hardening, Pattern Banks, Transition Materialization
After the orchestration detour, the roadmap was re-centered on the actual product.
Task batches improved:
- role-safe retrieval
- repetition penalties
- more section pattern banks
- transition event materialization
This reduced some of the "same loop again" feeling, but did not solve everything.
Phase 15: Device Automation Snapshots
The latest work pushed into:
- track device parameter snapshots
- return device parameter snapshots
- master section snapshots
- automation summaries
GLM implemented part of that, but the raw result was not correct.
The real fixes applied afterward were:
- use the already existing
SECTION_DEVICE_AUTOMATIONinstead of duplicate tables - use the already existing
MASTER_DEVICE_AUTOMATION - flatten
device_parametersinto the format the server actually applies - make the server accept both flat and nested payloads defensively
- add
mix_automation_summary - add
mix_automation_warnings
This work lives mainly in:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\song_generator.pyC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\server.py
Phase 16: Fix for "Only Piano" Issue (Audio Samples Not Loading)
The user reported: "lo que me generaste solo tiene algunos pianos, nada de sonidos de mi biblioteca!" - the generated tracks only had piano/MIDI sounds, no actual audio samples from the local library.
Root causes found and fixed:
-
ROLE_PATTERNSused non-recursive globs inreference_listener.pylines 1228-1254:- Patterns like
'*Kick*.wav'couldn't match files in subdirectories likeloops/kick/ - Fixed by changing to recursive patterns:
'**/*Kick*.wav'
- Patterns like
-
Method resolution bug in
reference_listener.py:_get_role_section_featuresand_section_character_bonusare methods inSectionDetectorclass- Were being called as
self._methodfromReferenceAudioListenerinstead ofself._section_detector._method - Fixed at lines 3247 and 3270-3272
After fixes, verification showed:
- Reference audio plan builds correctly with 30+ layers
_materialize_reference_audio_layers()creates actual audio tracks- Real samples from local library are used (e.g., "mt kick hit 10.wav", "bbh - primer impacto - bass loop 03 g#m.wav")
- 34 audio tracks created (8 derived + 26 base), 0 errors
What Currently Works
The system can currently:
- generate full projects in Arrangement View
- build MIDI plus stock-device layers
- build many
AUDIO ...layers from the local library - analyze a reference track
- build a retrieval-guided original result
- commit scenes to Arrangement
- create buses and returns
- route tracks into buses
- apply track, return and master snapshots
- diagnose the generated set
- validate the generated set
- use DirectML for deeper matching work
What Is Stable
The most stable route today is:
- analyze reference if one is available
- build config in
song_generator.py - materialize Session blueprint through the runtime
- commit Session to Arrangement
- place audio layers in Arrangement
- validate and diagnose
Do not casually change this order.
What Is Still Weak
The project still has real weaknesses:
- some generations still feel too loop-based
- retrieval can still pick poor family matches
- remake quality is not yet close enough to a convincing stylistic remake
- some runs still overuse familiar sound families
- loudness and gain staging can vary too much between runs
- derived resample layers are not always present in the final set
- the MCP wrapper for
generate_trackcan time out even when Live keeps working
Known Operational Problems
1. Recovery popup can block everything
If Live crashes or thinks recovery is needed, a modal popup can block the socket.
The file involved is:
C:\Users\ren\AppData\Roaming\Ableton\Live 12.0.15\Preferences\CrashRecoveryInfo.cfg
Practical fix used during development:
- kill Ableton
- blank that file
- restart Ableton
2. MCP wrapper timeout
The MCP tool wrapper around generate_track often times out at about 120 seconds.
Important:
- timeout does not always mean generation failed
- often the set keeps building inside Live
Safer checks after a timeout:
get_session_info()get_tracks()validate_set()diagnose_generated_set()
3. Runtime state can drift from stored manifest
At least once, get_generation_manifest() returned stale data from an older generation while the actual current set in Live was already different.
Trust runtime state first:
- session info
- track list
- diagnosis
- validation
Manifest is useful, but not always the freshest source.
4. GLM reports often exaggerate completion
Never trust a GLM report by itself. Always compare:
- report
- actual diff
- runtime result
Current Validation Habit
Minimum technical checks after code edits:
python -m py_compile "C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\audio_resampler.py"
python -m py_compile "C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\reference_listener.py"
python -m py_compile "C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\sample_selector.py"
python -m py_compile "C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\server.py"
python -m py_compile "C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\song_generator.py"
python -m py_compile "C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\__init__.py"
Minimum Live checks:
get_session_info()get_tracks()validate_set(check_clips=True, check_gain=True, check_routing=True)diagnose_generated_set()
Useful direct smoke test:
cd "C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server"
python socket_smoke_test.py
Current Roadmap Priority
The active roadmap is:
- better library retrieval
- more real section variation
- better transition and derived FX placement
- better device automation
- gain staging and loudness consistency
- better remake ability
- stronger QA and export
The roadmap source file is:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\ROADMAP_MASTER_GLM.md
Current GLM Workflow
Keep GLM on short, bounded tasks only.
Recent task files:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_NEXT_TASK_001_RETRIEVAL_ROLE_PENALTIES.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_NEXT_TASK_002_SECTION_PATTERN_BANK_EXPANSION.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_NEXT_TASK_003_TRANSITION_EVENT_MATERIALIZATION.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_NEXT_TASK_004_DEVICE_AUTOMATION_SNAPSHOTS.md
Corresponding reports:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_TASK_001_REPORT.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_TASK_002_REPORT.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_TASK_003_REPORT.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\GLM_TASK_004_REPORT.md
Recommended pattern:
- Codex writes one narrow task md
- GLM edits only 1 to 3 files
- Codex reviews diff
- Codex corrects technical and runtime mistakes
- only then decide whether the task is really done
Current Documentation Map
Core repo docs:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\README.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\CLAUDE.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\AI_HANDOFF.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\ARCHITECTURE.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\MCP_TOOLS.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\REMOTE_PROTOCOL.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\PROJECT_CONTEXT.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\docs\GPU_DIRECTML.md
Generator and retrieval docs:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\ABLETUNES_TEMPLATE_NOTES.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\MCP_Server\SAMPLE_SYSTEM_README.md
Roadmaps and backlog:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\roadmap.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\roadmap2.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\ROADMAP_MASTER_GLM.md
Automation detour docs:
C:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\automation\README.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\automation\MASTER_AUTONOMOUS_ROADMAP.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\automation\wsl\README.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\automation\wsl\DEPLOYMENT_SUMMARY.mdC:\ProgramData\Ableton\Live 12 Suite\Resources\MIDI Remote Scripts\AbletonMCP_AI\WSL_STACK.md
If You Need To Resume Development Safely
Start here:
- read this file
- read the current roadmap
- inspect local git status
- compile changed Python files
- restart Live if
__init__.pychanged - clear recovery popup if needed
- validate with
get_session_info,get_tracks,validate_set,diagnose_generated_set - only then touch generation logic
Current Honest Product Status
This project is no longer a basic prototype.
It is already an advanced Ableton generation system that can:
- create long arrangements
- use references intelligently
- retrieve from a large local library
- mix MIDI and audio material
- build buses, returns and snapshots
- generate editable results in Arrangement View
But it is still not finished.
The main gap is no longer raw plumbing. The main gap is artistic fidelity:
- better retrieval
- better variation
- better remake quality
- more consistent mix quality
That is the real work that remains.