specy/genshin-music/instrument-from-sequential-capture

Instrument from a sequential capture

The scripted version of the workflow that produced genshin/instruments/NightwindHorn (2026-08-13): one in-game recording of all 14 notes became a shipped, tested instrument. Capture protocol for individual takes lives in docs/skills/recording-sustained-instruments; loop-point work lives in docs/skills/audio-loop-analysis; this skill is the splitter and the registration checklist between them.

What the capture must look like

• Every button in meta.json notes order (top-left → bottom-right of the

keyboard) — segment order becomes button order, nothing reorders it.

• Sustained instruments: hold each note 6–12 s (the hold IS the max sustain a

player gets on a loopless instrument). Tap instruments: clean single hits.

• Let every note decay into real silence, with ≥ 1–2 s of gap — the

segmenter ends a note after 0.35 s below −55 dBFS, so bleed/reverb between notes merges them.

• Lossless WAV at the source rate is best; a high-bitrate MP3 capture works

(NightwindHorn came from one). Stereo is fine — output is downmixed to the house mono format.

Run the pipeline

Deps are one-off tooling, deliberately not in package.json (ESM resolves them from the repo root's node_modules; package.json/lockfile stay untouched):

npm i --no-save mpg123-decoder @breezystack/lamejs

Always dry-run first and read the table before writing any file:

node docs/skills/instrument-from-sequential-capture/scripts/extract-notes.mjs \
  capture.mp3 --dry-run --expect 14 --midi 60,62,64,65,67,69,71,48,50,52,53,55,57,59

• --expect fails loudly on a miscount — tune --on-db/--off-db/--gap-s for

noisy or bleed-heavy captures rather than accepting a wrong split. Extra presses after the last button (test notes) are real segments, not a tuning problem: cut the capture before them first (decode, keep the first N seconds as WAV — Vodyanitsa's capture was cut this way).

• --midi is the authored Note Ids in capture order. Omit it to use rounded

detected pitch, then CHECK the printed ids — they are identity (ADR-0001) and must be unique and inside the game's Song Grid for composer/import to work.

• **Nominal Ids name grid rows, not octaves — compare the DETECTED octave against

them.** When the whole capture sits whole octaves away from the ids you author (Sky's Contrabass: grid 60–84, sound C1–C3), keep the ids on the grid and author the sounded pitch of the lowest button as meta.json register ("C1" there; see docs/adding-instruments-and-games.md). Never repitch the audio to the grid and never "fix" the ids — the register belongs in config.

• Pitch is the median over many windows with parabolic interpolation. Trust it

over ad-hoc measurement: a naive single-window estimate read 30–50 cents sharp on the NightwindHorn material. Deviations within ~±10 cents: ship as-is (no repitch). A consistent larger offset: repitch to equal temperament (Catmull-Rom, see the recorder READMEs) — an out-of-tune instrument clashes with every other instrument in multi-track songs. PITCH-DEV flags > 12 c.

• The attack column is time to within 3 dB of the note's peak — it sets

sustain.minLength (0.1 default; NightwindHorn uses 0.2 because D4 swells for ~0.6 s). early RMS shows what a fast tap would actually sound like.

Then extract (add --wav when you'll need loop points — analysis must run on the pre-encode PCM, MP3 decode offsets shift per browser):

node docs/skills/instrument-from-sequential-capture/scripts/extract-notes.mjs \
  capture.mp3 --out /tmp/notes --midi 60,62,... [--wav]

Per note this trims to onset/tail (10 ms pre-roll, 3 ms fade-in, 100 ms fade-out), removes DC, downmixes, peak-normalizes to ~−3.5 dBFS and encodes 128 kbps CBR mono (m<midi>.mp3, the house recipe) — then decodes everything back and re-verifies pitch, level and onset. notes.json is a meta.json starter: review baseNote spellings (display choice, e.g. Vintage-Lyre's Db) and add icons yourself.

The encode ends by prepending a Xing/Info frame (mp3-gapless.mjs). lamejs does not write one, and without it a decoder has no way to know about the encoder's 576-sample lookahead — it hands it back as 25 ms of leading silence, which Instrument plays as latency on every attack. Nothing about the pitch or level of such a file looks wrong, so the verify pass checks the onset too: a lead over ~12 ms means the tag is missing or the trim left real silence in.

Samples encoded before this existed are repaired in place, losslessly — the audio frames are copied byte-for-byte and only the tag is added:

node docs/skills/instrument-from-sequential-capture/scripts/add-gapless-tag.mjs \
  [--dry-run] [--verify] [<file-or-dir> ...]

Choose the sustain authoring

• Tap (plucks, hits): no sustain at all.
• Loopless sustained (long natural holds that should end, e.g. wind/brass):

sustain with NO loop — a held note plays its file once, note-off starts the release fade. See genshin/instruments/NightwindHorn/ for the worked example and its README for the reasoning.

• Looping sustained (holds forever): extract with --wav, find regions

with docs/skills/audio-loop-analysis, author loop/loopMode per docs/skills/recording-sustained-instruments.

Several takes per button?

Some in-game instruments play one of several recordings per button at random (Sky's Voice of AURORA: 1–5 Takes per button, never the same one twice in a row). Split the capture per press, group presses into Takes by whole-note waveform correlation, then offer named sets of Takes as a Variant of the one instrument (settings in meta.json, docs/adding-instruments-and-games.md), never as sibling instruments: song files keep an instrument name forever (ADR-0017). The Aurora README documents the worked example.

Register the instrument

Everything an instrument touches (learned the hard way; the samples alone are maybe a third of the work):

1. src/lib/games/<game>/instruments/<Name>/ — samples, meta.json

(displayName, family, midiName must be a General MIDI name — it becomes the exported MIDI program), fill/clickColor, shape, optional sustain, register when the detected octave is off the nominal grid, notes. Add a README.md documenting source and processing.

1. Shape: reuse one from <game>/shapes.ts or add it there plus a Label Set in

games/shapes/labels.ts (every label array length must equal capacity — defineGame throws otherwise). Prefer slicing an existing set when the layout is a sub-grid, so keys/octave marks stay canonical (see STANDARD_14_LOW_LABELS / STANDARD_14_HIGH_LABELS: slice the rows the instrument's Nominal Ids occupy).

1. game.json → instruments.list (menu order).
2. i18n: key in src/lib/i18n/locales/en/index.ts instruments, and insert

the same key into every static/locales/*.json (anchor the insertion on a neighboring instrument key; translators localize later).

1. src/lib/core/BaseSettings.ts: bump BOTH counters marked

//change when instrument is added (player + zenKeyboard). They are shared across games — the other game's settings fixtures will change too; expected.

1. src/lib/core/Songs/instrumentSimilarity.ts: map it cross-game in both

directions (unmapped falls back to the target's default instrument).

1. Tests: instrument count in test/smoke.test.ts; add the name to the

POST_FREEZE sets in test/gameDefinitionConsistency.test.ts AND test/configSurface.test.ts (post-freeze instruments never join the v1 freeze). A family the game's roster didn't have yet re-routes MIDI-import suggestions for it and its General MIDI neighbours: update that game's EXPECTED_FAMILY_OUTCOMES in test/midiInstrumentSuggestion.test.ts.

1. npm run test:update-fixtures, then npm test, npm run check,

npm run build:<game> (the build exercises registry validation at prerender and copies the samples).

To see it live without clicking through menus: load /player with the instrument pre-selected by seeding localStorage before app code runs — key <StorageId>_Player_Settings, value the player object from test/fixtures/<Game>/settings-defaults.json with data.instrument.value changed (its settingVersion must match the just-bumped one).