singe/docs/lessons/27-midi.adoc

== Lesson 27: Music and MIDI

image::learn/27-midi.png[The finished lesson, 480]

This lesson is two lessons that happen to live together.

The first half is music: how a game plays a piece of music properly, why that
is not the same thing as playing a sound effect, and what to do when the music
has to stay in step with video.

The second half is MIDI, which most people have never met and which gives a
game something nothing else does. By the end of it, a keyboard plugged into
your computer will be playing your game.

=== A Sound Effect and a Piece of Music Are Different Things

You already know how to play a sound. Lesson ten loaded a `.wav` with
`soundLoad` and fired it with `soundPlay`, and if you point those two calls at
a four minute song they will play it.

Do not. Singe keeps sound effects and music apart, and the separation is
worth having for four reasons.

* *They have their own volumes.* `soundSetVolume` never touches the music and
  `musicSetVolume` never touches the gunshots. Every game that has ever
  shipped needed that, because players turn the music down and the effects up.
* *Music has no channel limit.* Sound effects share sixteen channels, and when
  all sixteen are busy the seventeenth does not play. A music track is not on
  a channel, so nothing can crowd it out.
* *One handle is one track*, and it pauses, resumes, stops, and fades on its
  own without disturbing anything else.
* *Music pauses with the game.* Hit the pause key and it holds where it is.
  Come back and it picks up. You write nothing to make that happen.

One thing that is *not* different is worth knowing, because people assume the
opposite: Singe decodes a music file into memory up front, exactly as it does
a sound effect. A long track costs memory. It is not streamed off the disk a
piece at a time. Five minutes of music is a few tens of megabytes of memory
while it is loaded, so load the track for the level you are on and
`musicUnload` it when you leave.

=== Playing a Track

Every reader of this book already has a piece of music, because the engine
unpacked one into your work folder the first time you ran anything. Start a file
with these three lines:

[source,lua]
----
local theme = musicLoad("Singe/menuIntro.flac")
musicSetVolume(64)
musicPlay(theme, -1)


function onOverlayUpdate()
    overlayClear()
    overlayPrint(2, 2, "Music: " .. (musicIsPlaying(theme) and "playing" or "stopped"))
    return OVERLAY_UPDATED
end
----

Run it with `Singe -R music`. Ten seconds of music, and then it starts
again, and it keeps starting again.

That is what the `-1` does. `musicPlay` takes a number of *loops*: leave it out
or pass `0` and the track plays once, pass `3` and it plays four times in all,
and pass `-1` and it repeats until something stops it. A menu, a title screen,
or a level's background is nearly always `-1`.

Two details in that little script are worth the ink. `musicLoad` takes a path
and not a `DIR ..` path, because `Singe/menuIntro.flac` is one of the engine's
own files and sits in the folder the engine unpacked into your work folder.
`musicIsPlaying` answers `true` or `false`, and the `and ... or ...` in the
middle of that `overlayPrint` is the Lua shorthand from lesson three for
choosing one of two values.

=== The Volume Scale Is Not the One You Know

`musicSetVolume(64)` sets it to about half. Sound effects run from `0` to
`63`; music runs from `0` to `128`, which is twice as far.

That is not a mistake and it is not a joke. Singe can run games written for
Hypseus, an engine whose music volume has always been a `0` to `128` scale,
and changing the numbers would silently make those games' music twice as loud
as their authors meant. So Singe keeps both scales, each attached to the
family of calls it belongs to. Effects: `0` to `63`. Music: `0` to `128`. Give
`musicSetVolume` a number outside its range and the script stops, which you
will see for yourself at the end of this lesson.

=== What Singe Will Play

Nearly anything. WAV, Ogg Vorbis, Opus, MP3, FLAC, WavPack, AAC, and a long
list past that, all through the same `musicLoad`.

Two families are more interesting than the ordinary ones, and both exist
because they are tiny.

*Tracker modules* -- `.mod`, `.s3m`, `.xm`, `.it`, and fifty-odd relatives --
hold a handful of short recorded instruments and a list of which notes to play
them at. A four minute tune can be under a hundred kilobytes.

*Chiptunes* -- `.nsf`, `.spc`, `.vgm`, `.gbs`, and their kin -- go further:
they hold the *register writes* for the actual sound chip of a Nintendo, a
Mega Drive, or a Spectrum, and Singe emulates the chip to play them. If you
want a game to sound like 1989 rather than to sound like a recording of 1989,
this is how.

Either loads with `musicLoad` like anything else. The manual's *Video, Audio,
and Container Formats* section has the full list, including which container
goes with which codec.

=== Stopping, Pausing, and Fading

[source,lua]
----
musicPause(theme)
musicResume(theme)
musicStop(theme)
musicStop(theme, 2000)
----

`musicPause` holds the track where it is and `musicResume` carries on from
that spot. `musicStop` ends it, and the track goes back to its beginning the
next time you play it. With a second number, `musicStop` fades the track out
over that many milliseconds instead of cutting it dead -- two seconds here.
That fade is the single easiest thing you can do to make a game feel finished.
Music that stops mid-bar when the player walks through a door sounds like a
bug, even to someone who could not say why.

Call any of the four with no handle at all and they act on every piece of
music at once, which is what a pause screen wants.

Now the honest limit, because you will look for this and it is not there:
*Singe cannot crossfade two tracks.* `musicSetVolume` is one gain over all
music, not a volume per track, so there is no way to bring one track up while
another goes down. What you can do is fade the old one out and start the new
one as it goes, which is close enough for a level change and audibly not the
same thing for a battle theme sliding in under an exploration theme. If you
need a true crossfade today, the way to get it is to make the two pieces one
file and seek within it, or to accept the overlap.

=== Music, Video, and Staying in Step

If your game plays video -- part three's subject -- there is one thing to know
about sound, and it is not a music call at all.

Singe shows a video frame when the sound that belongs with it is heard, not on
a timer. The audio is the clock and the picture follows it. That is the right
way round, because an ear notices a fifth of a second of drift in a voice and
an eye does not notice the same drift in a mouth.

The engine measures its own audio path when it starts. What it cannot measure
is everything after the operating system hands the sound away: an amplifier, a
television, an HDMI receiver, a Bluetooth headset. Each of those adds a delay
of its own, it is different on every machine, and it shows up as sound arriving
slightly after the thing that made it.

The fix is one number:

[source,lua]
----
singeSetAudioDelay(45)
----

Forty-five milliseconds of "the audio is late; hold the picture back to
match". You can also pass it as `--audiodelay=45` on the command line or put
it in the game's `games.dat` entry. `singeGetAudioDelay` reads back whatever
is in force, which is what a service menu shows next to its slider.

You do not have to guess the number. Singe's own menu has a calibration
screen: it plays a click once a second and flashes the screen white, you
adjust until the flash and the click happen together, and it saves the result
for every game on that machine. The manual's *Audio Sync* section describes it
and explains why it works. If you build a cabinet, run it once after you wire
the speakers, and never think about it again.

=== The Other Thing Called MIDI

MIDI is two unrelated things with one name, and keeping them apart is most of
understanding it.

Here is the first, in one paragraph. A recording -- a `.wav`, an `.mp3` -- is a
measurement of a sound, taken forty-four thousand times a second. MIDI is not
that. A MIDI file holds *what was played*: at this moment, on this channel,
press key sixty this hard; a little later, let go of it. It is closer to sheet
music, or to a player piano's punched roll, than to a recording. That is why a
MIDI file is a few kilobytes when the same music as audio is a few megabytes,
and it is also why a MIDI file does not contain any sound at all. Something
has to play the notes.

A MIDI file is music, so it plays like music:

[source,lua]
----
local tune = musicLoad("music/overworld.mid")
musicPlay(tune, -1)
----

But the thing that plays the notes has to come from somewhere, and that thing
is a *soundfont*: a file full of recorded instruments -- a piano, a trumpet, a
snare drum -- with the notes and the loop points marked, so the engine can
build any note of any instrument out of them.

Singe does not ship one. A good soundfont is tens of megabytes, and it would
be tens of megabytes carried by every copy of the engine for a format most
games never touch. So you name one with `--soundfont`, or you put one at
`Singe/soundfont.sf2` so that a packed game carries its own, or you let Singe
find the one your Linux distribution installed. With no soundfont anywhere,
loading a `.mid` fails and says so.

This is why the same MIDI file sounds different on two machines and why it
sounded so bad on the computers of the nineteen nineties. The file is a list
of notes. What you hear is whichever soundfont happened to play them: a good
one is a room full of instruments, a cheap one is a room full of toys, and the
notes are identical in both. If a MIDI file is going in your game, ship the
soundfont with it. Otherwise you have not chosen how your music sounds; you
have left it to whatever is installed on the player's machine.

=== MIDI as a Wire

Now the second thing, which is the interesting one.

MIDI is also a cable, and messages that travel along it. Plug a musical
keyboard into your computer and every key you press sends a message: *note on,
channel 1, key 60, velocity 90*. Turn a knob on a control surface and it sends
*controller 7 on channel 1 is now 40*. These messages exist whether or not
anything is playing any music. They are another input device, like a
joystick with eighty-eight buttons that know how hard you hit them.

And it goes the other way. Your game can send those messages out, to a
synthesiser, a sound module, a stage light controller, or anything else on the
wire.

Ports are numbered from zero, and Singe opens nothing until you ask -- a game
that never mentions MIDI never pays for MIDI. One input port and one output
port may be open at a time. Take stock of the machine like this:

[source,lua]
----
local inPort  = "none"
local outPort = "none"

if midiInputCount() > 0 and midiOpenInput(0) then
    inPort = midiInputName(0)
end
if midiOutputCount() > 0 and midiOpenOutput(0) then
    outPort = midiOutputName(0)
end
----

On a machine with no MIDI at all the counts are zero, the open calls answer
`false`, and nothing fails. That matters: it means you can write this code
into a real game and ship it, and the ninety-nine players in a hundred with no
MIDI keyboard will never know it is there. The names are for a settings screen
that lets a player choose which port, because port zero is rarely the one they
want. `midiRescan` looks again for something plugged in while the game is
running.

=== Listening

Every message that arrives goes to a callback, in the same way that every key
press goes to `onInputPressed`:

[source,lua]
----
local MIDI_NOTE_ON = 0x90


function onMidiMessage(status, data1, data2, bytes)
    if (status & 0xF0) ~= MIDI_NOTE_ON or data2 == 0 then
        return
    end
    heard = noteName(data1)
end
----

Most MIDI messages are three bytes long, and the three numbers you are handed
are those bytes exactly as the device sent them. A message with fewer bytes
hands you a `0` in place of each one it does not have.

`status` says what kind of message it is and which channel it came from, mixed
together: the kind is the top four bits and the channel is the bottom four.
`0x90` is note on, and a note on channel 1 arrives as `0x90` while the same
note on channel 5 arrives as `0x94`. `status & 0xF0` keeps the top half and
throws the channel away, which is what you want unless you care which channel
it was. (`&` is Lua's bitwise and, and `0x90` is a number written in
hexadecimal -- base sixteen -- because that is how every MIDI document in the
world writes them.)

`data1` and `data2` are the other two bytes, and what they mean depends on the
kind. For a note on they are the key and the velocity: key `60` is middle C,
and velocity is how hard it was struck, `1` to `127`.

The check for `data2 == 0` is not fussiness. A note on with a velocity of zero
means *note off*: it is how nearly every keyboard ever made says "I let go",
and if you do not filter it your game will fire twice for every key.

`bytes` is the whole message as a string, which is how you read a message
longer than three bytes.

=== Playing

Sending is easier than receiving.

[source,lua]
----
midiNoteOn(1, 60, 100)
midiNoteOff(1, 60)
----

Middle C on channel 1, struck at a hundred, and then released. Channels here
are `1` to `16` as a person counts them, not `0` to `15` as the wire does,
which is Singe being kind to you.

Every `midiNoteOn` needs a `midiNoteOff`. A note that is never released sounds
until something stops it, and the classic first MIDI bug is a game that
crashes or quits with three notes held down and leaves a chord hanging in the
room. So pair them, and pair them through a timer rather than by hand:

[source,lua]
----
function playTarget()
    if not midiIsOutputOpen() then
        return
    end
    local key = target
    midiNoteOn(1, key, 100)
    timerAfter(HOLD_MS, function()
        midiNoteOff(1, key)
    end)
end
----

`local key = target` is doing real work. `timerAfter` runs its function four
tenths of a second later, and by then `target` may have changed to a different
note; copying it into a local means the timer releases the note that was
actually pressed. Copy anything a timer will need later.

The other sending calls do the rest of what MIDI is: `midiProgramChange`
chooses which instrument a channel plays, `midiControlChange` moves a
controller (volume is 7, pan is 10, the sustain pedal is 64),
`midiPitchBend` bends a channel, and `midiSend` sends any message at all as
raw bytes for the things the named calls do not cover.

=== A Reason to Care

All of which is machinery. Here is what it is for.

Put the two halves together and the game asks for a note, the player plays it
on a real instrument, and the game knows whether they got it right:

[source,lua]
----
function onMidiMessage(status, data1, data2, bytes)
    if (status & 0xF0) ~= MIDI_NOTE_ON or data2 == 0 then
        return
    end
    heard = noteName(data1)
    if data1 == target then
        score = score + 1
        pickTarget()
        playTarget()
    end
end
----

Twelve lines, and you have the skeleton of a music teaching game. Add a clock
and you have a rhythm game that scores real playing rather than four plastic
buttons. Add `velocity` to the test and it cares how hard you hit the note.
Compare the time the note arrived against the time it was due and you are
grading timing to the millisecond.

And because MIDI goes out as well as in, a cabinet can do things nothing else
can. A real instrument wired to the machine. An external sound module playing
the score so the tune has depth no sample bank gives it. Stage lights on the
cabinet that flash on the beat because the game sent them the beat. If you are
the sort of person who builds arcade cabinets, this is the call that lets the
cabinet be a thing rather than a box round a computer.

The finished script is `learn/27-midi.singe`: the music controls, the port
list, and the guessing game, in about a hundred lines. The space bar sends the
note out, Left Alt pauses the music, Left Shift fades it, and the up and down
arrows move the volume. Run it with no MIDI hardware and the music half works
and the MIDI half says `none` twice, which is how it should behave on a
player's machine.

=== Cleaning Up

[source,lua]
----
function onShutdown()
    midiCloseInput()
    midiCloseOutput()
    musicUnload(theme)
end
----

`onShutdown` is called once when the game ends, and it is where a script gives
things back. Closing a port that is not open does nothing, so there is nothing
to check first.

=== Try It

. *Make it loop four times instead of forever.* Change `musicPlay(theme, -1)`
  to `musicPlay(theme, 3)` and count. The number is how many times it
  *repeats*, not how many times it plays.
. *Fade instead of stopping.* Change the button that stops the music to
  `musicStop(theme, 4000)` and listen to the difference four seconds makes.
  Then try `musicStop(theme, 100)`.
. *Show the channel.* In `onMidiMessage`, work out the channel with
  `(status & 0x0F) + 1` and print it. Play your keyboard and see which channel
  it is on -- and then find its settings and change that channel, to prove to
  yourself that the number is real.
. *Score the velocity.* Give a point for the right note, and three points if
  `data2` is over a hundred, so the game rewards playing it like you mean it.
. *Play the answer.* When the player gets a note right, make the game play a
  small chord back at them with three `midiNoteOn` calls a couple of notes
  apart and three timers to release them. You will need a real output port or
  a software synthesiser for this, and on Linux that is any program that
  registers itself with ALSA.

=== Break It on Purpose

The two volume scales are easy to mix up, and Singe would rather stop than
play your music at a level you did not mean. Add one line under
`musicPlay(theme, -1)`, using the number that would be full volume for a sound
effect if you doubled it in your head and got it wrong:

[source,lua]
----
musicSetVolume(200)
----

Singe stops before the window opens, naming the line that call is on in your
file:

----
107:musicSetVolume:  Invalid music volume value:  200
----

The line, the call that objected, and the complaint, which here quotes the
number back at you so there is no argument about what arrived. Music is `0` to
`128`. Sound effects are `0` to `63`. When a volume call stops your game, the
question to ask is which of the two families you are talking to.

=== What You Learned

* Music and sound effects are separate in Singe: their own volumes, no channel
  limit for music, and a pause that follows the game's pause.
* A music file is decoded into memory up front, so a long track costs memory
  and should be unloaded when you leave the level.
* `musicPlay(track, -1)` loops forever; a positive number is how many times it
  repeats.
* Music volume is `0` to `128` and effects volume is `0` to `63`, and the
  difference is Hypseus compatibility, not a mistake.
* `musicStop(track, milliseconds)` fades out, and a fade is the cheapest
  polish there is. There is no crossfade, because one gain covers all music.
* Singe presents video against the audio clock, and `singeSetAudioDelay`
  corrects for the delay of whatever the sound comes out of.
* A MIDI file holds notes, not sound, and needs a soundfont to be heard at
  all. Singe ships none, so ship your own or your music is whatever the
  player's machine happens to have.
* MIDI ports are a separate thing entirely: live messages in and out, on a
  wire, with nothing to do with MIDI files.
* `onMidiMessage` hands you the raw bytes. The top four bits of `status` are
  the kind, the bottom four are the channel, and a note on with velocity zero
  means note off.
* Every `midiNoteOn` needs a `midiNoteOff`, and a timer is how you pair them.
* On a machine with no MIDI, every call answers `false` or zero and nothing
  fails, so MIDI support is safe to ship.

=== Next Time

Your game has music, and a score that a MIDI keyboard can run up. Lesson
twenty-eight takes that score off the machine: accounts, high score tables
that other people can see, and the catalogue that lists your game to the
world.