singe/docs/lessons/29-databases.adoc

== Lesson 29: Keeping Data

image::learn/29-databases.png[The finished lesson, 480]

Lesson twelve saved a high score: one number, under one name, there the next
morning. That is exactly right for a high score, and wrong for almost
everything else a game wants to keep. A list of every round anybody has played.
The best time on each of twenty tracks. Forty levels somebody designed in a
spreadsheet. The question "what is my best on the hard stage?", which a pile of
saved numbers cannot answer without you writing the answer by hand.

That is what a database is for, and Singe has one built in. This lesson takes
the reaction test from lesson twenty-eight, gives it stages that ship with the
game, and has it keep every run anybody plays and answer questions about them.

=== What a Database Is

A database is a file of *tables*. A table is rows and columns, like a
spreadsheet with the column names decided in advance: a table of runs might
have a stage, a score, and a time for every row. You put rows in and you ask
questions to get rows out, and the questions are written in *SQL*, a small
language that has been doing this job since before most programming languages
you have heard of.

The database Singe carries is *SQLite*. It is not a server you install or a
service you sign up for. It is a library, the database is one ordinary file,
and the same SQLite is in your phone, your browser, and probably your car. You
do not have to know SQL to follow this lesson; every statement in it is
explained as it goes past, and the three you will use most -- `CREATE TABLE`,
`INSERT`, and `SELECT` -- are all here.

=== Opening One

[source,lua]
----
local sqlite3 = require("sqlite3")

local runs = sqlite3.open(singeGetDataPath() .. "runs.db")
----

`require("sqlite3")` loads a library that comes with Singe and hands it back as
a table, so everything SQLite can do is `sqlite3.` something. `sqlite3.open`
opens a database file and
creates it if it is not there yet, so the first run of your game makes an
empty one and every run after that finds it again.

The file goes in `singeGetDataPath()`, and for the same reason lesson twelve's
save does: a game's own folder may be on a disc, on a share, or packed into one
file, and none of those can be written to. Everything a game writes goes in its
data directory.

=== Making a Table

An empty database has no tables. You make one by describing it:

[source,lua]
----
runs:exec([[
    CREATE TABLE IF NOT EXISTS runs (
        id    INTEGER PRIMARY KEY,
        stage TEXT    NOT NULL,
        score INTEGER NOT NULL,
        at    INTEGER NOT NULL
    )
]])
----

`exec` runs SQL that does not hand anything back. The double square brackets
are Lua's long string: everything between them, newlines included, is one
string, which is how you keep SQL readable.

`CREATE TABLE` names the table and its columns, each with a type: `TEXT` for
words and `INTEGER` for whole numbers. `NOT NULL` means every row must have a
value there. `INTEGER PRIMARY KEY` is special: SQLite fills it in by itself,
counting up, so every row gets its own number without you thinking about it.

`IF NOT EXISTS` is what makes this safe to run every time the game starts. The
first time, it makes the table. Every time after that, the table is already
there and the line does nothing. Without those three words the second run of
your game would stop with an error saying the table already exists.

=== Putting a Row In

You could build the SQL for a new row by gluing strings together:

[source,lua]
----
runs:exec("INSERT INTO runs (stage, score, at) VALUES ('" .. stage .. "', " .. score .. ", " .. os.time() .. ")")
----

Do not. It works until a name has an apostrophe in it, at which point the quote
ends the string early and the statement is nonsense. In a game that lets
players type, it gets worse: whatever they type becomes part of your SQL.

The right way is to write the statement once, with question marks where the
values go, and hand the values over separately:

[source,lua]
----
local addRun = runs:prepare("INSERT INTO runs (stage, score, at) VALUES (?, ?, ?)")

local function record(stage, points)
    addRun:bind_values(stage, points, os.time())
    addRun:step()
    addRun:reset()
end
----

`prepare` turns the SQL into a *statement* that can be used again and again.
`bind_values` puts a value in each question mark, in order. Whatever a value
contains, apostrophes included, it is a value and never part of the SQL. `step`
runs the statement, and `reset` makes it ready to run again. `os.time()` is
plain Lua: the current time as a number of seconds, which is the easiest way to
store "when".

Preparing once and reusing the statement is also faster, because SQLite works
out how to run it only once. For a game that writes one row a minute that does
not matter. For a level editor that writes a thousand, it does.

=== Asking Questions

Rows come back out with `nrows`, which runs a `SELECT` and hands you each row as
a table with the column names as keys:

[source,lua]
----
for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC LIMIT 5") do
    debugPrint(row.stage .. "  " .. row.score)
end
----

Read the SQL left to right. `SELECT stage, score` says which columns you want.
`FROM runs` says which table. `ORDER BY score DESC` sorts the rows by score,
highest first, and `LIMIT 5` stops after five. That is a high score table, and
it never needed sorting code of yours.

The questions can be cleverer than the ones you would want to write by hand:

[source,lua]
----
for row in runs:nrows("SELECT stage, MAX(score) AS top, COUNT(*) AS tries FROM runs GROUP BY stage ORDER BY stage") do
    debugPrint(row.stage .. ":  best " .. row.top .. " of " .. row.tries)
end
----

`GROUP BY stage` gathers the rows into one group per stage, and `MAX(score)` and
`COUNT(*)` are worked out for each group: the best score on that stage and how
many times it has been played. `AS top` names the answer, which is the key you
read it back with.

When a question has one answer, `urows` hands back the columns themselves
rather than a table, which is tidier for a count:

[source,lua]
----
for count in runs:urows("SELECT COUNT(*) FROM runs") do
    played = count
end
----

=== Ask When It Changes

None of those questions belong in `onOverlayUpdate`. A query is fast, but sixty
of them a second, to get answers that change once a minute, is the game asking
the same question and getting the same answer fifty-nine times out of sixty.

So the game asks when the answer can have changed -- once at the start, and
again each time a run is recorded -- and keeps what came back in ordinary
tables:

[source,lua]
----
local function refresh()
    best = {}
    for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC, id LIMIT 5") do
        best[#best + 1] = row
    end
    -- and the same for the other two questions
end
----

`onOverlayUpdate` draws from `best`, which costs nothing. `record` calls
`refresh` after every new row, so the table on screen is never out of date.

=== Data That Ships With the Game

The other half of a game's data is the part you write before anybody plays:
the stages, the levels, the items, the lines of dialogue. That can be a
database too, made with any SQLite tool -- the `sqlite3` program on the
command line, or a program with windows such as DB Browser for SQLite -- and put
in your game's folder beside its art. The art kit has one, `stages.db`, with a
name and the shortest and longest wait for each stage.

A game opens that one differently:

[source,lua]
----
local stages  = {}
local shipped = sqlite3.open(DIR .. "art/stages.db", sqlite3.OPEN_READONLY)

for row in shipped:nrows("SELECT name, waitMin, waitMax FROM stages ORDER BY id") do
    stages[#stages + 1] = row
end
shipped:close()
----

`DIR` is your game's folder, where the art is. `sqlite3.OPEN_READONLY` says you
only mean to read it, and that matters for the day you ship. When you pack your
game into one file in the next lesson, a database opened read only is read
straight out of that file, with nothing unpacked. One opened for writing is
copied, once, to the game's data directory, and the copy is what you change from
then on; the file you shipped is never written.

So keep the two kinds apart. What you made goes in your folder and is opened
read only. What the game makes goes in the data directory. The stages here are
read once at the start into an ordinary table and the database is closed,
because a list of four stages is not worth keeping a file open for.

=== Closing Up

Statements and databases hold memory and a file open until you say you are
finished, and the tidy place to say it is `onShutdown`:

[source,lua]
----
function onShutdown()
    addRun:finalize()
    runs:close()
end
----

`finalize` is `close` for a statement. A database will not close while a
statement made from it is still around, so the statements go first.

=== The Whole Script

[source,lua]
----
dofile("Singe/Framework.singe")

local sqlite3 = require("sqlite3")

local PERFECT = 1000

-- What ships with the game:  the stages, read once and closed.
local stages  = {}
local shipped = sqlite3.open(DIR .. "art/stages.db", sqlite3.OPEN_READONLY)

for row in shipped:nrows("SELECT name, waitMin, waitMax FROM stages ORDER BY id") do
    stages[#stages + 1] = row
end
shipped:close()

-- What the game writes:  every run anybody plays, kept in the data directory.
local runs = sqlite3.open(singeGetDataPath() .. "runs.db")

runs:exec([[
    CREATE TABLE IF NOT EXISTS runs (
        id    INTEGER PRIMARY KEY,
        stage TEXT    NOT NULL,
        score INTEGER NOT NULL,
        at    INTEGER NOT NULL
    )
]])

local addRun   = runs:prepare("INSERT INTO runs (stage, score, at) VALUES (?, ?, ?)")
local stageAt  = 1
local state    = "attract"
local goAt     = 0
local score    = 0
local best     = {}
local perStage = {}
local played   = 0


-- The questions, asked when a run is added rather than every frame.
local function refresh()
    best = {}
    for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC, id LIMIT 5") do
        best[#best + 1] = row
    end
    perStage = {}
    for row in runs:nrows("SELECT stage, MAX(score) AS top, COUNT(*) AS tries FROM runs GROUP BY stage ORDER BY stage") do
        perStage[#perStage + 1] = row
    end
    for count in runs:urows("SELECT COUNT(*) FROM runs") do
        played = count
    end
end


local function record(stage, points)
    addRun:bind_values(stage, points, os.time())
    addRun:step()
    addRun:reset()
    refresh()
end


local function play()
    local stage = stages[stageAt]

    state = "waiting"
    goAt  = singeGetTicks() + math.random(stage.waitMin, stage.waitMax)
    score = 0
end


function onInputPressed(what)
    if what == SWITCH_BUTTON2 and state ~= "waiting" and state ~= "go" then
        stageAt = stageAt % #stages + 1
        return
    end
    if what ~= SWITCH_BUTTON1 then
        return
    end
    if state == "waiting" then
        state = "result"
        score = 0
        record(stages[stageAt].name, score)
    elseif state == "go" then
        score = math.max(0, PERFECT - (singeGetTicks() - goAt))
        state = "result"
        record(stages[stageAt].name, score)
    else
        play()
    end
end


function onOverlayUpdate()
    if state == "waiting" and singeGetTicks() >= goAt then
        state = "go"
    end

    overlayClear()
    overlayPrint(2, 0, "REACTION TEST:  " .. stages[stageAt].name)

    if state == "attract" then
        overlayPrint(2, 2, "Button 1 to play, button 2 for another stage.")
    elseif state == "waiting" then
        overlayPrint(2, 2, "Wait for it...")
    elseif state == "go" then
        overlayPrint(2, 2, "NOW!")
    else
        overlayPrint(2, 2, "You scored " .. score .. ".  Button 1 plays again.")
    end

    overlayPrint(2, 4, "Runs kept:  " .. played)
    overlayPrint(2, 6, "BEST RUNS")
    for place, row in ipairs(best) do
        overlayPrint(2, 6 + place, string.format("%d.  %-8s %4d", place, row.stage, row.score))
    end
    overlayPrint(24, 6, "BY STAGE")
    for line, row in ipairs(perStage) do
        overlayPrint(24, 6 + line, string.format("%-8s best %4d of %d", row.stage, row.top, row.tries))
    end

    return OVERLAY_UPDATED
end


function onShutdown()
    addRun:finalize()
    runs:close()
end


refresh()
----

Run it. Button 1 starts a round and stops it, as in lesson twenty-eight, and
button 2 between rounds moves to the next stage. Play a few on each. Then quit
and run it again: every run is still there, the best five at the left and each
stage's best at the right.

=== What Just Happened

[source,lua]
----
local runs = sqlite3.open(singeGetDataPath() .. "runs.db")
----

Two files are in play, and they are not alike. `art/stages.db` is yours: you
wrote it, it ships with the game, and the game only reads it. `runs.db` is the
game's: it did not exist until the first run made it, it lives in the data
directory, and nobody ships it.

[source,lua]
----
    if state == "waiting" then
        state = "result"
        score = 0
        record(stages[stageAt].name, score)
----

A button pressed too soon still records a run, with a score of nothing. The
database is a record of what happened, and a jumped start happened. Whether to
show those rows is a question for the `SELECT`, not something to hide by never
writing them.

[source,lua]
----
    for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC, id LIMIT 5") do
----

`ORDER BY score DESC, id` sorts by score, and where two scores are equal, by
`id` -- the order they were played. Without the second key, two equal scores
come back in whatever order SQLite finds convenient, which can change between
runs. A table that reshuffles itself for no visible reason looks broken even
when it is not.

=== Try It

. *Show the date.* `os.date("%Y-%m-%d", row.at)` turns a stored time back into
  something a person can read. Add the date to each of the best runs.
. *Ignore the jumped starts.* Add `WHERE score > 0` to the best-runs question,
  between `FROM runs` and `ORDER BY`. Then count the jumped starts on each stage
  with a question of your own.
. *Add a stage.* Open `stages.db` in a SQLite tool, add a row, and run the game.
  Nothing in the script changes. That is the point of keeping data out of the
  script.
. *Reset the history.* `runs:exec("DELETE FROM runs")` empties the table. Put it
  on a button, call `refresh` after it, and watch everything go to nothing.
  This is the "clear high scores" in a service menu.
. *Write a thousand rows.* Loop `record` a thousand times and time it with
  `singeGetTicks`. Then put `runs:exec("BEGIN")` before the loop and
  `runs:exec("COMMIT")` after it, and time it again. A *transaction* makes many
  writes one, and the difference is large enough to be worth remembering.

=== Break It on Purpose

Take `addRun:reset()` out of `record` and play three rounds on three different
stages.

No error. The game plays perfectly. But the table at the right says you played
the first stage three times, and the best runs are three copies of your first
score. `bind_values` could not put new values into a statement that had already
run and not been reset, so it gave up -- quietly, returning a number you were
not checking -- and `step` ran the statement again with the values it already
had.

This is the bug that matters in this lesson, because it does not look like one.
A query that fails is loud. A write that writes the wrong thing is silent, and
you find it a week later in the data. When the rows in a database are not what
you expect, look at the code that put them there before the code that reads
them.

Put `reset` back, and break it the loud way: change `score` to `scroe` in the
first `SELECT` in `refresh`.

----
Error running script:  no such column: scroe
stack traceback:
	[C]: in method 'nrows'
	29-databases.singe:43: in local 'refresh'
	29-databases.singe:133: in main chunk
----

SQLite reads the column names when it prepares the question, so a misspelled
one stops the game at once, and the message names it exactly. The traceback
reads from the inside out: `nrows` complained, `refresh` called it at line 43,
and the script itself called `refresh` at line 133, the last line, as the game
started. That is why it says *running script* rather than naming a function
the engine called: it never got as far as the first frame.

=== What You Learned

* A database is a file of tables; a table is rows with named columns.
* `require("sqlite3")` gives you SQLite. `sqlite3.open` opens a file and creates
  it if it is missing.
* What the game writes goes in `singeGetDataPath()`. What you ship goes in your
  folder, opened with `sqlite3.OPEN_READONLY`.
* `CREATE TABLE IF NOT EXISTS` is safe to run every time the game starts.
* Values go in through `prepare`, `bind_values`, `step`, and `reset`, never by
  gluing strings into SQL.
* `nrows` hands back rows as tables; `urows` hands back the columns themselves.
* `ORDER BY`, `LIMIT`, `GROUP BY`, `MAX`, and `COUNT` answer the questions you
  would otherwise write loops for.
* Ask when the answer can change, not every frame.
* Statements are finalized, then the database is closed, in `onShutdown`.

=== Next Time

Your game now has a folder full of things: a script, art, sounds, and a
database or two. Lesson thirty turns that folder into one file a stranger can
copy onto their own machine and play -- and your read-only database goes with
it, still opening exactly as it does now.