== Lesson 29: Keeping Data image::learn/29-databases.png[The finished lesson, 480] Lesson twelve saved a high score: one number, under one name, there the next morning. That is exactly right for a high score, and wrong for almost everything else a game wants to keep. A list of every round anybody has played. The best time on each of twenty tracks. Forty levels somebody designed in a spreadsheet. The question "what is my best on the hard stage?", which a pile of saved numbers cannot answer without you writing the answer by hand. That is what a database is for, and Singe has one built in. This lesson takes the reaction test from lesson twenty-eight, gives it stages that ship with the game, and has it keep every run anybody plays and answer questions about them. === What a Database Is A database is a file of *tables*. A table is rows and columns, like a spreadsheet with the column names decided in advance: a table of runs might have a stage, a score, and a time for every row. You put rows in and you ask questions to get rows out, and the questions are written in *SQL*, a small language that has been doing this job since before most programming languages you have heard of. The database Singe carries is *SQLite*. It is not a server you install or a service you sign up for. It is a library, the database is one ordinary file, and the same SQLite is in your phone, your browser, and probably your car. You do not have to know SQL to follow this lesson; every statement in it is explained as it goes past, and the three you will use most -- `CREATE TABLE`, `INSERT`, and `SELECT` -- are all here. === Opening One [source,lua] ---- local sqlite3 = require("sqlite3") local runs = sqlite3.open(singeGetDataPath() .. "runs.db") ---- `require("sqlite3")` loads a library that comes with Singe and hands it back as a table, so everything SQLite can do is `sqlite3.` something. `sqlite3.open` opens a database file and creates it if it is not there yet, so the first run of your game makes an empty one and every run after that finds it again. The file goes in `singeGetDataPath()`, and for the same reason lesson twelve's save does: a game's own folder may be on a disc, on a share, or packed into one file, and none of those can be written to. Everything a game writes goes in its data directory. === Making a Table An empty database has no tables. You make one by describing it: [source,lua] ---- runs:exec([[ CREATE TABLE IF NOT EXISTS runs ( id INTEGER PRIMARY KEY, stage TEXT NOT NULL, score INTEGER NOT NULL, at INTEGER NOT NULL ) ]]) ---- `exec` runs SQL that does not hand anything back. The double square brackets are Lua's long string: everything between them, newlines included, is one string, which is how you keep SQL readable. `CREATE TABLE` names the table and its columns, each with a type: `TEXT` for words and `INTEGER` for whole numbers. `NOT NULL` means every row must have a value there. `INTEGER PRIMARY KEY` is special: SQLite fills it in by itself, counting up, so every row gets its own number without you thinking about it. `IF NOT EXISTS` is what makes this safe to run every time the game starts. The first time, it makes the table. Every time after that, the table is already there and the line does nothing. Without those three words the second run of your game would stop with an error saying the table already exists. === Putting a Row In You could build the SQL for a new row by gluing strings together: [source,lua] ---- runs:exec("INSERT INTO runs (stage, score, at) VALUES ('" .. stage .. "', " .. score .. ", " .. os.time() .. ")") ---- Do not. It works until a name has an apostrophe in it, at which point the quote ends the string early and the statement is nonsense. In a game that lets players type, it gets worse: whatever they type becomes part of your SQL. The right way is to write the statement once, with question marks where the values go, and hand the values over separately: [source,lua] ---- local addRun = runs:prepare("INSERT INTO runs (stage, score, at) VALUES (?, ?, ?)") local function record(stage, points) addRun:bind_values(stage, points, os.time()) addRun:step() addRun:reset() end ---- `prepare` turns the SQL into a *statement* that can be used again and again. `bind_values` puts a value in each question mark, in order. Whatever a value contains, apostrophes included, it is a value and never part of the SQL. `step` runs the statement, and `reset` makes it ready to run again. `os.time()` is plain Lua: the current time as a number of seconds, which is the easiest way to store "when". Preparing once and reusing the statement is also faster, because SQLite works out how to run it only once. For a game that writes one row a minute that does not matter. For a level editor that writes a thousand, it does. === Asking Questions Rows come back out with `nrows`, which runs a `SELECT` and hands you each row as a table with the column names as keys: [source,lua] ---- for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC LIMIT 5") do debugPrint(row.stage .. " " .. row.score) end ---- Read the SQL left to right. `SELECT stage, score` says which columns you want. `FROM runs` says which table. `ORDER BY score DESC` sorts the rows by score, highest first, and `LIMIT 5` stops after five. That is a high score table, and it never needed sorting code of yours. The questions can be cleverer than the ones you would want to write by hand: [source,lua] ---- for row in runs:nrows("SELECT stage, MAX(score) AS top, COUNT(*) AS tries FROM runs GROUP BY stage ORDER BY stage") do debugPrint(row.stage .. ": best " .. row.top .. " of " .. row.tries) end ---- `GROUP BY stage` gathers the rows into one group per stage, and `MAX(score)` and `COUNT(*)` are worked out for each group: the best score on that stage and how many times it has been played. `AS top` names the answer, which is the key you read it back with. When a question has one answer, `urows` hands back the columns themselves rather than a table, which is tidier for a count: [source,lua] ---- for count in runs:urows("SELECT COUNT(*) FROM runs") do played = count end ---- === Ask When It Changes None of those questions belong in `onOverlayUpdate`. A query is fast, but sixty of them a second, to get answers that change once a minute, is the game asking the same question and getting the same answer fifty-nine times out of sixty. So the game asks when the answer can have changed -- once at the start, and again each time a run is recorded -- and keeps what came back in ordinary tables: [source,lua] ---- local function refresh() best = {} for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC, id LIMIT 5") do best[#best + 1] = row end -- and the same for the other two questions end ---- `onOverlayUpdate` draws from `best`, which costs nothing. `record` calls `refresh` after every new row, so the table on screen is never out of date. === Data That Ships With the Game The other half of a game's data is the part you write before anybody plays: the stages, the levels, the items, the lines of dialogue. That can be a database too, made with any SQLite tool -- the `sqlite3` program on the command line, or a program with windows such as DB Browser for SQLite -- and put in your game's folder beside its art. The art kit has one, `stages.db`, with a name and the shortest and longest wait for each stage. A game opens that one differently: [source,lua] ---- local stages = {} local shipped = sqlite3.open(DIR .. "art/stages.db", sqlite3.OPEN_READONLY) for row in shipped:nrows("SELECT name, waitMin, waitMax FROM stages ORDER BY id") do stages[#stages + 1] = row end shipped:close() ---- `DIR` is your game's folder, where the art is. `sqlite3.OPEN_READONLY` says you only mean to read it, and that matters for the day you ship. When you pack your game into one file in the next lesson, a database opened read only is read straight out of that file, with nothing unpacked. One opened for writing is copied, once, to the game's data directory, and the copy is what you change from then on; the file you shipped is never written. So keep the two kinds apart. What you made goes in your folder and is opened read only. What the game makes goes in the data directory. The stages here are read once at the start into an ordinary table and the database is closed, because a list of four stages is not worth keeping a file open for. === Closing Up Statements and databases hold memory and a file open until you say you are finished, and the tidy place to say it is `onShutdown`: [source,lua] ---- function onShutdown() addRun:finalize() runs:close() end ---- `finalize` is `close` for a statement. A database will not close while a statement made from it is still around, so the statements go first. === The Whole Script [source,lua] ---- dofile("Singe/Framework.singe") local sqlite3 = require("sqlite3") local PERFECT = 1000 -- What ships with the game: the stages, read once and closed. local stages = {} local shipped = sqlite3.open(DIR .. "art/stages.db", sqlite3.OPEN_READONLY) for row in shipped:nrows("SELECT name, waitMin, waitMax FROM stages ORDER BY id") do stages[#stages + 1] = row end shipped:close() -- What the game writes: every run anybody plays, kept in the data directory. local runs = sqlite3.open(singeGetDataPath() .. "runs.db") runs:exec([[ CREATE TABLE IF NOT EXISTS runs ( id INTEGER PRIMARY KEY, stage TEXT NOT NULL, score INTEGER NOT NULL, at INTEGER NOT NULL ) ]]) local addRun = runs:prepare("INSERT INTO runs (stage, score, at) VALUES (?, ?, ?)") local stageAt = 1 local state = "attract" local goAt = 0 local score = 0 local best = {} local perStage = {} local played = 0 -- The questions, asked when a run is added rather than every frame. local function refresh() best = {} for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC, id LIMIT 5") do best[#best + 1] = row end perStage = {} for row in runs:nrows("SELECT stage, MAX(score) AS top, COUNT(*) AS tries FROM runs GROUP BY stage ORDER BY stage") do perStage[#perStage + 1] = row end for count in runs:urows("SELECT COUNT(*) FROM runs") do played = count end end local function record(stage, points) addRun:bind_values(stage, points, os.time()) addRun:step() addRun:reset() refresh() end local function play() local stage = stages[stageAt] state = "waiting" goAt = singeGetTicks() + math.random(stage.waitMin, stage.waitMax) score = 0 end function onInputPressed(what) if what == SWITCH_BUTTON2 and state ~= "waiting" and state ~= "go" then stageAt = stageAt % #stages + 1 return end if what ~= SWITCH_BUTTON1 then return end if state == "waiting" then state = "result" score = 0 record(stages[stageAt].name, score) elseif state == "go" then score = math.max(0, PERFECT - (singeGetTicks() - goAt)) state = "result" record(stages[stageAt].name, score) else play() end end function onOverlayUpdate() if state == "waiting" and singeGetTicks() >= goAt then state = "go" end overlayClear() overlayPrint(2, 0, "REACTION TEST: " .. stages[stageAt].name) if state == "attract" then overlayPrint(2, 2, "Button 1 to play, button 2 for another stage.") elseif state == "waiting" then overlayPrint(2, 2, "Wait for it...") elseif state == "go" then overlayPrint(2, 2, "NOW!") else overlayPrint(2, 2, "You scored " .. score .. ". Button 1 plays again.") end overlayPrint(2, 4, "Runs kept: " .. played) overlayPrint(2, 6, "BEST RUNS") for place, row in ipairs(best) do overlayPrint(2, 6 + place, string.format("%d. %-8s %4d", place, row.stage, row.score)) end overlayPrint(24, 6, "BY STAGE") for line, row in ipairs(perStage) do overlayPrint(24, 6 + line, string.format("%-8s best %4d of %d", row.stage, row.top, row.tries)) end return OVERLAY_UPDATED end function onShutdown() addRun:finalize() runs:close() end refresh() ---- Run it. Button 1 starts a round and stops it, as in lesson twenty-eight, and button 2 between rounds moves to the next stage. Play a few on each. Then quit and run it again: every run is still there, the best five at the left and each stage's best at the right. === What Just Happened [source,lua] ---- local runs = sqlite3.open(singeGetDataPath() .. "runs.db") ---- Two files are in play, and they are not alike. `art/stages.db` is yours: you wrote it, it ships with the game, and the game only reads it. `runs.db` is the game's: it did not exist until the first run made it, it lives in the data directory, and nobody ships it. [source,lua] ---- if state == "waiting" then state = "result" score = 0 record(stages[stageAt].name, score) ---- A button pressed too soon still records a run, with a score of nothing. The database is a record of what happened, and a jumped start happened. Whether to show those rows is a question for the `SELECT`, not something to hide by never writing them. [source,lua] ---- for row in runs:nrows("SELECT stage, score FROM runs ORDER BY score DESC, id LIMIT 5") do ---- `ORDER BY score DESC, id` sorts by score, and where two scores are equal, by `id` -- the order they were played. Without the second key, two equal scores come back in whatever order SQLite finds convenient, which can change between runs. A table that reshuffles itself for no visible reason looks broken even when it is not. === Try It . *Show the date.* `os.date("%Y-%m-%d", row.at)` turns a stored time back into something a person can read. Add the date to each of the best runs. . *Ignore the jumped starts.* Add `WHERE score > 0` to the best-runs question, between `FROM runs` and `ORDER BY`. Then count the jumped starts on each stage with a question of your own. . *Add a stage.* Open `stages.db` in a SQLite tool, add a row, and run the game. Nothing in the script changes. That is the point of keeping data out of the script. . *Reset the history.* `runs:exec("DELETE FROM runs")` empties the table. Put it on a button, call `refresh` after it, and watch everything go to nothing. This is the "clear high scores" in a service menu. . *Write a thousand rows.* Loop `record` a thousand times and time it with `singeGetTicks`. Then put `runs:exec("BEGIN")` before the loop and `runs:exec("COMMIT")` after it, and time it again. A *transaction* makes many writes one, and the difference is large enough to be worth remembering. === Break It on Purpose Take `addRun:reset()` out of `record` and play three rounds on three different stages. No error. The game plays perfectly. But the table at the right says you played the first stage three times, and the best runs are three copies of your first score. `bind_values` could not put new values into a statement that had already run and not been reset, so it gave up -- quietly, returning a number you were not checking -- and `step` ran the statement again with the values it already had. This is the bug that matters in this lesson, because it does not look like one. A query that fails is loud. A write that writes the wrong thing is silent, and you find it a week later in the data. When the rows in a database are not what you expect, look at the code that put them there before the code that reads them. Put `reset` back, and break it the loud way: change `score` to `scroe` in the first `SELECT` in `refresh`. ---- Error running script: no such column: scroe stack traceback: [C]: in method 'nrows' 29-databases.singe:43: in local 'refresh' 29-databases.singe:133: in main chunk ---- SQLite reads the column names when it prepares the question, so a misspelled one stops the game at once, and the message names it exactly. The traceback reads from the inside out: `nrows` complained, `refresh` called it at line 43, and the script itself called `refresh` at line 133, the last line, as the game started. That is why it says *running script* rather than naming a function the engine called: it never got as far as the first frame. === What You Learned * A database is a file of tables; a table is rows with named columns. * `require("sqlite3")` gives you SQLite. `sqlite3.open` opens a file and creates it if it is missing. * What the game writes goes in `singeGetDataPath()`. What you ship goes in your folder, opened with `sqlite3.OPEN_READONLY`. * `CREATE TABLE IF NOT EXISTS` is safe to run every time the game starts. * Values go in through `prepare`, `bind_values`, `step`, and `reset`, never by gluing strings into SQL. * `nrows` hands back rows as tables; `urows` hands back the columns themselves. * `ORDER BY`, `LIMIT`, `GROUP BY`, `MAX`, and `COUNT` answer the questions you would otherwise write loops for. * Ask when the answer can change, not every frame. * Statements are finalized, then the database is closed, in `onShutdown`. === Next Time Your game now has a folder full of things: a script, art, sounds, and a database or two. Lesson thirty turns that folder into one file a stranger can copy onto their own machine and play -- and your read-only database goes with it, still opening exactly as it does now.