Testing

How this workspace tests its parsers, and the habits that make a passing test worth something.

A test here has one job: prove that what we wrote matches what the engine does. Most of the ways a test fails at that job still look like success, so the practices below are mostly about arranging for a check to be capable of failing before you rely on it passing.

Strategy

We use a gray-box approach. White-box knowledge of the engine, gathered from the audits under formats/, is used to build strictly-validated black-box tests. Tests target how the engine actually behaves, not a mock of it.

When you add a new format, include:

• Fixture-backed tests. Full round-trip coverage over synthetic files in fixtures/. Never commit real game assets; run cargo test --test gen_fixtures -- --ignored to generate them. Byte-exact round-trip assertions are the bar for any format the engine reads byte for byte.
• Mutation tests. A pass confirming the parser rejects malformed and truncated input without panicking, usually through corruption_matrix.rs.
• Module documentation. A rustdoc block showing the format layout.

Reference a fixture with concat!(env!("CARGO_MANIFEST_DIR"), "/../../fixtures/name.ext") rather than a path relative to the source file, so moving the file does not break it.

Writing a check

Make it prove it arrived

Assert on what the check examined, not only on what it found. Count the subjects it compared and assert the count is not zero.

This is the single highest-value habit on the page, because the default failure of a check is silence rather than noise. cargo test <filter> prints test result: ok when the filter matches no tests at all, which is why scripts/test-filter.sh exists and fails when a filter matches nothing.

The same shape recurs at every level. A value comparison walked one side’s fields and skipped any label the other side lacked, so renaming a field on both sides passed. A test for diagnostic grouping hit a type-mismatch rule first and returned before reaching the grouping. Schema checks recurse only where an entry declares child fields, so anything below a childless entry was never visited; three separate checks did this independently, and descending by default is what stops the fourth.

Count what the check compared, not what it visited. Those are different numbers, and the gap between them is where this hides.

Take the input from somewhere the subject does not control

If the thing under test also supplies the test’s input, whatever is missing from the subject is missing from the input that would reveal it. Adding more cases cannot break that loop.

A check that a view writes no label its schema fails to declare built its input from that same schema. A saved object writes either a Portrait resref or a PortraitId row index, and Portrait was declared nowhere. No Portrait was generated, so the read always took the id branch, so the missing declaration was never written, so the check never saw it.

The weaker version of this is a generated fixture. A generated fixture exercises only what its generator produced. Child fields it does not build, lists it leaves empty, and branches it does not select are all outside reach, and none of them look like gaps from inside the test:

• A check written to catch skipped levels built its input with Default::default(), whose lists are empty, so it stopped above every level it existed to cover.
• A coverage check on Git built its input from the schema, which filled UseTemplates with an arbitrary non-zero value. That flag selects between two forms of the file, so every run took the same branch and the Saved* types were never called.

That last one deserves care, because it leaves no trace. Nothing is missing, nothing is empty, and the walk completes normally. It goes one way at a fork and the other way is invisible.

Compare in a vocabulary the code under test did not choose

Two things have to be made comparable before a check can compare them. When the code under test is what makes them comparable, the check inherits its blind spot exactly.

A typed view models a field only where the engine reads it at that path, which is right for reading and wrong for writing anything back. A round-trip guard read a file into a view, wrote it out, and compared the two views. Every field it compared agreed, because every field the view does not model was missing from both sides. A writer that lost a save’s entire session state would have passed.

The tell is that the check never leaves the model. “Read a Uti, write a Uti, compare the Utis” is a closed loop, and no fact from outside can get in to fail it. That is visible while you are writing it, which is what separates this from a check that turns out afterwards to have been looking at the wrong thing.

Here the outside vocabulary is the file’s own bytes: read a real file, write the parsed tree back, compare against what was read. One such check runs over a single saved area carrying labels no view models, another over every GFF the install ships.

Break it on purpose before you trust it

Reintroduce the defect, run the check, and confirm the assertion you expected is the one that fails. Most of the problems on this page were caught this way and by nothing else. A test you have never seen fail has not yet earned anything.

Break each half of a compound check separately. A grouping keyed on both a field’s path and its reason had to fail differently for each half, which is what made the key defensible rather than merely plausible.

Two specific shapes need more than one break:

• A fallback with several preconditions needs a case that removes all of them. A portrait falls back only when Portrait and PortraitId are both missing, so dropping either one left the other in the file and the read took a branch rather than its fallback. List the reachable states before trusting a probe that varies one input.
• If a check runs over several subjects, breaking one subject must fail it. A fix that declared a label on four types and asserted both branches were produced was satisfied by any one of them, so reverting a single declaration left three siblings covering for it.

Choosing your evidence

Prefer real files, and know what yours cannot show

Real game files are the only source nobody on this project wrote, which makes them the only source that can disagree with us.

The minigame types read their enemy and obstacle lists off the wrong parent. The reader had it wrong, the module diagram in are.rs had it wrong, and the round-trip test had it wrong, because one person wrote all three from one reading of the format. Three sources agreeing was one mistake stored three times, and every enemy and obstacle in the game parsed as absent for as long as the type existed: 95 enemies and 67 obstacles across the four minigame areas.

A corpus still has to be large enough to tell two explanations apart. A waypoint reader keyed its map note off the note text and derived the HasMapNote flag back from it, defended by a comment observing that the two never disagree across the fixture saves. They do not disagree, and that is not what the fixtures showed: across a larger local corpus the flag is present on every waypoint while the text is present on a minority. What looked like two fields tracking each other was one field always being written.

A corpus too small to hold a counter-example cannot tell a relationship from a constant, and the reading that assumes a relationship is the one that then justifies ignoring a field.

Know which populations no corpus can reach

Ranking work by how often a fallback fires in the corpus puts saved forms last, because their fallbacks never fire: the engine’s saver writes saved objects exhaustively, so every field is present on every saved entry.

So the population with no corpus evidence is exactly the one where documentation is the only oracle that will ever exist, and a firing-rate ranking sends it to the back of the queue. A fallback no corpus can reach is not low priority, it is unfalsifiable by measurement, and it needs the reading rather than less of it.

When a check built on documentation disagrees with the code, the documentation decides

Unless you can say from the documentation why it is wrong. Settling it the other way turns the check into a comparison between the code and itself.

A schema entry’s absent-value records what the engine holds for a field a file omits, populated from the audit pages under docs/src/formats/. A check compared each reader’s fallback against it, and that check is worth more than a round-trip precisely because its two sides come from different places and can disagree.

Utd’s Invulnerable fell back to the file’s Plot value, and the schema entry was changed to match on the reasoning that the reader looked right. It was not: the engine reads Invulnerable before Plot, so its fallback sees the constructor’s zero and never the file’s value, which is what the page says. Every disagreement settled that way turns one more entry into a copy of the reader.

Counting and measuring

Ask what the population can exhibit

A count can be wrong while its arithmetic is right, because the error is in the population. Three counts in one audit each needed a whole category subtracted after the fact:

The shared cause: the inflating members were structurally unable to exhibit the property being counted. You cannot agree without an opinion, cannot be unconditional if you are conditional, cannot constrain a merge if you never write. In each case the measure looked at the label while the property lived at the label and its context.

The third was predicted from the first two before it was found, which is what makes this a pattern to apply rather than three anecdotes. It is three-for-three rather than a law, so check the fourth.

Recompute derived properties after you subset

The same error runs the other way. Having crossed those divergences against the write set, the survivors were reported with the axis each had been classified under before the crossing. Two changed: OnHeartbeat and OnUserDefined diverged on liveness only because of a mount that declares them and never writes them, so removing non-writers left them diverging on default alone.

Membership changed, so every property derived from membership changed with it. One version of this error is a property held by members that cannot exhibit it; this is a property computed on a population that no longer exists.

Spot-check a probe, and say which way it errs

Two probes written for those counts had a predicate that did not match the property. One used “is this path present when the view is default-constructed” as a proxy for “is this field written unconditionally”. It returned 31 against a true 83, because a list child is absent at default when the list is empty, not when the write is conditional. The other missed the multi-line form of a helper whose label is its second string argument, so it under-counted writers.

Both were caught by checking a handful of cases the author already knew the answer to. Neither would have been caught by reading the code again.

Note the direction as well as the size. Under-counting writers removes real constraints, so that probe failed toward “safe to share”, the direction that produces a bad merge rather than a noisy report. A probe’s error direction belongs alongside its result, because a conservative probe and an optimistic one with the same error rate are not equally usable.

Check the catalogues before calling a finding new

Several findings in one session were reported as fresh and turned out to be on record already. A liveness condition sat in an issue comment while the issue body listed the others. An exclusion’s reasoning sat in a comment on the issue it constrained. A modelling gap was already carried by the schema, at the declaration for the path it was about.

The records were good. None were reachable from where the question got asked, which is the part worth fixing. Before flagging an oddity, check the schema tables, the declaration for the path itself, and the issue comments rather than only the issue bodies.

Reading a result

A pass can mean the check compared nothing

Two shapes produce this, and neither looks like a skip.

The data is not there. A check comparing a type’s Default against the engine’s constructor values reached a nested block through its parent, whose field for that block is an Option defaulting to None. Nothing is written under it, so every path resolves to nothing, so there is nothing to compare, and having nothing to compare reads exactly like agreement. An optional container that defaults to absent hides its children from anything reaching them through the parent. Check the child type directly rather than making the parent write something it otherwise would not.

The entry opted out. Schema entries carrying Unexamined were skipped deliberately, since nobody has read the page and failing them would report unread pages as reader defects. The effect is not “unknown” but exempt from every guard: an unpopulated declaration and a verified-correct one become indistinguishable, while the entry is still walked, still counted as reached, and still passes the coverage assertion meant to catch checks that stop short. Thirteen saved-form script hooks and two trap fields read empty where the engine holds "default" and an armed flag, under a guard with a test named after that exact type and green results throughout.

A skip condition on the data is a coverage hole that reports as coverage. Treat a per-entry opt-out as a bucket to be reported, never as a state to be skipped past.

A failure can be the guard’s fault

A list of labels the typed views deliberately stop writing needs a guard, because a wrong entry silently drops a field from files that carry it. The obvious guard asks whether each listed label is still in the corpus validator’s “written but in no file” set, and it passes today. Then the omission lands, the views stop writing those labels, and a label nobody writes is no longer in that set. Every successfully omitted label reports as a violation.

A check derived from a set the change mutates cannot stay true after the change it guards. The tell is a guard asking whether X is still in S when the change is precisely what removes X from S. It reads as solid right up to the moment it matters.

Derive the guard from the invariant that survives the change, not the state that holds now. Here that invariant is “no real file carries this label”, a fact about the corpus alone, so the guard asks the corpus and means the same thing on both sides of the change.

This one was caught immediately because it was total: every label in the first batch failed at once, which is obviously a broken check rather than that many broken fields. The dangerous version is partial. A mix of already-omitted and not-yet-omitted labels would have produced a handful of reds that looked like findings, and findings get explained.

A comment that explains a finding away is a claim

validate gff compares the typed views against the vanilla corpus in both directions: labels real files carry that no view models, and labels a view writes that appear in no file. Repos_Posy sat in the first list and Repos_PosY in the second, which is one field read under one spelling and written under another, reported twice from two angles. The tool’s own module documentation named the pair and explained it as the label-normalization policy working as intended. Every container item this library wrote lost its grid position for as long as that note stood.

The note was not lazy. It was written from the same reading of the format that produced the writer, which is a check repeating the code’s mistake wearing different clothes. What makes it worth separating is where it lands: not in a test that agrees with the code, but in prose that removes a disagreeing result from consideration. Running the tests cannot falsify a sentence.

Two habits follow. Write down what you checked rather than what you concluded, because “known false positive” is unfalsifiable while “these 4215 files carry the label and none carry the other spelling” is something a later reader can re-run and disagree with. And treat a matched pair as a question: a label the corpus has and the view lacks, alongside a label the view has and the corpus lacks, is the shape a rename leaves behind.

None of this argues for writing more tests. It argues for knowing what the ones you have have actually looked at.