Architecture Guide

This document outlines how the rakata workspace is structured and the design principles we try to stick to.

Core Principles

1. Vanilla K1 First

   • By default, we target the original vanilla behavior of KotOR 1.
   • Compatibility for TSL or community tools is strictly opt-in behind feature flags, not the default assumption.
   • When deciding how to parse something, the original game engine is our ultimate source of truth. We use local fixtures and original game data to prove our parsers work, rather than just copying how older community tools did things.

2. Aim for Lossless

   • We want to be able to read a file and write it back out to the exact same bytes. We’ve largely achieved this for standard archives and data formats (GFF, ERF, RIM, KEY, TLK, etc.).
   • For highly complex formats (like MDL/MDX models), there are some known divergences where achieving a byte-exact roundtrip is essentially impossible due to how the original compilers ordered geometry blocks. We track these exceptions, but the output still safely runs in-game.
   • No Lazy Pass-throughs: undocumented fields don’t get read as an opaque Vec<u8> and passed through. The goal is to map every struct boundary properly.
   • Reserved fields are the exception: where the binary layout defines a reserved field whose meaning we haven’t cracked, we map it as a correctly-sized reserved value rather than dropping data the engine might rely on. Explicit blank padding isn’t stored at all; it’s recalculated on write.
   • Layer scope: this lossless guarantee applies to the byte-level format layer (rakata-formats). Typed views in rakata-generics (Utc, Uti, Are, …) are explicitly honest projections that model only the fields they enumerate; byte-exact preservation stays with the raw Gff tree. See Typed Views and Raw GFF below for the full rule.

3. Strict Text Handling

   • All text decoding goes through rakata-core::text.
   • Localized text (TLK entries, strings) uses language-aware encodings (Windows-1252, Shift-JIS, etc.) to match what the engine expects.
   • Binary strings (like node names or texture paths) use TextEncoding::Windows1252 since that’s what the engine actually uses under the hood. No silently stripping weird characters with lossless backups.

(For day-to-day coding rules around iterators, zero-cost abstractions, and memory safety, see the Idiomatic Rust section in the contributing.md guide!)

Workspace Boundaries

Note: this layout is a moving target. rakata-audio is planned and does not exist yet, and the tools are at very different stages, so expect these crates to fill out and new siblings to appear as the roadmap goals land.

The workspace is organized in a clean dependency chain. Crates can only depend on crates listed “above” them:

rakata-core          (no workspace deps)
  rakata-formats     (depends on: core)
    rakata-extract   (depends on: core, formats)
    rakata-generics  (depends on: core, formats)
    rakata-lint      (depends on: core, formats, extract, generics)
    rakata-patcher   (depends on: core, formats, extract)
    rakata-save      (depends on: core, formats)
  rakata-install     (depends on: core)
rakata               (facade: re-exports all library crates)

rakata-ui            (no workspace deps; the GUI tools only)

  rakata-audio       (planned, not yet created)

Library Crates (crates/)

• rakata-core: The absolute basics (ResRef, IDs) and core utilities like file streams and text encoding.
• rakata-formats: Our massive library of parsers and writers (GFF, ERF, BIF, MDL, TPC, etc.). This parses bytes into objects, but doesn’t know anything about how the game actually uses them.
• rakata-audio (planned): Audio streaming and decoding for the engine’s various sound formats (WAV, ADPCM, MP3). Not yet created; WAV reading currently lives in rakata-formats.
• rakata-formats-derive: The proc macro behind #[derive(GffModel)]. Only rakata-formats depends on it, but it is where a field’s declaration is turned into a reader, a writer and a schema entry at once, and where several contracts are enforced as build errors rather than as tests.
• rakata-generics: Strongly-typed Rust models for all the different GFF files (like Doors, Items, Characters).
• rakata-extract: The logic for hunting down actual game files in the wild. It knows how to look inside ERFs, check the Override folder, and resolve files just like the engine does.
• rakata-lint: Our rule engine for scanning modded files and checking them against vanilla schema constraints.
• rakata-save: High-level logic for safely reading, editing, and backing up save files.
• rakata-install: Finds the game on this machine. It reports every copy it can see rather than picking one, because anybody who mods has several. The only crate that knows what operating system it is on.
• rakata-patcher: Installing a loose-file mod and taking it back. It reads a payload as a folder or as the archive it downloaded as, works out what installing it would replace, and keeps a copy of every file it overwrites beside a record naming them.
• rakata-ui: What the GUI tools share. Colours named for what they mean, the controls somebody should recognise between tools, the two window shapes, and the log each tool writes.
• rakata: A handy facade crate that re-exports everything so you only need to add one dependency.

Tool Crates (tools/)

• rakata-modinstaller: The desktop application for installing loose-file mods where the filesystem tells Override and override apart.
• rakata-saveeditor: The actual desktop application for editing save files.
• vanilla-inspector: A testing utility for validating our parsers against the actual mass of game files.

Format API Guidelines

Public API Shape

Every format parser in rakata-formats generally provides the same clean interface:

• read_<fmt><R: Read>(reader: &mut R) -> Result<T, E>
• read_<fmt>_from_bytes(bytes: &[u8]) -> Result<T, E>
• write_<fmt><W: Write>(writer: &mut W, data: &T) -> Result<(), E>
• write_<fmt>_to_vec(data: &T) -> Result<Vec<u8>, E>

Formats with multiple output modes (like exporting models to ASCII text or JSON) just use variations of these names (read_mdl_ascii()).

• Generic Traits: We strongly prefer accepting generic I/O trait bounds (Read, BufRead, Write, Seek) over concrete types. Accept the narrowest trait that covers your API’s needs so callers aren’t forced to jump through hoops.

Error Handling

Robust parsing means strict error boundaries:

• Each format module must define its own domain-specific error enum (e.g., GffError, ErfError) using the thiserror crate. Do not use generic stringly-typed errors or Box<dyn Error>.
• Low-level read failures (like sudden bounds exhaustion or bad magic numbers) should wrap our shared BinaryLayoutError.
• Never unwrap() at an API boundary! Only fail explicitly via Result or use .expect() with a hardcoded rationale if it is impossible to fail.

Memory & Ownership

While we try to avoid deep cloning and heavy allocations behind the scenes, we default to owned data types when crossing public API boundaries. Unless a module is explicitly built and documented as a zero-copy “View” type, you should avoid passing nasty lifetimes into the caller’s lap.

Keeping Concerns Separated

• Dumb Parsers: Format modules in rakata-formats are intentionally “dumb”. They solely translate between raw byte streams and Rust structs without any awareness of game architecture, filesystems, or what a “module” is.
• Smart Extractors: The environment logic lives in rakata-extract: finding loose files, enforcing precedence (checking Override/ before extracting from a BIF), and assembling composite files. That separation is why the parsers work on an isolated test fixture and on a live install alike.

How 2DA Tables Reach a Decoded View

Decoded views resolve file-native values against 2DA tables, so something has to hand them a table. resolve() takes &mut impl TwoDaSource, and the three concerns that twoda_cache.rs once fused into one file live in three crates:

Generics needs identity and capability, not implementation, so rakata-extract appears nowhere in the generics manifest, neither in [dependencies] nor [dev-dependencies]. Tests spanning both crates live in the rakata facade.

TwoDaSource::twoda returns Option<&TwoDa> rather than a Result, because every consumer discarded the error. A caller that needs to tell “no such table” from “the bytes would not parse” asks the cache directly through its inherent method, which still carries the full TwoDaCacheError.

rakata-lint also depends on rakata-extract, and that is not the same call. Lint is a leaf, so a concrete dependency propagates to nobody. Generics is foundational, so its dependencies reach everything above it. Weigh this by where a crate sits in the graph rather than by whether the edge looks tidy on its own.

Tracing & Telemetry

We strongly encourage instrumenting format parsers with tracing::instrument spans to help pinpoint exactly where a badly formed file breaks during a parse. However, this telemetry must remain entirely zero-cost for consumers who don’t need it! We achieve this by wrapping public parser entry points in conditional attributes: #[cfg_attr(feature = "tracing", tracing::instrument(...))]. If a user doesn’t explicitly opt-in via their Cargo.toml, the Rust compiler strips the instrumentation entirely.

Serialization (Serde)

Just like tracing, serde support for exporting our parsed files to JSON or YAML must be treated as a zero-cost, opt-in feature. Format structs and types should generously derive Serialize and Deserialize when the serde feature flag is enabled. This allows downstream utilities (like the Save Editor) to effortlessly convert memory layouts into text formats, while ensuring the core parsers stay extremely light for purely binary-focused applications.

Beyond Basic Parsing

While rakata-formats gives us the ability to parse isolated bytes, the game engine is much more complicated. Our higher-level crates exist to bridge that gap between “dumb bytes” and “actual game logic”.

Finding Files (rakata-extract)

rakata-extract handles the messy reality of finding files scattered across a massive KOTOR installation. It mirrors the vanilla engine’s lookup hierarchy in three distinct layers:

1. Primitives: Grabbing a file out of a single archive (like unpacking a standalone ERF or BIF file).
2. Composition: Treating related archive sets as a single “Module” (like grouping a .mod file with its matching _s.rim and _dlg.erf files so they load transparently together).
3. Game-wide: A GameVfs rooted at one install that owns each tier (chitin / BIFs, the Override/ directory, caller-pushed extra overrides, the single active CompositeModule, and a mounted save above all of them) and resolves resrefs through them in engine precedence order. See Resource System & Resolution for the full tier order and what the save tier does and does not shadow.

Because we want our extraction to perfectly mirror vanilla behavior, lookups are strictly case-insensitive, and loading precedence is explicitly designed to mirror how the original game works (so a file in the Override folder automatically beats a file buried in a BIF archive).

Strongly-Typed Data (rakata-generics)

When we parse a .utc Character file, rakata-formats just hands us a raw GFF tree of untyped labels and values. rakata-generics wraps those raw data blobs in strongly-typed Rust structs (Utc, Uti, Are, Git, Dlg, Ifo, and friends). This guarantees that if a developer needs to access a character’s “Strength” stat, they get a guaranteed u8 property rather than blindly guessing string handles inside a raw binary tree.

Typed Views and Raw GFF

These typed structs sit beside the raw Gff tree, not on top of it. They are projections, not replacements. You construct one with Uti::from_gff(&gff) and round-trip back with uti.to_gff(); the original Gff stays accessible the whole time.

The projection layer follows one load-bearing rule: model what’s enumerated; drop what isn’t. from_gff extracts the fields each typed view documents and silently ignores anything else; to_gff writes only those documented fields. There is intentionally no extra_fields: Vec<GffField> accumulator on Utc / Uti / Are / etc. that would round-trip unmodelled fields through the typed layer.

The reason is correctness. Unmodelled fields often depend semantically on neighbouring fields (a savegame’s animation state only makes sense at the exact moment of save; a toolset’s custom annotations describe a specific revision). If the typed view silently preserved them while a caller edited a modelled field, the output would be internally inconsistent. The staleness contract is real, but it belongs explicitly with whoever needs byte-exact preservation, not buried inside a layer whose only job is type-safe access to known fields.

This splits by what a caller is doing to the file, which is not the same question as what kind of tool it is:

• Reading it, to inspect or report. A linter or an inspector wants the typed view: type-safe access to known fields, and unmodelled bytes are nobody’s concern because nothing goes back to disk.
• Authoring a new file. The typed view again, and reconstruction is correct here because there is no original to lose.
• Editing a file that already exists. The raw Gff and its path API, never the typed view. A projection written back drops every field it does not model, and in a savegame that is most of them: another area’s object state, a mod’s own labels, anything unaudited. Nothing warns you, because a dropped field and a field the file never had look identical on the way out.

The verb decides it, not the program. A save editor reads through a typed view and edits through the tree, and it is the same save editor doing both. Any tool preserving toolset annotations or auditing round trips is in the third case for the same reason.

What gets enumerated: the engine has to read it

“Model what’s enumerated” only helps once you know what earns a place in the enumeration. The criterion is narrow on purpose:

A typed view models a field if and only if the engine reads it at that GFF path.

A field the engine never reads is a dead field. It gets recorded as dead where a reader will meet it, with the evidence, and it gets diagnosed by a Phase 1 lint rule walking the raw Gff. It does not go in the typed view. That is not a hole in the projection. crates/rakata-lint/ARCHITECTURE.md’s first design principle already puts schema and intra-resource checks on the raw tree for exactly this reason: they exist to validate the fields typed views drop.

Four clarifications carry most of the weight.

“At that path” is the whole test. A door blueprint’s Tag is read; the Tag on a door placement in a GIT is not, because the engine takes a templated door’s tag from the blueprint. Same label, same concept, one live copy and one dead one. A concept being live somewhere else says nothing about this copy.

Prevalence is not a criterion, and neither is carrying a real value. WaypointList[].TemplateResRef appears in every waypoint in a retail install and no load path reads it. The counts mislead in every direction, and all three have happened:

The corpus tells you where to look, never what to model.

“The blueprint wins” is not the rule, and assuming it gets half the cases backwards. The canonical dead field is a placement’s copy losing to the blueprint’s, and there is a family running the other way. A door’s Tag is read only on the blueprint, so the placement’s copy is dead. A door’s TransitionDestin and its LinkedTo siblings are read on the blueprint and then overlaid from the placement immediately afterward, so it is the blueprint’s copy that is dead. Triggers repeat both halves.

Same engine, same load, adjacent fields, opposite winner. What decides it is whether an overlay step exists for that field, not which struct the read was handed. So the question to ask of a new field is “does anything write over this after the read”, and a finding reporting only which struct a value came from has not answered it.

“Path” means the GFF path, not a decode-time variant. The rule stops at the structural position and does not reach inside a decoded enum. PropertyList[].Subtype is read at its path, so it is modelled, even though which property kinds actually consume it varies.

Ask whether there is any configuration in which the engine reads this label at this structural position. For Subtype, yes. For WaypointList[].TemplateResRef, never.

The point of the lint rule is that a dead field is a modder trap: someone sets it, expects an effect, and gets nothing. Silence is the worst answer, and the typed view is the wrong place to break that silence.

Ifo.Mod_Hak, Utc.SaveWill and Utc.SaveFortitude are modelled and documented dead. They predate this rule and round-trip harmlessly, so they stay. Treat them as legacy exceptions rather than precedent; the next field’s case has to stand on the criterion above rather than on their existence.

When in doubt: if you need byte-exact preservation across a parse-then-write cycle, work with the raw Gff. If you need ergonomic, type-safe access to the fields Rakata has audited, work with the typed view.

Decoded Views: Removed, Pending a Consumer

A second layer used to sit on top of the typed structs. A decoded view resolved file-native ids against the 2DA tables the engine consults, so a caller could ask a creature for its race label rather than its race id. It covered UTI and UTC, and it has been removed.

Nothing irreplaceable went with it. A decoded view resolved through ContentSource at run time, so the interpretation lived in the game’s own tables rather than in our source; rebuilding it re-derives from data we still read. What made it worth removing is that it was built against an architecture we have since replaced, it reached two of the fourteen typed views, and no consumer ever stated what it needed. It returns once one can, which is after MountedSave and the write path.

Two things the rebuild inherits rather than re-deciding:

• Projection and resolution are separate stages. File-native variant dispatch is scope-free and runs once; resolving against a context runs per scope. Mod-conflict analysis, vanilla-versus-modded diffs, and “read this the way a mounted save sees it” all reduce to one projection resolved several ways.
• A mod-extensible table needs an Unknown variant. Mods add rows to itempropdef.2da, so a decode that cannot name a kind has to surface the raw entry instead of dropping it. Per entry, rather than a struct-level accumulator.

ContentSource, NoSource and InstallContent stay put for it.

High-Level Interaction (rakata-save & rakata-lint)

Finally, crates at the top of the stack use our extraction logic and strongly typed generic structs to actually do things. rakata-lint compares typed structs against vanilla constraints to catch modding errors, while rakata-save gracefully handles unpacking, editing, and re-compressing massive save-game directories without corrupting the player’s campaign!