Architecture Guide

This document outlines how the rakata workspace is structured and the design principles we try to stick to.

Core Principles

1. Vanilla K1 First

   • By default, we target the original vanilla behavior of KotOR 1.
   • Compatibility for TSL or community tools is strictly opt-in behind feature flags, not the default assumption.
   • When deciding how to parse something, the original game engine is our ultimate source of truth. We use local fixtures and original game data to prove our parsers work, rather than just copying how older community tools did things.

2. Aim for Lossless

   • We want to be able to read a file and write it back out to the exact same bytes. We’ve largely achieved this for standard archives and data formats (GFF, ERF, RIM, KEY, TLK, etc.).
   • For highly complex formats (like MDL/MDX models), there are some known divergences where achieving a byte-exact roundtrip is essentially impossible due to how the original compilers ordered geometry blocks. We track these exceptions, but the output still safely runs in-game.
   • No Lazy Pass-throughs: If a file has undocumented fields, we don’t just read them as an opaque Vec<u8> blob and blindly pass them through. Our goal is to properly reverse-engineer and map every single struct boundary. However, if we identify defined “reserved” fields in the binary layout that we haven’t cracked the meaning of yet, we will map them as properly sized reserved values so we don’t accidentally drop data the engine might rely on. (Note: explicit blank padding bytes aren’t stored in memory at all - we just recalculate those dynamically on write).
   • Layer scope: this lossless guarantee applies to the byte-level format layer (rakata-formats). Typed views in rakata-generics (Utc, Uti, Are, …) are explicitly honest projections that model only the fields they enumerate; byte-exact preservation stays with the raw Gff tree. See Typed Views and Raw GFF below for the full rule.

3. Strict Text Handling

   • All text decoding goes through rakata-core::text.
   • Localized text (TLK entries, strings) uses language-aware encodings (Windows-1252, Shift-JIS, etc.) to match what the engine expects.
   • Binary strings (like node names or texture paths) use TextEncoding::Windows1252 since that’s what the engine actually uses under the hood. No silently stripping weird characters with lossless backups.

(For day-to-day coding rules around iterators, zero-cost abstractions, and memory safety, see the Idiomatic Rust section in the contributing.md guide!)

Workspace Boundaries

Note: This layout is a living target! rakata-saveeditor is under active development, and rakata-audio is planned but does not exist yet. As we tackle our near-term roadmap goals – like building out the rakata-lint validation engine – expect these crates to flesh out, alongside brand new sibling crates being added to the ecosystem.

The workspace is organized in a clean dependency chain. Crates can only depend on crates listed “above” them:

rakata-core          (no workspace deps)
  rakata-formats     (depends on: core)
    rakata-extract   (depends on: core, formats)
    rakata-generics  (depends on: core, formats)
    rakata-lint      (depends on: core, formats, extract, generics)
    rakata-save      (depends on: core, formats)
rakata               (facade: re-exports all library crates)

  rakata-audio       (planned, not yet created)

Library Crates (crates/)

• rakata-core: The absolute basics (ResRef, IDs) and core utilities like file streams and text encoding.
• rakata-formats: Our massive library of parsers and writers (GFF, ERF, BIF, MDL, TPC, etc.). This parses bytes into objects, but doesn’t know anything about how the game actually uses them.
• rakata-audio (planned): Audio streaming and decoding for the engine’s various sound formats (WAV, ADPCM, MP3). Not yet created; WAV reading currently lives in rakata-formats.
• rakata-generics: Strongly-typed Rust models for all the different GFF files (like Doors, Items, Characters).
• rakata-extract: The logic for hunting down actual game files in the wild. It knows how to look inside ERFs, check the Override folder, and resolve files just like the engine does.
• rakata-lint: Our rule engine for scanning modded files and checking them against vanilla schema constraints.
• rakata-save: High-level logic for safely reading, editing, and backing up save files.
• rakata: A handy facade crate that re-exports everything so you only need to add one dependency.

Tool Crates (tools/)

• rakata-saveeditor: The actual desktop application for editing save files.
• vanilla-inspector: A testing utility for validating our parsers against the actual mass of game files.

Format API Guidelines

Public API Shape

Every format parser in rakata-formats generally provides the same clean interface:

• read_<fmt><R: Read>(reader: &mut R) -> Result<T, E>
• read_<fmt>_from_bytes(bytes: &[u8]) -> Result<T, E>
• write_<fmt><W: Write>(writer: &mut W, data: &T) -> Result<(), E>
• write_<fmt>_to_vec(data: &T) -> Result<Vec<u8>, E>

Formats with multiple output modes (like exporting models to ASCII text or JSON) just use variations of these names (read_mdl_ascii()).

• Generic Traits: We strongly prefer accepting generic I/O trait bounds (Read, BufRead, Write, Seek) over concrete types. Accept the narrowest trait that covers your API’s needs so callers aren’t forced to jump through hoops.

Error Handling

Robust parsing means strict error boundaries:

• Each format module must define its own domain-specific error enum (e.g., GffError, ErfError) using the thiserror crate. Do not use generic stringly-typed errors or Box<dyn Error>.
• Low-level read failures (like sudden bounds exhaustion or bad magic numbers) should wrap our shared BinaryLayoutError.
• Never unwrap() at an API boundary! Only fail explicitly via Result or use .expect() with a hardcoded rationale if it is impossible to fail.

Memory & Ownership

While we try to avoid deep cloning and heavy allocations behind the scenes, we default to owned data types when crossing public API boundaries. Unless a module is explicitly built and documented as a zero-copy “View” type, you should avoid passing nasty lifetimes into the caller’s lap.

Keeping Concerns Separated

• Dumb Parsers: Format modules in rakata-formats are intentionally “dumb”. They solely translate between raw byte streams and Rust structs without any awareness of game architecture, filesystems, or what a “module” is.
• Smart Extractors: All the messy environment logic – hunting down loose files, enforcing vanilla precedence rules (e.g., checking the Override folder before extracting from a BIF archive), and assembling composite files – lives safely isolated inside rakata-extract. This separation guarantees our parsers can cleanly process isolated test files just as well as they operate in a massive live-game workflow.

How 2DA Tables Reach a Decoded View

Decoded views resolve file-native values against 2DA tables, which means something has to hand them a table. For a long time that something was TwoDaCache itself, and since the cache is GameVfs-backed, rakata-generics depended on rakata-extract to get it. That edge pointed backwards: generics is foundational and extract sits above it, so every consumer that only wanted to parse and resolve bytes got the whole VFS along for the ride.

The fix was to notice that twoda_cache.rs had three separable things fused into one file:

• Identity – what tables exist and what they are called: TwoDaName, tables::*. This is game-content knowledge of the same kind ResourceType carries, so it lives in rakata-core. A 2DA parser has no business knowing appearance.2da exists.
• Capability – something can hand me a table by name: the TwoDaSource trait. It returns &TwoDa, so rakata-formats is the lowest crate that can name the return type. A home in core would need core -> formats, which would be a worse inversion than the one being fixed.
• Implementation – the VFS hands me tables and I remember them: TwoDaCache, TwoDaCacheError. The cache holds a &GameVfs, and nothing below extract knows what a VFS is, so this stays in rakata-extract.

Generics needs identity and capability. It never needed implementation; it was taking implementation in order to get the other two. resolve() now takes &mut impl TwoDaSource, and rakata-extract appears nowhere in the generics manifest – not in [dependencies], and not in [dev-dependencies] either. Tests that span both crates live in the rakata facade, which is the umbrella over all four members and the natural home for tests that cross them.

TwoDaSource::twoda returns Option<&TwoDa> rather than a Result. Every consumer discarded the error, so an error type would have carried a variant nothing reads. A caller that genuinely needs to tell “no such table” from “the bytes would not parse” asks the cache directly through its inherent method, which still carries the full TwoDaCacheError.

rakata-lint also depends on rakata-extract, but not under the same rule, and the two should not be cited together. Lint is a leaf: nothing depends on it, so a concrete dependency propagates to nobody and there is nothing to revisit. Generics is foundational, and that is the whole reason its case stayed open long after it was first flagged. Weigh this kind of call by where the crate sits in the graph, not by whether the edge looks tidy in isolation.

One process note worth keeping. This was flagged as an architectural error once and correctly deferred, on the grounds that every real consumer already depended on rakata-extract anyway, so building the trait then would have been the speculative abstraction the rest of this guide warns against. The deferral named a condition to revisit on. That condition quietly came and went and nothing prompted anyone to look, because a trigger only works if something is watching it. The second time it came up it became a ticket instead.

Tracing & Telemetry

We strongly encourage instrumenting format parsers with tracing::instrument spans to help pinpoint exactly where a badly formed file breaks during a parse. However, this telemetry must remain entirely zero-cost for consumers who don’t need it! We achieve this by wrapping public parser entry points in conditional attributes: #[cfg_attr(feature = "tracing", tracing::instrument(...))]. If a user doesn’t explicitly opt-in via their Cargo.toml, the Rust compiler strips the instrumentation entirely.

Serialization (Serde)

Just like tracing, serde support for exporting our parsed files to JSON or YAML must be treated as a zero-cost, opt-in feature. Format structs and types should generously derive Serialize and Deserialize when the serde feature flag is enabled. This allows downstream utilities (like the Save Editor) to effortlessly convert memory layouts into text formats, while ensuring the core parsers stay extremely light for purely binary-focused applications.

Beyond Basic Parsing

While rakata-formats gives us the ability to parse isolated bytes, the game engine is much more complicated. Our higher-level crates exist to bridge that gap between “dumb bytes” and “actual game logic”.

Finding Files (rakata-extract)

rakata-extract handles the messy reality of finding files scattered across a massive KOTOR installation. It mirrors the vanilla engine’s lookup hierarchy in three distinct layers:

1. Primitives: Grabbing a file out of a single archive (like unpacking a standalone ERF or BIF file).
2. Composition: Treating related archive sets as a single “Module” (like grouping a .mod file with its matching _s.rim and _dlg.erf files so they load transparently together).
3. Game-wide: A GameVfs rooted at one install that owns each tier (chitin / BIFs, the Override/ directory, caller-pushed extra overrides, the single active CompositeModule, and a mounted save above all of them) and resolves resrefs through them in engine precedence order. See Resource System & Resolution for the full tier order and what the save tier does and does not shadow.

Because we want our extraction to perfectly mirror vanilla behavior, lookups are strictly case-insensitive, and loading precedence is explicitly designed to mirror how the original game works (so a file in the Override folder automatically beats a file buried in a BIF archive).

Strongly-Typed Data (rakata-generics)

When we parse a .utc Character file, rakata-formats just hands us a raw GFF tree of untyped labels and values. rakata-generics wraps those raw data blobs in strongly-typed Rust structs (Utc, Uti, Are, Git, Dlg, Ifo, and friends). This guarantees that if a developer needs to access a character’s “Strength” stat, they get a guaranteed u8 property rather than blindly guessing string handles inside a raw binary tree.

Typed Views and Raw GFF

These typed structs sit beside the raw Gff tree, not on top of it. They are projections, not replacements. You construct one with Uti::from_gff(&gff) and round-trip back with uti.to_gff(); the original Gff stays accessible the whole time.

The projection layer follows one load-bearing rule: model what’s enumerated; drop what isn’t. from_gff extracts the fields each typed view documents and silently ignores anything else; to_gff writes only those documented fields. There is intentionally no extra_fields: Vec<GffField> accumulator on Utc / Uti / Are / etc. that would round-trip unmodelled fields through the typed layer.

The reason is correctness. Unmodelled fields often depend semantically on neighbouring fields (a savegame’s animation state only makes sense at the exact moment of save; a toolset’s custom annotations describe a specific revision). If the typed view silently preserved them while a caller edited a modelled field, the output would be internally inconsistent. The staleness contract is real, but it belongs explicitly with whoever needs byte-exact preservation, not buried inside a layer whose only job is type-safe access to known fields.

This naturally splits into two audiences served by one storage layer:

• Tools (save editor, mod linter, format inspector) reach for the typed views. They want type-safe access to known fields and don’t care about unmodelled bytes.
• Engines or byte-fidelity workflows (a future engine shim, a roundtrip auditor, anyone preserving toolset annotations) work directly with the raw Gff from rakata-formats. They own the staleness contract explicitly.

What gets enumerated: the engine has to read it

“Model what’s enumerated” only helps once you know what earns a place in the enumeration. The criterion is narrow on purpose:

A typed view models a field if and only if the engine reads it at that GFF path.

A field the engine never reads is a dead field. It gets recorded as dead where a reader will meet it, with the evidence, and it gets diagnosed by a Phase 1 lint rule walking the raw Gff. It does not go in the typed view. That is not a hole in the projection. crates/rakata-lint/ARCHITECTURE.md’s first design principle already puts schema and intra-resource checks on the raw tree for exactly this reason: they exist to validate the fields typed views drop.

Three clarifications carry most of the weight:

• “At that path” is the whole test. A door blueprint’s Tag is read; the Tag on a door placement in a GIT is not, because the engine takes a templated door’s tag from the blueprint. Same label, same concept, one live copy and one dead one. A concept being live somewhere else says nothing about this copy.
• Prevalence is not a criterion. WaypointList[].TemplateResRef appears in every waypoint in a retail install and no load path reads it. The corpus tells you where to look, never what to model.
• “Path” means the GFF path, not a decode-time variant. The rule stops at the structural position and does not reach inside a decoded enum. PropertyList[].Subtype is read at its path, so it is modelled, even though which property kinds actually consume it varies; decoded/uti.rs keeping subtype_id on the kinds that ignore it is shape parity across variants, and correct. The question to ask is whether there is any configuration in which the engine reads this label at this structural position. For Subtype, yes. For WaypointList[].TemplateResRef, never.

The point of the lint rule is that a dead field is a modder trap: someone sets it, expects an effect, and gets nothing. Silence is the worst answer, and the typed view is the wrong place to break that silence.

Ifo.Mod_Hak, Utc.SaveWill and Utc.SaveFortitude are modelled and documented dead. They predate this rule and round-trip harmlessly, so they stay. Treat them as legacy exceptions rather than precedent; the next field’s case has to stand on the criterion above rather than on their existence.

The one place where projection meets enumeration-by-design is rakata_generics::decoded::DecodedProperty. UTI item properties carry a PropertyName that indexes into itempropdef.2da, a table mods can extend with new rows. The enum has an Unknown variant that preserves the raw fields for one entry within an enumerated list, so an unrecognized property kind still surfaces through the decode pass instead of being dropped. It is a per-entry catch-all, not a struct-level accumulator, and the staleness risk is low because property entries are independent records.

When in doubt: if you need byte-exact preservation across a parse-then-write cycle, work with the raw Gff. If you need ergonomic, type-safe access to the fields Rakata has audited, work with the typed view.

Decoded Views: Projection and Resolution

The typed structs (Uti, Utc, etc.) bring file-native fields into Rust types. A second layer on top, the decoded view, resolves those file-native fields against external context. For UTI, that context is the 2DA tables the engine consults at item-property evaluation time: itempropdef.2da for property-kind dispatch, baseitems.2da for combat / equip metadata, the iprp_* cost tables for magnitude resolution.

A decoded view splits into two stages so cross-scope analysis is a first-class operation:

• Uti::project(itempropdef) -> UtiProjection<'_>. File-native typed-variant dispatch. Cheap, scope-free, takes only the minimal context (the property-kind dispatch table) needed to pick variants. The projection is the intermediate from which one or many resolutions are built.
• UtiProjection::resolve(&mut impl TwoDaSource) -> UtiResolved<'_>. Resolves the projection against a full per-scope context. Loads every table the resolved view’s query methods could need and caches the values. All query methods on the resolved view are &self borrow-free reads against that cache.
• Uti::resolve(&mut impl TwoDaSource) -> UtiResolved<'_>. Single-scope shortcut for project(...).resolve(...). Most callers want this.

The split exists because tools, the linter, and a future engine shim want to ask “what does this UTI look like under condition X” without re-running the file-native dispatch step for each context. Mod conflict analysis (does this item resolve differently with mod A loaded?), vanilla-vs-modded diffs, and reading a resource as a mounted save sees it all reduce to “build one projection, resolve under several contexts, compare.” The projection step is shared across resolutions; only the per-scope resolution repeats.

A resolved view does not retain the cache borrow once constructed. To query under a different scope, call projection.resolve(&mut other_cache) again on the same projection. The typed-variant dispatch is not redone.

The cost-table magnitude resolution recipe each resolved UTI view bakes in is documented in the Cost-Table Magnitude Resolution subsection of the UTI engine audit: which iprp_costtable.2da index every typed property kind dispatches through, which column carries the magnitude, and which handlers bypass the dispatch chain entirely.

UTC follows the same shape with format-specific differences: Utc::project() takes no minimal context (UTC has no single dispatch table; typed list dispatch happens at resolve time against per-list 2DAs), while UtcProjection::resolve(&mut impl TwoDaSource) loads racialtypes.2da / appearance.2da / portraits.2da / soundset.2da / classes.2da / spells.2da and caches scalar-id resolutions, typed DecodedClass variants, and typed DecodedSpecialAbility variants. UtcResolved exposes the same &self borrow-free query surface (race_label, classes, total_level, is_force_user, is_droid, has_class, special_abilities, equipment, inventory, etc.). Any future generic that grows a decoded view follows the same two-stage rule.

High-Level Interaction (rakata-save & rakata-lint)

Finally, crates at the top of the stack use our extraction logic and strongly typed generic structs to actually do things. rakata-lint compares typed structs against vanilla constraints to catch modding errors, while rakata-save gracefully handles unpacking, editing, and re-compressing massive save-game directories without corrupting the player’s campaign!