17. The standard library at a glance

Hanki's libraries form four tiers along two distribution mechanisms. The mechanism settles a tier's other properties, because two questions that could be answered separately are deliberately made to coincide - is it part of the language? = is it baked into the compiler? - splitting the four tiers cleanly in two:

• Baked - compiled into the compiler binary (and the stdlib_baked.bin type-index) via include_str!. This is the language's own surface. → core, extra.
• Packages - resolved by the package manager, versioned with independent SemVer, and pure Hanki only. → contrib, universe.

A third, stricter line sits inside the baked half: the native @intrinsic seam is confined to core alone - neither extra nor any package may cross it. (The package manager has landed across all three tiers — transitive universe-dependency resolution, contrib-tier resolution from the toolchain's seeded source, and local path deps — with the resolve/add/publish CLI verbs and the effect-growth gate on top (§21); only the hosted registry remains unimplemented; this section fixes their model.)

The baked tiers (core, extra) are the language's own curated surface (core locked, extra evolving); their module names stay flat (no path prefix at import sites), and the directory their source lives in carries the tier. Because they compile into one artifact, core and extra share a single version - the toolchain/language version; the tier difference is only the bump policy on it (a breaking change is a major bump for core, a minor bump for extra - one notch looser - while additions are minor and fixes patch). The package tiers (contrib, universe) are not part of the language: they never enter the compiler binary or the type-index, carry their own independent SemVer (they are distributed separately, so versioning granularity tracks the mechanism), and the native seam stays closed to them by design (statically enforced - see The boundary, locked). universe is a provisional name for the decentralized tier; the final name is deferred.

Tier 1: core

Locked surface - a breaking change in any of these cascades into every program, so promotion/demotion takes a deliberate decision ticket. Opened by convention.

The index surface is int. Every collection length, byte/element offset, and count - length, get, slice/slice_from, find, split pieces, BytesReader.take!/position!/remaining!, the serializer count headers, each_with_index!'s index - returns and accepts the lowercase scripting-default int, not a fixed width. Unsuffixed literals and int loop counters therefore compose with no conversions (xs.get(i + 1), 0 seeds), and there is no platform cap on sizes. An index that is out of range, beyond i64, or an in-band sentinel behaves like any other out-of-range value: slices clamp, reads return None. Crossing to a wire width goes through the conversion bridge (length.to_i64().to_u64(), u32_value.to_int() - §3). The int representation has a small-int fast path: a magnitude within ±2^62 is held inline as a machine word (no allocation), and only a larger value boxes as a bignum, so hot index loops stay allocation-free. The split is never observable - a value's arithmetic, equality, ordering, hash, and rendering are identical whichever form it takes - and both execution tiers share the same boundary.

Planned for core (not yet implemented): float. (The numeric-tower trait hub landed as numeric, above.)

Tier 2: extra

Curated, foundational capability/utility modules - evolving and not locked like core. Import-gated (§14): in scope only after a file use/opens them, so a file declares the capability vocabulary it pulls in. Qualified by default - write use net rather than open net; io is conventionally opened (for bare print!) but still requires the import like every other extra module.

Planned: regex, log, test. Each new module's tier is decided at module-creation time and recorded by its directory placement.

Serialization. Structured-data interchange is CBOR (RFC 8949), with CDDL (RFC 8610) for schemas, as the interoperability format; JSON (RFC 8259, the json module) is supported for compatibility with existing ecosystems, not as the primary format. The two are deliberately different shapes: CBOR is type-driven (@derive(Encode, Decode) gives any type a codec through the Serializer/Deserializer framework), whereas JSON is a standalone JsonValue tree you parse! and then pattern-match, walk with accessors, or decode into / encode from your own types through hand-written FromJson / ToJson impls - its arbitrary key order, single number type, and text escapes don't fit the positional framework, so it is intentionally not auto-derived. Types stay the single schema source: a type's @derive(Encode, Decode) fixes its CBOR encoding, and hanki cddl <file> emits the RFC 8610 CDDL describing that encoding - a publishable contract for non-Hanki consumers (a struct → a positional array, a sum → a choice of [tag, …] arrays, List/Option/Map → [* T] / (T) / null / {* K => V}). Using Hanki's own value syntax as a serialization format ("Hanki-source-as-data") was considered and declined for wire use: it serves only Hanki-to-Hanki exchange and would force map and tuple literals into the language - the Hanki-native human-readable need is met by the configuration DSL (§21) instead. Binary payloads and codecs traffic in the bytes core type (§4).

Tier 3: contrib

Globally useful but not foundational and not part of the language - a TUI library, a money-handling library, and the like. Curated at a lower bar than extra and distributed with the toolchain, but realized as a bundled package, not baked code: a contrib library is an ordinary package the manager resolves from a pre-seeded local source shipped in the toolchain directory, so it never bloats the compiler binary or stdlib_baked.bin. Versioned with independent SemVer per library. Because the set is curated, contrib libraries earn short flat names (use money) - the curation is what authorizes the name. Pure Hanki only: like every package, a contrib library carries no @intrinsic and builds only on the effect faces core / extra expose.

Contrib resolution has landed, with no bespoke loader: it is the package manager's local-source path. A deps entry with no URL binding is contrib (§21), and the resolver maps it to the pre-seeded contrib/<name>/ source shipped beside the hanki executable (in a dev tree, the repo-root contrib/; the root is threaded, so tests point it at a scratch dir). The toolchain bundles exactly one version per contrib package, so resolution checks the declared constraint against that shipped version and errors naming it when unsatisfied; the lockfile pins the reserved contrib:<name>@<version> identity with a deterministic fnv1a: tree hash over the seeded source (it has no git), and --frozen hash-verifies that tree — never fetches — since the tier ships with the toolchain. Two packages EXIST in-tree and are now resolvable - contrib/sql/, an sqlx-flavored SQL toolkit over extra/sqlite, and contrib/tui/, a ratatui-flavored immediate-mode terminal UI toolkit over extra/terminal. The curation/inclusion policy stays open.

Tier 4: universe

The decentralized long tail: libraries spread across git repositories worldwide, over which the Hanki project has no governance say. A universe package's identity is its git URL (host/user/repo), so namespacing is inherent and name-squatting is structurally impossible - there is no flat global namespace to grab. The project offers at most a discovery index (a searchable directory of opt-in-listed repositories), never an identity or gatekeeping authority. For ergonomics a project may bind a project-local alias to a package's git URL in its own manifest (http = "git.sr.ht/~user/http"), so its source reads use http without minting a global short name; the alias is scoped to that project, keeping the global namespace alias-free and squatting-free. That one binding does double duty: it is both the use http import alias and the dependency's URL source — a deps entry names it by short name (Dep(source = "http", …)), and the presence of the binding is what marks that dep as universe rather than contrib (§21). Independent SemVer; pure Hanki only, exactly as contrib. universe is a provisional working name (final name deferred); unlike the curated tiers it is not a governed set, only "everything else, discoverable."

The boundary, locked

The decision locks these commitments - the parts a future change must not quietly break (the descriptive table above is not itself the contract):

1. The native seam stays core-only. No package - curated contrib or open universe - may add native code, so the entire trusted native surface remains exactly the @intrinsic defs in core even with a full third-party ecosystem. *(Enforced at check time: an @intrinsic declared outside the core stdlib - whether in extra or in user/package code - is a compile error at the def site, not a deferred runtime failure.)*
2. Resolution and trust descend by tier: baked core / extra (never re-fetched) → bundled contrib → git-fetched universe. The manager will resolve in that order, with trust diminishing along it.
3. Universe identity is its git URL, so name-squatting is structurally impossible and the index stays discovery-only; short flat names are reserved for the curated tiers (core, extra, contrib), and universe is aliased only project-locally.

Mechanism - directory hierarchy

A module's tier is derived from the directory its source lives in; there is no per-file annotation.

Compiler-enforced invariant: a core stdlib module cannot use or open an extra stdlib module. A violation surfaces as a check-phase diagnostic naming both modules and their tiers. The rule is internal to the stdlib - user modules are untiered and can freely use or open either tier.

Native-seam rule (enforced). All native code is confined to core: an @intrinsic def is allowed only in a core module - never in extra, and never in user or package code (a check-phase diagnostic at the def site in each case). The OS-capability primitives live in one core module, sys; the extra capabilities (io, fs, process) are pure-Hanki faces that declare their effect and delegate to a sys primitive - io.print! calls sys.stdout_write!, fs.read! calls sys.file_read!. So the entire trusted native surface is exactly the @intrinsic defs in core, the same closure The boundary, locked (invariant 1) holds against user and package code. (module keeps its own intrinsics as core substrate; it is core, not a face.)

Promoting an extra module to core (or demoting a core module to extra) takes a fresh, explicitly recorded decision that quotes the existing carve-up.

Combinators and method-style surface

Pure generic combinators have short names (map, filter, fold); effectful variants get ! suffix (map!, filter!, fold!). The effectful variant accepts a callback with any effects; those effects bubble up to the call site.

Generic containers (Option<T>, List<T>) and the built-in string expose their stdlib helpers as inherent methods, not free functions. Call sites use receiver-first method-chain form:

open list
open option

def surface(xs: List<i32>, x: i32) -> ()
  Some(7i32).unwrap_or(0i32)
  xs.append(x).length
  "foo".concat("bar")
  ()
end

The associated function for empty containers also lives on the type: List<i32>.empty().

The sys terminal seam. Low-level terminal control lives on the same native seam: sys.stdin_is_tty!() / sys.stdout_is_tty!() (-> bool [io], false when piped or redirected), sys.term_size!() -> Result<TermSize, TermError> [io] (TermSize { columns, rows }, int fields per the index-surface convention; NotATty when stdout is not a terminal), sys.term_set_raw!(enabled) -> Result<(), TermError> [io] (character-at-a-time, no echo; idempotent in both directions; the FIRST enable saves the original terminal state), and sys.term_restore_write!(s) [io] (register bytes written to stdout at process exit - single slot, overwrite semantics, empty clears; a full-screen face registers alt-screen-leave + cursor-show here). Its input sibling is sys.stdin_read!(max: int, timeout_ms: i32) -> StdinRead [io] - the timed raw read behind a terminal event loop ("give me input bytes or time out"): Bytes(bytes) / TimedOut / Eof / Failed(message), one OS read of up to max bytes (clamped >= 1, no accumulation), timeout_ms < 0 blocking indefinitely and 0 polling immediately; it reads the raw descriptor directly (no buffering layer, so interleaving with stdin_read_line! cannot swallow bytes), a parked read is cancellable by actor.shutdown! (S15), and the timeout is real wall time outside the --deterministic virtual clock (terminal input is inherently non-replayable, like the read itself). The runtime guarantees the exit restore: the first raw-mode save (or restore-bytes registration) installs a C-runtime exit hook that puts the saved terminal state back and writes the registered bytes on every exit path of both tiers - a normal main! return and the fatal 254/253/252 exits alike - so a crashed program never strands the user's terminal. TermError is NotATty / Other(message). The ergonomic escape-sequence/key-input face over this seam is a staged extra library.

How stdlib functions are implemented. Most stdlib items are written in Hanki itself; their sources are embedded into the compiler at build time. Items that must bottom out in host code - sys.stdout_write!, string.concat, Display<i32>::to_string, and the like - are declared (in a core module only; see the native-seam rule above) with the @intrinsic annotation:

@intrinsic def stdout_write!(s: string) -> () [io]
end

@intrinsic is a top-of-item attribute (sibling of @derive). The compiler ignores the def's body and emits a single host-call thunk that dispatches by name into a runtime-registered Rust closure. @intrinsic is reserved for the stdlib - and within the stdlib, for core only - user code cannot declare its own intrinsics; the call site looks like any other Hanki call. The signature is the entire contract, so the body may be empty.

io.read_line! returns a sum, not a bare string. Empty input on a still-open stream is a legitimate value (the user pressed Enter) and must be distinguishable from end-of-stream (Ctrl+D, closed pipe). To keep both cases first-class, read_line! returns io.Line:

type Line
  Line(string)
  Eof
end

@intrinsic def read_line!() -> Line [io]

Read loops pattern-match and break on Eof. With use io the call site and variants stay qualified; with open io they read bare:

use io

def handle!(line: string) -> () [io]
  io.print!(line)
end

def repl!() -> () [io]
  loop
    io.print!("> ")
    match io.read_line!()
      io.Eof        -> break
      io.Line(line) -> handle!(line)
    end
  end
end

open io

def handle!(line: string) -> () [io]
  print!("got: #{line}\n")
end

def repl!() -> () [io]
  loop  print!("> ")
    match read_line!()
      Eof        -> break
      Line(line) -> handle!(line)
    end
  end
end

The dedicated sum is favoured over Option<string> for self-documenting variant names (Eof is clearer than None for the EOF case) and because it leaves room for future variants (Interrupted, Closed) without breaking the API. Reading past Eof is the caller's responsibility - the scheduler keeps returning Eof, but a sensible loop breaks on the first one.

Compiler reflection (experimental)

The compiler core module exposes the front end's outputs to a running Hanki program, so the off-path developer tools (fmt, doc, lint, effects, …) can be written in Hanki over a faithful parse/check model instead of in Rust. It runs the in-process compiler over a source string and marshals the result into ordinary Hanki records:

def api_version() -> int                                 # reflection schema revision (13)
@intrinsic def parse(src: string) -> Result<Ast, ParseError>              # shallow item headers
@intrinsic def type_decls(src: string) -> Result<List<Decl>, ParseError>  # deep structured decls
@intrinsic def imports(src: string) -> Result<List<Import>, ParseError>   # use/open directives
@intrinsic def diagnostics(src: string) -> List<Diagnostic>
@intrinsic def effect_surface(src: string) -> EffectSurface
@intrinsic def doc_items(src: string) -> List<DocItem>
@intrinsic def stdlib_modules() -> List<StdModule>       # the embedded stdlib (name, source) set
@intrinsic def bundled_card() -> string                  # embedded HANKI-CARD.md
@intrinsic def bundled_reference() -> string             # embedded HANKI.md
@intrinsic def bundled_skills() -> List<BundledSkill>    # embedded agent skills (name, summary, body)

• parse is shallow: Ast is the list of top-level item headers - Item { kind, name, doc: Option<string>, exported, action, span }, where kind is "import" / "fn" / "act" / "struct" / "type" / "trait" / "impl" / "actor" / "effect" / "provide" / "meta" / "test". Expression and pattern trees are deliberately not mirrored - the deep syntactic AST is the future pure-Hanki parser's job, not this seam's.
• type_decls is the deep counterpart to parse: the type- and signature-bearing declarations as List<Decl> (import / provide / test items omitted), each Decl { name, exported, action, generics, doc, body, span }. The body is a sum over the declaration kinds - Fn(Sig) (a def / def!, the ! shown on action), Struct, Sum, Trait, Impl, Actor, Effect, Const - carrying that kind's shape: a struct's fields, a sum's variants (each with its 0-based tag), a Sig { params, return_type, effects }, an actor's state_fields, handlers and has_migrate, and so on. A member row (a trait/impl method, actor handler, or effect op) is a Method { name, action, prop, generics, sig, span }, whose generics are the member's own type parameters - the container's live on the enclosing Decl. Every type position is a reflected Type tree mirroring the parser verbatim - Name(string) (a primitive, nominal type, or generic parameter alike), App(string, List<Type>) (List<T>, Map<K, V>, a user generic), Unit, Fn(List<Type>, Type, List<Effect>), or Infer. An effect row - a Sig's effects and a function type's Fn row alike - is a List<Effect> in source order, each Simple(string) (a bare tag io / fs / Crash / a user effect, or - when its name is a single lowercase letter e / r - an effect variable, recognised by that rule as an implicit single-uppercase type parameter is) or Throws(Type) (a throws E clause with its error type). Like parse it is purely syntactic (no inference, and expression bodies are still not mirrored); it backs the type-driven ports (cddl, api-diff). Full record shapes live in stdlib/core/compiler.hk.
• diagnostics returns what hanki check reports - Diagnostic { message, code, span }, with code the stable H#### string (a parse failure surfaces as a single diagnostic). effect_surface returns the capability surface (§6) as EffectSurface { effects: List<string>, sites: List<EffectSite> }, backing hanki effects. doc_items returns hanki doc's documentable symbols as DocItem { name, qualified, kind, anchor, signature, effects: List<string>, doc: Option<string>, has_doctest, heading, level, line }, where anchor is the stable cross-reference slug matching the hanki doc Markdown {#anchor} and --format=json anchor, heading is the rendered Markdown section heading (the bare name, or a head like impl Display<Point>), level is its heading depth (2 for a top-level item, 3 for a member), and line is the 1-based source line of the declaration (its name token, for file:line diagnostics like hanki doc --check). diag_codes takes no source and returns the whole stable-diagnostic registry as DiagCodeInfo { code, title, explanation } in registry order - the hanki explain data as values, so an agent can turn a Diagnostic.code into its explanation and fix recipe without shelling out. bundled_card and bundled_reference likewise take no source and return the compiler-embedded HANKI-CARD.md / HANKI.md as strings - the docs hanki new scaffolds into a project - so a generated project's own agent can read the canonical reference at runtime, and the Hanki hanki new port reads them from the same single embed. bundled_skills completes that embedded set: the agent-agnostic skills as BundledSkill { name, summary, body } records, the source of truth hanki new writes into a project's skills/ (and indexes from AGENTS.md), so the Hanki port ships the same set as the Rust command without a second copy to drift. These typed/effect/doc views reuse the compiler's own analysis and stay this seam's lasting surface even after syntactic tools migrate to the Hanki parser.
• imports returns the source's use/open directives - Import { path: List<string>, is_open, alias: Option<string>, span } - the one header detail parse's Item drops, collapsing an import to its dotted path. is_open distinguishes open M (folds M's names into bare scope) from use M (reached as M.name); alias is the use M as N qualifier. stdlib_modules takes no source and returns the embedded standard library as StdModule { name, source } pairs in load order. Together they let a Hanki tool reconstruct name qualification - which bare names an open pulls in - without re-deriving it: walk each stdlib module's type_decls for its exported names, then fold the ones the user file opens. This backs the api-diff port's open-vs-qualified normalization.
• A Span { file_id, start, stop } is a half-open [start, stop) byte range into the file file_id identifies. The reflected source is file_id 1, so its offsets slice the src you passed; a diagnostics span may instead point at a stdlib definition site (a different file_id, indexing that file).

A reflected Decl reports is_meta for a top-level meta def (the field is is_meta, not meta, which is a keyword), since the modifier shows in the rendered signature. A meta constant's reflected body carries its value twice: value_source is the verbatim slice, and value_rendered is the same expression printed back from the parse. A comparison wants the rendered one, since two spellings differing only in layout render identically and a reformat is not a value change; a tool showing the user what they wrote wants the slice. Experimental. The surface is destined for the 1.0 contract but stays unstable until the tool ports validate its shape - treat the records as provisional, and check api_version() for a revision change. The intrinsics run on both tiers: on the bytecode tier (hanki run / test / REPL) directly, and in an AOT-compiled binary that calls one by linking the embedded front end - the same statically-linked compile-pipeline + VM a load!-using build links (§14) - so a reflected value is byte-identical across tiers. The cost is paid only where used: a hanki build of a program that calls no reflection intrinsic keeps the lean, LLVM-free AOT baseline and pays nothing, while one that does pulls the front end into the binary's size/trust surface (the same tradeoff load! makes). Comptime reflection is a follow-up.