Spec M44 -- packages: source distribution, Go-style, from a registry of sources
Owner's request (2026-09-04, translated from the Portuguese): "a package manager for mc, in the
style of Go: it provides the source, and the manager is the repository of registered sources."
Read as three
requirements: (1) a package is SOURCE, fetched as source and compiled by mc like everything
else; (2) the model is Go's -- a path-identified source tree, versioned by git tags, minimums in
the manifest, a lock that pins content; (3) the manager is a REGISTRY OF SOURCES -- an index that
maps a name to where the source is and which versions exist, never a host of binaries.
Goal: [deps] in mc.toml, mc.lock beside it, mc pkg as a sixth bundled part, a registry
that is one git repository of TOML files, and a build that never reaches the network. Sequencing:
after M41 (the parts and subcommand()), after M42's fix batch if it lands first (no dependency
either way); M43 (the sandbox) is the milestone that makes an UNTRUSTED package's compiler module
safe to run and is named below as the seam, not as a prerequisite.
What already exists #
- The host layer has exactly what M25 needed and nothing more:
open/creat/read/write/close,mkdir,unlink,posix_spawnp/waitpidwith a stdout file action (src/host_macos.mc:35-45,src/driver.mc:220-277),host_home(),host_downloader()/host_downloader_alt(). Noopendir, norename, nogetenv, no HTTP, no TLS. Every design choice below that looks odd -- an explicit file list in the manifest, a hash over listed files, a manifest written after verification instead of an atomic rename -- follows from this line. mc sysroot fetchis already a package fetcher for one fixed package list (src/sysroot.mc:456-749): spawncurl --proto '=https' --proto-redir '=https' -fLsS(orwget),sha256fromsrc/sha256.mc:132over the bytes, size check, onetarspawn with--strip-components, every archive member probed, marker files probed, and only THENmanifest.toml(no date,docs/determinism.md).sysroot_download,sysroot_extract,sysroot_hexare the three functions M44 generalises;sysroot_extractis the only one bound to a row index ofsrc/sysroots.mc.- The pinned-row shape (
src/sysroots.mc:24-40:target host kind url sha size strip member) is a registry version row already.docs/reference/sysroot.md§ 8 mirrors it andscripts/check-sysroots.shdiffs the two with no network. [include].pathsis an anonymous root list (src/lex.mc:509-551): tried in order, only after the includer's own directory fails, "so a project never shadows a relative include that already resolved". Roots are stored with a trailing/and go throughpath_join/path_norm.#embedresolves through the same function.--include=DIRis the CLI form (src/cli.mc:230).#include <name>has no filesystem fallback, on purpose (src/lex.mc:619-626,docs/build.md§ M15: "<name>means the copy that shipped with this binary, and the answer must not depend on the working directory"). The bundle catalogue is 65 names + 2 regenerated (docs/reference/bundle.md), all reachable through the lexer's one pointerbopen_fn.toml_parsefills ONE global table (src/toml.mc:65,394-403); M25 Decision 3 put the sysroot rows in a.mctable for that reason, and M23 wrotelim_read_usage(src/limits.mc:252) as "its own tiny reader on purpose". M44 needs to read four kinds of TOML in one run (project, lock, registry entry, package manifest); a third tiny reader is the road not to take.mc.tomlis already rewritten in place bymc:lim_fix_write(src/limits.mc:515) replaces one key in one section and copies every other byte through. That is the precedent formc pkg addtouching[deps].mc buildis two processes (src/driver.mc:599-652): the taught compiler is generated from[compiler].core+[compiler].modulesand SPAWNED.drv_apply_configruns for the entry only (cfg = 1), never for the compiler (:315-316). A compiler-module package must reach the first half; a library package the second; so the[deps]roots apply to both and[libs]/[externs]keep applying to one.- Subcommands are registrations (
src/hooks.mc:773-830,MAXSUBCMD 16, three used) and a part registers its own (src/core_build.mc:34-47).src/core.mcis six include lines andsrc/main.mc:29-36names every part's*_init. A sixth part is one line in each. - A library and a compiler module are the same thing in two files:
lib/float.mc(registers types, intrinsics; a MODULE) +lib/float_rt.mc(a runtime a PROGRAM includes) + a six-linelib/user_float.mcthat defines the oneuser_init.examples/conc/mc.tomlstacks two module files from two directories (modules = ["../lang/lang.mc", "conc.mc"]) and pays for it with thelg_morechain, because a compiler holds exactly oneuser_init. - Versions are git tags,
vX.Y.Z, no pre-releases (docs/ci.md§ Versioning,scripts/next-version.shrejects0.2.0-rc1); the compiler carries no version string at all (grep -rn MC_VERSION src/ lib/is empty; the tags "are the only source of truth"). - Exit code 2 is "the environment is not ready" and is one message shape (
mc: no sysroot for .../tried:/run:),docs/reference/cli.md§ Exit codes,diagnostics.md§ 11. - The consumer. The
ngenport of teko (M41.5.md§ 1 in the review worktree) is written "againstdocs/against an unmodified compiler": a set of.mccompiler modules plus a runtime. It is the first package of the second kind, and it is exactly whatexamples/conc's../lang/lang.mcrelative path does today by hand.
Design #
1. Identity, names and versions #
A package is a source tree with an mc.toml at its root carrying a [package] table. Its
identity in Go is its import path (github.com/user/pkg, a URL prefix); in mc it is a
registry name, because the owner asked for the registry to BE the manager: the name is what
the index maps to a location. The location is a detail of the index row, not of the source that
uses the package.
Name rule: [a-z][a-z0-9_]*, at most 32 bytes. Why that set: it is a bare TOML key
(src/toml.mc: A-Za-z0-9_-, minus - and upper case so the name is also a valid identifier
prefix, geo_init), a valid path component on all three hosts, and a name that cannot collide
with the bundle's mc/... namespace. A name that the bundle serves is refused (float,
sys, i128, prelude, ...): the check is bopen_fn through the lexer, the same pointer
limits.mc uses, so <mc/core_build> never depends on <mc/core_bundle>. Reserved outright:
mc, deps, build.
Versions are semver X.Y.Z, one git tag vX.Y.Z per version (docs/ci.md § Versioning applied
to packages; no pre-releases, for the reason next-version.sh gives). [deps] geo = "1.2.0"
means at least 1.2.0 -- Go's require semantics, not an exact pin; the exact version is the
lock's business (§ 3). A requested version must exist in the index (Go requires the same).
2. The import spelling changes the language by zero lines #
A dependency geo is included as
#include "geo/geo.mc"
-- the quote form, unchanged. The lexer gains named roots: lex_add_named_root(name, dir),
consulted in lex_find_path_from AFTER the includer's own directory and BEFORE the anonymous
[include].paths roots: when rel begins with <name>/ and <name> is a registered root, the
rest is joined to that root's directory. M14's rule survives verbatim ("a project never shadows a
relative include that already resolved"), and the match is exact on the first component, so two
packages' files with the same basename cannot shadow each other -- the one failure mode a flat
deps/ root added to [include].paths would have.
<name> stays what docs/build.md promises: the copy that shipped with this binary, no
filesystem fallback. The boundary is one sentence: angle brackets are the bundle, quotes are
files on disk, and a dependency is files on disk. The bundle/package collision question
dissolves at the syntax level and is closed at the registry level by the reserved-name rule.
Go's analogue: the standard library is not a module; fmt and github.com/x/y share one import
syntax but the toolchain never fetches the former.
For a compiler module the same spelling goes into [compiler].modules:
[compiler]
modules = ["teach/mc_teach.mc", "user.mc"]
drv_gen_compiler writes #include "teach/mc_teach.mc" into the generated file (a value with no
leading / and a first component that is a named root is emitted as written, not ../-adjusted:
~6 lines in drv_include), and the named roots are registered for BOTH compilations
(drv_parse, before drv_apply_config's cfg test).
A package is closed. A file under a package's root may include or #embed only: its own
tree, the bundle (<...>), and the roots of the packages its lock row names as deps. A resolved
path that lands anywhere else -- the project's files, an absolute path, another package it did
not declare -- is geo/vec.mc:3: package geo reaches outside its tree: /etc/hosts, exit 1. ~35
lines in lex.mc (lex_root_of(path) = longest registered root that is a string prefix of the
normalised path; an edge list (from_root, to_root) in registration order). This is the M15
"no filesystem fallback, on purpose" stance applied to packages, and it is the cheapest honest
answer to #embed as an exfiltration primitive (§ Risks 1).
3. The lock, the tree hash, and what "deterministic" means here #
mc.lock, beside mc.toml, written only by mc pkg, rows sorted by name (bytewise, an insertion
sort on unique keys -- rule 2 of docs/determinism.md forbids qsort's tie-breaking, not
ordering; the sort key is total):
# written by `mc pkg sync` -- do not edit (docs/reference/packages.md)
[[package]]
name = "geo"
version = "1.2.0"
sha256 = "9f1c...e2" # tree hash, below
deps = ["mathx"]
[[package]]
name = "mathx"
version = "1.1.0"
sha256 = "41b0...7a"
deps = []
Not inside mc.toml: the lock is machine-written and complete, mc.toml is hand-written and
minimal, and lim_fix_write shows how much care a machine edit of a human file costs. Go keeps
go.sum beside go.mod for the same reason.
The tree hash is a dirhash-style content hash -- the shape of Go's dirhash.Hash1, but written
in plain hex, not Go's base64 h1: form -- in manifest order: for mc.toml first and then each entry of [package].files in the order written, one
line hex(sha256(file bytes)), two spaces, the path, \n; the hash is sha256 of those lines.
Go sorts the file list because it derives it from a zip; mc cannot list a directory, so the
package AUTHOR lists the files (the precedent is tools/bundle.list and the member column of a
sysroot row), and manifest order is the canonical order. The hash is of CONTENT, never of the
archive: GitHub's archive/refs/tags/*.tar.gz bytes changed under a git upgrade on 2023-01-30,
broke every archive-checksum consumer (Homebrew among them) and were rolled back -- a policy,
not a format guarantee. A content hash is also the same function for all three ways a tree can
arrive (fetched, vendored, replaced by a path), which is what lets mc pkg verify be one
routine.
The lock is checked, not trusted. mc build rehashes every locked package on every build
(sha256.mc already hashes the whole output executable per 4 KiB page on every --exe; a
dependency's source is smaller than that and is about to be lexed anyway) and refuses on any
disagreement:
| disagreement | message | exit |
|---|---|---|
| a file's bytes differ from the cache manifest's line | mc: geo 1.2.0: vec.mc does not match mc.lock | 2 |
the tree hash differs but no file line does (the files list changed) | mc: geo 1.2.0: mc.toml does not match mc.lock | 2 |
[deps] names a package the lock lacks, or asks a minimum above the lock | mc: mc.lock is stale: run mc pkg sync --yes | 2 |
| the lock names a version that is neither vendored nor cached | mc: geo 1.2.0 is not fetched + run: mc pkg sync --yes | 2 |
a file the build READ under a package root is not in that package's files | geo/extra.mc:1: not declared in geo's [package].files | 1 |
Exit 2 because every one of these is "the environment is not ready" in the M25 sense, and a
script must be able to tell it from "your program does not compile". The last row is the
post-parse walk of the lexer's once-only list (inc_at, src/lex.mc:557) and is what makes the
files list a real boundary rather than documentation.
Version selection is MVS, precisely Go's (cmd/go/internal/mvs, Russ Cox 2018): the build
list starts from the project's [deps]; for every selected (name, version) the requirements of
THAT version are added; a name's selected version is the maximum over every minimum that
mentions it; repeat to a fixed point. No SAT, no search, no "latest": the answer is a function of
the manifests alone, and the lock freezes it so that the index can move afterwards without
moving the build. The registry row carries each version's requirements inline (Go's proxy serves
.mod files before .zip for the same reason), so MVS runs over the index snapshot with no
archive download. Semver comparison is the --gt arithmetic of scripts/next-version.sh,
rewritten in .mc (~20 lines).
Majors. Go handles a major bump with semantic import versioning (/v2 in the path). mc does
not: if the selected version's major differs from the major of ANY requirement that named it,
the resolution is refused -- mc: mathx: 1.1.0 and 2.0.0 are different majors: no solver. Two
majors of one name in one build is the case MVS is not designed for, and pricing a /v2 scheme
is out of scope.
Yanked. A registry row may gain yanked = true (Go's retract, 1.16) and nothing else may
ever change in a row. mc pkg add|update skip yanked versions; sync of a lock that already
pins one warns and proceeds, so a build never breaks retroactively.
4. Fetching: the M25 road, generalised #
src/fetch.mc (in <mc/core_build>, included before sysroot.mc) takes the three functions out
of sysroot.mc in their general form: fetch_get(src, file) -- an https:// source spawns the
downloader exactly as today (--proto '=https' --proto-redir '=https' -fLsS, then the wget
fallback), and a source with no scheme is a local path copied with read_file/write_file;
fetch_extract(archive, dest, strip, members); hex64(digest). sysroot_extract becomes a
four-line call into it. The local-path branch is what makes the test suite need no network and
what prices a private registry at zero (§ 5).
Tarballs over git: tarballs. A GitHub tag has a URL
(https://github.com/<u>/<r>/archive/refs/tags/v1.2.0.tar.gz, top directory <r>-1.2.0, hence
strip = 1), tar -xzf reads it on all three hosts (M25 § 2 verified bsdtar, GNU tar and
Windows' libarchive tar.exe on gzip), and git stays off the dependency list -- the same
argument M25 made for curl over an HTTP client, one level up. The index row carries the url
explicitly, so a non-GitHub forge is a different URL and nothing else.
Cache: host_home()/.mc/pkg/<name>/<version>/ holds the extracted tree and
host_home()/.mc/pkg/<name>/<version>.toml (beside, not inside, so a package file named
manifest.toml cannot collide) holds [source] name version url sha256 and one [[file]] path
sha256 per hashed file. --pkg-dir DIR on mc pkg and mc build overrides the root, as
--sysroot-dir does, so CI depends on no HOME. Order of operations, unchanged from M25 because
there is no rename: download into <pkgdir>/<name>/<version>.tar.gz, extract into the final
directory, hash, and on any failure unlink every listed file (the sysroot_unbless idea) and
write no manifest; the manifest is the claim that the directory holds what the lock says, written
last. The tree is hashed and compared right after extraction, before anything else: the
archive itself is not checksummed (§ 3 says why), so unlike M25 the refusal comes after the
bytes are on disk -- but still before any manifest exists and before any build can consume the
tree, and a refused tree is unlinked.
Vendoring: mc pkg vendor copies each locked package (mc.toml + files) into deps/<name>/
in the project. When deps/<name>/ exists it is used and the cache is not consulted (Go: a
vendor/modules.txt consistent with go.mod makes -mod=vendor the default since 1.14). Both
forms are hashed against the lock, so the choice cannot change bytes silently. deps/ plus
mc.lock in git is the fully offline project.
Replacement: [replace] geo = "../geo" points a name at a local tree for development (Go's
replace directive). A replaced package is registered as a named root from that path, its lock
row says path = "../geo" and carries no hash, and mc pkg sync|list print
1 replaced dependency: not pinned by mc.lock. Go's go.sum also omits path-replaced modules.
mc build never downloads (M25, architect's addition (a)). It reads the lock, finds each tree
in deps/ or the cache, hashes, registers roots, compiles. Nothing else.
5. The registry: one git repository of TOML #
Compared: Go's proxy.golang.org (immutable zips served by a service) + sum.golang.org (a
transparency log of hashes) is the right shape and the wrong size for a one-owner compiler --
it needs a server, a log, and an operator. A Homebrew tap is a git repository of formula files
that anyone forks and PRs, and brew reads it as files. A tap of TOML is the choice, with
the sumdb's one property kept: a published row never changes.
Repository schivei/mc-registry:
README.md the three rules (name, immutability, how to register)
index/<name>.toml one file per package, added by PR
.github/workflows/check.yml `mc pkg check` on every changed index file; refuses a diff that
edits an existing [[versions]] row except to add `yanked = true`;
a scheduled job re-checks every row (link rot, like check-sysroots)
# index/geo.toml
[package]
name = "geo"
repo = "https://github.com/schivei/mc-geo"
description = "2-D vectors"
[[versions]]
version = "1.0.0"
url = "https://github.com/schivei/mc-geo/archive/refs/tags/v1.0.0.tar.gz"
strip = 1
sha256 = "<tree hash>"
deps = []
[[versions]]
version = "1.2.0"
url = "https://github.com/schivei/mc-geo/archive/refs/tags/v1.2.0.tar.gz"
strip = 1
sha256 = "<tree hash>"
deps = ["mathx 1.1.0"]
mc reads the index one file at a time: <registry>/index/<name>.toml, where <registry> is
https://raw.githubusercontent.com/schivei/mc-registry/main by default (PKG_REGISTRY in
src/pkg.mc), [registry].url in mc.toml, or --registry URL|DIR. An https:// registry is
fetched with fetch_get into <pkgdir>/index/<name>.toml (the snapshot mc pkg reads offline);
a DIR registry is read in place. So: a private registry is a directory or any URL with the
same layout -- a git clone of a private tap and [registry] url = "/path/to/it" -- and
costs no code. That is the price of "private registries": zero, and no access control beyond the
repository's own.
Who can register: anyone, by pull request, which is Go's "anyone with a public repository"
translated to a registry that has a human in it. The owner (CODEOWNERS on index/) reviews; the
CI check is the gate that matters: it downloads each new row with --yes, recomputes the tree
hash, and refuses a row whose hash, [package].name or deps do not match the archive's own
mc.toml, a name outside the rule, a name the bundle serves, or a repo change on an existing
name without the owner's review. Name squatting is a policy, not a mechanism: first PR wins the
name, transfers go through the owner. Registration is versioned in git, so the history of every
row is the audit log the sumdb would have provided.
6. Two kinds of package, and both #
The manifest does not enforce a kind; it names entries:
# the package's own mc.toml, at the repository root
[package]
name = "geo"
files = ["geo.mc", "vec.mc", "mc_geo.mc"]
lib = "geo.mc" # optional: the file a PROGRAM includes
module = "mc_geo.mc" # optional: the file a COMPILER includes; it provides geo_init()
[deps]
mathx = "1.1.0"
- A library is
#include "geo/geo.mc"from a program; it lands through the named root the driver registers for the entry compilation (what[include].pathsdoes today, named). - A compiler module is
modules = ["geo/mc_geo.mc", "user.mc"]; it lands in the generated compiler source. The rule a package must follow, stated once inpackages.md: a package never definesuser_init. It exports<name>_init()and the project's own module calls it --lib/user_float.mcis the six-line precedent, andexamples/conc'slg_morechain is what happens without the rule.mc buildgenerates nothing here: the project writes the six lines. (D8 prices the alternative.) - Both is
<float>:float.mcteaches the compiler,float_rt.mcis included by programs. The teko package is this shape --ngen/modules plus a runtime -- and needs nothing beyond two entries in one manifest.
lib/module are informational (mc pkg list and the registry page print them); what a file
does is decided by who includes it, as it is for every .mc file today.
7. Commands, and where they live #
A new part, <mc/core_pkg> (src/core_pkg.mc), that includes <mc/core_build> (once-only) and
src/pkg.mc, and registers one subcommand. The M41 debloat argument applies twice:
- The READ side --
[deps]/[replace]parsing, the lock reader, the tree hash, root registration, the build-time refusals -- issrc/deps.mcinside<mc/core_build>. A compiler withmc buildand withoutmc pkgstill builds a project from its lock and itsdeps/tree. That is the CI/consumer shape and it matches "mc build never downloads". - The WRITE and network side -- index fetch, MVS, lock writing, archive fetch, vendor copy,
[deps]editing,hash,check-- issrc/pkg.mcinside<mc/core_pkg>. A recreated compiler that will never resolve a dependency omits the part and thepkgusage line disappears with it (subcommand_usage).
mc pkg sync [DIR] [--config FILE] [--yes] [--registry URL|DIR] [--pkg-dir DIR]
mc pkg add NAME[@VERSION] [DIR] [--config FILE] [--yes] [--registry ...] [--pkg-dir DIR]
mc pkg update [NAME] [DIR] [--config FILE] [--yes] [--registry ...] [--pkg-dir DIR]
mc pkg list [DIR] [--config FILE] [--pkg-dir DIR]
mc pkg vendor [DIR] [--config FILE] [--pkg-dir DIR]
mc pkg verify [DIR] [--config FILE] [--pkg-dir DIR]
mc pkg hash DIR
mc pkg check INDEX.toml [--yes] [--pkg-dir DIR]
| command | Go analogue | what it does |
|---|---|---|
sync | go mod tidy + go mod download | reads [deps], fetches the index rows it needs, runs MVS, writes mc.lock (drops rows nothing requires), fetches missing trees. Without --yes: prints the plan -- each index file and archive it would download, with hash and destination -- and nothing was downloaded: re-run with --yes, exit 0 (M25 D10; mc has no isatty). With a DIR registry and everything cached it completes with no download |
add | go get pkg@v | writes NAME = "VERSION" into [deps] (newest non-yanked when no @) by the lim_fix_write method -- one key, one section, every other byte through -- then sync |
update | go get -u | raises the [deps] minimum(s) to the newest non-yanked index version, then sync |
list | go list -m all | one line per lock row: name version sha256[0..12] vendored|cache|path -- no absolute path, so it is a golden |
vendor | go mod vendor | copies each locked tree into deps/<name>/, then verify |
verify | go mod verify | rehashes every locked tree (vendored or cached) and checks lock vs [deps] consistency; exit 0 or 2 with the § 3 messages |
hash | (dirhash.Hash1) | prints the tree hash of a checkout: the author's tool and the registry CI's |
check | -- | the registry-side gate of § 5 |
mc build gains --pkg-dir DIR only. The single-file CLI gains nothing: a mc x.mc build has
no mc.toml and therefore no dependencies, and --include=DIR already reaches a vendored tree
by hand.
8. Publishing a package #
A package repository contains: mc.toml with [package] (and [deps] if any); the files it
lists; optionally a [project] for its own tests (a package that is also a program is fine; a
program's mc.toml without [package] is not a package). Cutting a version is git tag v1.2.0
plus git push --tags. Registering it is one PR to mc-registry adding a [[versions]] row
whose sha256 the author gets from mc pkg hash . on the tagged checkout; the registry CI
re-derives the hash from the tarball and refuses a mismatch, so a moved tag (§ Risks 4) is caught
before it is published, and a tag moved AFTER publication is caught by every consumer's fetch.
The bundle is the standard library and is not in the registry. <float>, <i128>, <sys>
are versioned with the compiler and need no fetch; they are what "the binary alone is the
toolchain" (M15) means. Their names are reserved in the registry. Promoting a registry package
into the bundle is a normal tools/bundle.list change, and demoting a bundled library to a
package is possible in principle (the [package] manifest describes lib/ as well) but is not
this milestone.
9. TOML: one parser, re-entrant #
toml_push()/toml_pop(frame) in src/toml.mc (~30 lines): save the eight table globals
(tm_ents tm_entcap tm_n tm_aot_name tm_aot_n tm_aotcap tm_naot tm_file) into an arena frame,
zero them, and restore. The lock, an index entry and a package manifest are each parsed inside a
push/pop; what is needed is copied into deps.mc's own flat tables (name, version, hash, deps in
source order -- rule 1) before the pop. Plus toml_occurrences(name) (~8 lines) so [[package]]
and [[versions]] rows can be counted. This replaces the third bespoke reader before it is
written; lim_read_usage stays as it is.
Out of scope #
- Binary artifacts of any kind: no prebuilt objects, no compiled compilers in the registry.
- A server. The registry is a git repository read as files; no API, no search service, no transparency log. The git history is the log.
- Dependency solving beyond MVS; two majors of one name in one build are refused, not
solved. Semantic import versioning (
/v2) unpriced. - Private registries as a feature: a directory or a URL with the index layout is one, at
zero lines; authentication is the transport's business (a private URL
curlcan reach with the user's own~/.netrc). - Git as a fetch transport,
git+https://sources inmc.toml, unregistered URL deps. A package is registered or replaced by a path; nothing in between. Priced as a follow-up: ~60 lines inpkg.mcfor a[deps.x] url = ...form, and a second spelling of the same list. - A minimum-
mc-version key. The compiler has no version string; a package that uses a hook the running compiler lacks fails ascall to unknown function(src/gen_resolve.mc:389), which is loud. Revisit when 1.0.0 givesmca version to check. - The sandbox itself (M43). M44 runs a package's compiler module exactly where a
[compiler].modulesfile runs today: inside the spawned taught compiler. - Any change to
stage0/. The lexer changes are insrc/lex.mconly;stage0/lex.ccannot see named roots and never has to (it compilessrc/mc.mc, which has no dependencies). - Directory listing,
rename,getenv: not added.
Files and estimated deltas #
| file | lines | what |
|---|---|---|
src/fetch.mc | ~120, new | fetch_get (https via downloader, else local copy), fetch_extract, hex64: the M25 three, general |
src/sysroot.mc | -75 / +12 | calls fetch_*; sysroot_extract reads its row and delegates |
src/deps.mc | ~340, new | name rule, [deps]/[replace]/[registry] reading, lock reader, tree hash over mc.toml + files, cache/vendor/path resolution, per-file verification, named-root and edge registration, the five refusals of § 3 |
src/pkg.mc | ~760, new | semver, index fetch/snapshot, MVS, lock writer (sorted), archive fetch + hash + manifest, vendor copy, [deps] line edit, hash, check, list, verify, pkg_cmd dispatch |
src/core_pkg.mc | ~22, new | includes core_build.mc + pkg.mc; mc_pkg_init() = one subcommand("pkg", &pkg_cmd, ...) |
src/core_build.mc | +2 | fetch.mc, deps.mc in the list |
src/core.mc, src/main.mc | +1 / +1 | the sixth part; mc_pkg_init() |
src/toml.mc | +38 | toml_push/toml_pop, toml_occurrences |
src/lex.mc | +65 | named roots table (grow, a T_NAMEDROOT tag), lex_add_named_root, the branch in lex_find_path_from, lex_root_of, the closure rule in lex_include and the #embed path, the post-parse files walk's accessor |
src/driver.mc | +48 | --pkg-dir; drv_apply_config split into drv_apply_deps (both halves) and the entry-only rest; drv_include keeps a named-root path verbatim; the post-parse files check |
src/cli.mc | 0 | |
tools/bundle.list, src/bundle_data.mc | +4, regenerated | mc/fetch, mc/deps, mc/pkg, mc/core_pkg |
tests/pkg/registry/index/{geo,mathx,teach,bad}.toml | fixture | rows with url = "@ARCHIVES@/geo-1.2.0.tar.gz" rewritten by the script; hashes hard-coded |
tests/pkg/src/{geo-1.0.0,geo-1.2.0,mathx-1.0.0,mathx-1.1.0,mathx-2.0.0,teach-1.0.0,bad-1.0.0}/ | fixture | each with mc.toml [package]; teach registers a syntax_stmt("unless", ...) as teach_init(); bad includes outside its tree |
tests/pkg/app/{mc.toml,obj.toml,main.mc,user.mc,mc.lock.expect,add.toml,add.toml.expect} | fixture | the consumer: [deps] geo = "1.2.0", mathx = "1.0.0", teach = "1.0.0"; main.mc uses unless and geo_dot |
tests/golden/pkg-list.txt | new | |
.gitattributes | +1 | tests/pkg/src/** -text so the fixture hashes hold on a Windows checkout |
scripts/check-pkg.sh | ~320, new | § Acceptance 1-17 |
scripts/check-parts.sh | +25 | case 1b for <mc/core_pkg>; the offline-consumer probe |
scripts/check-docs.sh | +1 | the pkg_/deps_ families and the --pkg-dir/--registry flags |
Makefile | +8 | check-pkg, inside check |
docs/reference/packages.md | ~420, new | everything above, every message, the manifest, the lock, the registry layout, the closure rule |
docs/guide/25-packages.md | ~240, new | using one; publishing one; the user.mc six lines; vendoring for CI |
docs/reference/toml.md | +80 | [deps], [replace], [registry], [package] |
docs/reference/cli.md | +45 | mc pkg, --pkg-dir, exit 2 rows |
docs/reference/diagnostics.md | +55 | § 13 packages |
docs/reference/bundle.md, docs/build.md, docs/determinism.md, docs/plan.md, docs/README.md, docs/guide/98-recreating-the-compiler.md | +15 / +30 / +12 / +1 row / +1 / +10 | the sixth part, reserved names, the lock as a determinism input |
schivei/mc-registry (separate repository) | README.md ~80, .github/workflows/check.yml ~70, index/ | seeded with the owner's first package; the teko port when it exists |
stage0/, lib/ | 0 |
Net new src/ lines ~1 350; the five goldens move once (the bundle grows and core.mc gains a
part), under M17 step A's protocol and M41's note 9.
Acceptance (ordered; scripts/check-pkg.sh build/mc1, no network anywhere) #
- The fixture registry is built by the script, not checked in as archives: each
tests/pkg/src/<name>-<v>/istar -czf'd into$tmp/archives/<name>-<v>.tar.gzwith the top directory<name>-<v>(strip = 1, GitHub's shape), andtests/pkg/registry/index/*.tomlis copied to$tmp/registry/index/with@ARCHIVES@replaced. gzip timestamps make the archives non-reproducible; nothing hashes them. - The hash is stable across hosts:
mc pkg hash tests/pkg/src/geo-1.2.0prints the value hard-coded in the fixture index, on macOS, Linux and Windows CI. - The plan is printed and nothing is fetched:
mc pkg sync tests/pkg/app --registry $tmp/registry --pkg-dir $tmp/c1lists three archives with hashes and destinations, printsnothing was downloaded: re-run with --yes, exits 0, and$tmp/c1holds no<version>.toml. - MVS, not "latest": the same with
--yeswritestests/pkg/app/mc.lockbyte-identical tomc.lock.expect--mathxat1.1.0(geo 1.2.0's minimum wins over the app's 1.0.0), never at the registered2.0.0, rows sorted; three<version>.tomlmanifests exist with[[file]]rows; runningsync --yesagain downloads nothing and rewrites the lock identically. - A two-package chain builds and runs:
mc build tests/pkg/app --pkg-dir $tmp/c1builds the taught compiler fromteach/mc_teach.mc+user.mc, compilesmain.mc(which includes"geo/geo.mc", whose own#include "mathx/mathx.mc"resolves through the closure), and the binary's stdout/exit matchmain.mc's header. - Byte-identical objects from two fetches: a second
sync --yesinto$tmp/c2and a build withobj.toml(kind = "obj") from each cache givecmp-identical objects; the two lock files are identical. - The lock refuses a tampered source: append one byte to
$tmp/c1/geo/1.2.0/vec.mc;mc buildexits 2 withgeo 1.2.0: vec.mc does not match mc.lock;mc pkg verifysays the same; restore, both exit 0. Then editfilesin the cachedmc.toml: exit 2 namingmc.toml. - Stale lock, stale tree: raise
[deps] geoto"1.9.0"in a copy of the config ->mc buildexits 2 withmc.lock is stale: run mc pkg sync --yes; delete$tmp/c1/mathx/-> exits 2 withmathx 1.1.0 is not fetchedand therun:line.mc buildmust not have spawnedcurlin either case (the script puts acurlshim onPATHthat fails if invoked). - Vendoring is the offline road:
mc pkg vendorpopulatestests/pkg/app/deps/{geo,mathx, teach}/;mc build --pkg-dir $tmp/emptysucceeds with no cache at all; its object iscmp-identical to step 6's;mc pkg listprintsvendoredin every row and matchestests/golden/pkg-list.txt. - A package is closed: add
bad = "1.0.0"to a copy of the config, sync, build -> exit 1 withbad/bad.mc:2: package bad reaches outside its tree: ...; a#embedof an absolute path insidebadis refused with the same words; a filegeo/extra.mcplanted in the cache and included by a planted line isnot declared in geo's [package].files. - Names:
[deps] float = "1.0.0"->float: a bundled name;Geo = ...->invalid package name; both at the key'sfile:line:col, exit 1. - Majors are refused, not solved: a fixture package requiring
mathx 2.0.0next to the app's1.x->mathx: 1.1.0 and 2.0.0 are different majors: no solver, exit 1, no lock written. - A failed fetch leaves no claim behind: an index row whose
urlnames a missing archive -> exit 2 with the M25-shaped message (mc: the download failedfor a URL,cannot openfor a path), no<version>.toml, and a followingmc buildsaysnot fetchedrather than reading debris. A row whosesha256is wrong ->checksum mismatch for geo 1.2.0, the listed files unlinked, no manifest. mc pkg addedits one line: onadd.toml(no[deps]),mc pkg add mathx@1.0.0 --yesproducesadd.toml.expectbyte for byte -- every comment and byte outside the new[deps]section untouched;add mathxwith no version picks1.1.0, never the registered-and-yanked1.2.1row the fixture carries.mc pkg checkis the registry gate: on$tmp/registry/index/geo.toml --yes --pkg-dir $tmp/chkexit 0; with one hash altered exit 2; withname = "float"exit 1.- Parts:
check-parts.shshows<mc/core_min>+<mc/core_pkg>compiles alone; a probe compiler assembled fromcore_min+core_machines+core_writers+core_build+core_bundle(nocore_pkg) builds the vendored app of step 9 and prints a usage with nopkgline; the measured table gains the+ <mc/core_pkg>row. - Inert:
scripts/check-inert.shclean for every object of a project without[deps];check-standalone,check-obj32/32,check-build21/21,check-sysrootsandtests/golden/sysroot-list.txtunchanged;mcwith no arguments prints today's usage plus exactly thepkglines;mc sysroot fetch linux-aarch64 --yes --sysroot-dir $tmp/s(the CI step, the only networked check, unchanged) still writes the four files -- proving thefetch.mcmove is behaviour-neutral. Goldens rewritten once, after an empty--dump-asmdiff betweenmc1andmc2andcmp build/mc2.o build/mc3.o.make check-docsgreen.
Risks #
- Supply chain: a compiler module is code that runs on the developer's machine at build
time. Go's
go buildexecutes no package code (no build scripts), which is a security property Go chose on purpose; mc's compiler modules are proc-macro-shaped and DO run, inside the spawned taught compiler, with the developer's file system. A library package is only slightly better:#embedreads any file the compiler can. M44's brakes are the closure rule (§ 2: a package reads its own tree, the bundle and its declared deps, nothing else, for#includeand#embedalike) and the lock (nothing runs that is not the reviewed bytes). What they do not stop is a module that opens a file byextern openatuser_inittime. That is M43's job and the seam is the spawn indrv_teach: the taught compiler is already a separate process, so a sandbox wraps oneposix_spawnp. Until M43 lands,packages.mdsays in its first paragraph that a compiler-module package is trusted code. - Name squatting and transfers are policy in a one-owner registry: first PR wins, the
owner reviews,
repochanges need the owner. Cheap now; revisit if the index passes a few hundred names. - Tarball regeneration (GitHub, 2023-01-30) does not move the tree hash; it would move an archive hash, which is why there is none.
- Tag mutability: a tag moved after registration fails every consumer's fetch with
checksum mismatch-- loud, and the right outcome. A tag moved BEFORE registration is a registry row that never matched, refused bymc pkg check. - Network in CI: none in
make check; the fixture registry is a directory. Only the registry repository's scheduled job and the existingmc sysroot fetchstep touch the network, and a dead URL is a maintenance issue there (M25 § Risks), not a red PR. toml_push/toml_popregressions: a pop forgotten on an error path leaves the project table swapped out. Every parse of a foreign file is wrapped in one function that pops before returning or dies (_exit, where the table no longer matters).- The
fileslist as the boundary: a package author who forgets a file ships a package that fails withnot declared in ... [package].fileson the first include -- loud, at the consumer.mc pkg checkcould cross the list againsttar -toutput (captured with thedrv_sdkfile-action trick) and warn; priced at ~40 lines, optional. - Line endings on Windows checkouts move every fixture hash;
.gitattributes-textontests/pkg/src/**is in the file table, and the Windowschecksubset runs step 2 first. mc pkg addediting a human file:lim_fix_write's method keeps every other byte, and step 14cmps the result; the failure mode is a[deps]table written twice in a file that already has it under an unusual spelling ([ deps ]), which the key scan does not recognise. Refuse when the scan finds no[deps]buttoml_get("deps.x")says one exists.- No version string in
mc: a package needing a hook from a newermcfails withcall to unknown functionrather than "needs mc >= 0.9". Acceptable until 1.0.0 (§ Out of scope). - Six parts and a sixth
*_init:main.mc's list grows;check-parts.shcase 1b is the regression net M41 built for exactly this. - Diagnostics in cached packages print absolute paths (
/Users/me/.mc/pkg/geo/1.2.0/vec.mc:3) while vendored ones printdeps/geo/vec.mc:3. Objects carry no path (rule 4, and there is noN_OSO), so determinism of OUTPUT holds -- step 6 and step 9 prove it -- but two machines' error texts differ. A display name (geo/vec.mc, the bundle's "errors point at the bundled name" precedent) is ~15 lines inlex_pushand a follow-up; M30's DWARF will want it too.
Decisions (architect) -- to ratify with the owner #
- D1 -- import spelling:
#include "geo/geo.mc"through a named root, not<geo/...>and not a new directive. Zero language change; the bundle's "no filesystem fallback" promise stays intact; collisions with the bundle become a registry rule instead of a lexer rule. The alternative -- extending<name>to packages -- reads well and breaks a documented promise. - D2 -- identity is a registry name, and
[deps] name = "min.version". The Go path (github.com/u/r) is the location; the owner asked for the registry to be the manager, so the name is the identity and the row is the location. A[deps.x] url = ...form for unregistered sources is out of scope, priced at ~60 lines. - D3 -- registry = one git repository of TOML, one file per package, PR to register, owner
review,
mc pkg checkas the CI gate, rows immutable exceptyanked. Not a proxy, not a sumdb: no operator, no server; the git history is the log. - D4 --
mc.lockbesidemc.toml, machine-written, rows sorted by name, one content hash per package, edges inline. Not insidemc.toml. - D5 -- the hash is Go's h1 over
mc.toml+[package].filesin manifest order, in hex; never the archive. The author lists the files becausemchas noopendir, and the list is also the vendor-copy list and the build-time boundary. - D6 -- MVS, precisely Go's; two majors in one graph are refused. No solver.
- D7 --
mc buildrehashes every dependency on every build and refuses any disagreement (exit 2). "Checked, not trusted." Cost is linear in dependency source size, which the lexer reads anyway. Alternative: trust the cache manifest and rehash only inverify-- faster, and it makes the tamper acceptance averify-only proof; recommend against. - D8 -- a package never defines
user_init; it exports<name>_init()and the project's own six-line module calls it. Alternative:mc buildgeneratesuser_initfrom a[package].initkey -- ~40 driver lines and a second place a compiler's init order is decided; recommend against until a second real consumer wants it. - D9 -- tarballs by URL,
tar -xzf, nogitdependency; GitHub'sarchive/refs/tagsas the documented default shape, any https tarball accepted. - D10 -- cache
~/.mc/pkg/<name>/<version>/+<version>.tomlbeside it;--pkg-dir DIRoverride onmc pkgandmc build;deps/is the vendor directory (owner's word; Go's isvendor/) and wins over the cache when present. - D11 --
[replace] name = "path"for development, unhashed and announced. Go'sreplace. - D12 --
mc pkgis a sixth part<mc/core_pkg>; the read side (deps.mc,fetch.mc) lives in<mc/core_build>so a compiler withoutcore_pkgstill builds from a lock anddeps/. - D13 --
sync|add|updaterequire--yesto download and print the plan otherwise (M25 D10, the same reason: noisatty, no prompt). - D14 --
toml_push/toml_popinsrc/toml.mcrather than a third bespoke reader; M25 D3 stands for its own case (sysroots.mcstays a.mctable). - D15 -- the bundle is the standard library, outside the registry; its names are reserved.
- D16 -- the closure rule is in M44, the sandbox is M43;
packages.mdstates the trust model plainly until M43 lands. - D17 -- no 1.0.0 on the back of this milestone (M42 D8): 1.0.0 waits for the roadmap and
for the teko/
ngenconsumer, which is also the first package of the second kind this spec should be validated against before it is called done.
Architect's additions: (a) step 8's curl shim -- mc build must be PROVED never to spawn a
downloader, not just documented; (b) the registry repository's README.md is written in the same
PR as packages.md, with the three rules and one worked registration; (c) docs/guide/25-packages.md
is written for the teko port's author and ends with the six-line user.mc.
Amendment (owner, 2026-09-04): angle brackets, ~/.mc/libs, the slim binary, install/update/upgrade #
The owner's two rulings, translated from the Portuguese:
(1) "For M44 I would change one thing: since these are external libraries referenced in the toml, they should be reachable as
#include <pack/lib.mc>. What that changes: even what we ship embedded today could become an unpacking into a directory~/.mc/libs/pack_name/v<version>/, and with that we gain even more extensibility."(2) "I would say we even gain a smaller binary: it could carry only the executable and require an
mc install, downloading even what is embedded today; anmc updateupdates the packages' versions; anmc upgradeself-updates the version and downloads the basic libraries for mc; and so on."
Read as four requirements that override D1, D2 and D15 and extend the scope: (a) a dependency is
spelled with angle brackets, #include <pack/lib.mc>, so there is ONE resolution model for "a
library that did not come from my own tree"; (b) the bundle's entries are a package like any other,
materialisable under ~/.mc/libs/<pack>/v<version>/; (c) a second, slimmer release flavour carries
no blob and gets its libraries through mc install; (d) three verbs, install, update,
upgrade, the last one a self-update. What follows rewrites the draft where the rulings touch it and
says, item by item, what does not move.
What survives untouched, and why. The lock (§ 3), the h1 tree hash over mc.toml +
[package].files, MVS with the two-majors refusal, the registry as one git repository of TOML
(§ 5), the closure rule (a package reads its own tree, the bundle and its declared deps, nothing
else), the sixth part <mc/core_pkg> with the read side in <mc/core_build> (§ 7), "mc build
never downloads", toml_push/toml_pop (§ 9), D3-D9, D11-D14, D16-D17. None of them depends on
HOW an include is spelled or WHERE a tree lives: they are about identity, content and selection.
The rulings change the spelling, the directory, and add a distribution channel for the compiler's own
libraries; they do not change what a package IS.
A. Angle brackets and one resolution model #
A1. The spelling. A dependency geo is included as
#include <geo/geo.mc> // a file of the package geo, at the version mc.lock pins
#include <geo> // the package's `lib` entry (mc.toml: lib = "geo.mc"), if it has one
#include <float> // unchanged: a name the binary ships
#include <mc/core> // unchanged
<name> stops meaning "the bundle" and starts meaning "a library that is not in my tree":
resolved from the lock, from the bundle, or from the installed copy of the compiler's own package --
in that order, below -- and never from the working directory. Quotes keep meaning "a file on disk
relative to me or to [include].paths" (src/lex.mc:536, lex_find_path_from, unchanged). The
draft's named roots for the quote form (§ 2, lex_add_named_root, the branch in
lex_find_path_from) are DROPPED: one spelling for one thing, which is what the owner asked for.
[compiler].modules takes the same spelling -- modules = ["<teach/mc_teach.mc>", "user.mc"] --
and drv_gen_compiler emits a value that starts with < verbatim, the rule src/driver.mc:417-425
already applies to [compiler].core (core = "<mc/core_min>", M41). Nothing in the language
changes except one token (A2).
A2. A fact the draft did not check: . is not a token. tok_init (src/lex.mc:246-294)
registers the keywords and the operator lexemes; there is no "." among them, and <name> is not
tokenised specially -- do_directive (src/parse.mc:1610-1626) reassembles the lexemes between
< and >, on purpose, so that --dump-tokens stays byte for byte what stage0/lex.c produces.
So #include <geo/geo.mc> fails today at the . with unexpected character (src/lex.mc:902),
and the owner's spelling needs one of two things: (i) tok_add(".", 1) appended at the END of
tok_init, after the last existing lexeme, so no existing id moves (K_U8..K_EXTERN stay 256..269
and every punct keeps its id), --dump-tokens is unchanged for every file that has no bare . --
which is every file check-lex compares, since float literals are consumed raw by syntax_lit
(lib/float.mc:343) before the punct scan and lib/float_rt.mc is already seed-skip -- and
examples/lang/lang.mc:44's own tok_add(".", 1) lands on the same id because tok_add is
idempotent (the comment two lines above it says so); or (ii) accept only <geo/geo> and append
.mc on disk, the bundle's own convention (lex_strip_mc, src/lex.mc:574, exists because bundle
names carry no .mc). Recommendation: (i), and lex_include_name strips a trailing .mc from
the reassembled name so <geo/geo.mc> and <geo/geo> are one name, exactly as <mc/core> and a
relative "core.mc" inside the bundle are today. Cost: one line in tok_init, one call in
lex_include_name. stage0/lex.c is untouched: it never sees a <...> with a dot (tests/mc/ is
where such files live, and that directory is already outside the mc0 cross-checks).
A3. The resolution order for <X> -- in lex_include_name (src/lex.mc:626), which today is
one call to bopen_fn (:498) and one error. It becomes three steps through TWO pointers, the
existing bopen_fn and a new lopen_fn registered by lex_set_libs from mc_build_init()
(src/core_build.mc:36), so the lexer still depends on nothing and lexdump/astdump keep
compiling with mc0:
| step | who answers | for which names | where the bytes come from |
|---|---|---|---|
| 1 | lopen_fn(X, 0) -- deps.mc's libs_open, the LOCK road | X's first path component is a package the lock names (geo, teach, or a bundled name pinned in [deps], A5) | deps/<pack>/<rest> if vendored, else <libs>/<pack>/v<version>/<rest>; <pack> alone is the lock row's lib entry |
| 2 | bopen_fn(X, 0) -- the bundle, unchanged (src/bundle.mc, bundle_open; <mc/host> still rewritten by host_bundle_open, src/core_bundle.mc) | every name in tools/bundle.list, plus the two synthetic ones | the blob |
| 3 | lopen_fn(X, 1) -- the INSTALLED mc package | the same names as step 2, when the binary carries no blob (B) | <libs>/mc/v<mc_version()>/ + the path bundle.list maps the name to |
| -- | neither | anything else | prog.mc:1: unknown bundled include: no/such/module, unchanged text |
<libs> is host_home()/.mc/libs (src/host_macos.mc, src/host_linux.mc, src/host_windows.mc:
the HOME=/USERPROFILE walk sysroot_cache_dir already uses, src/sysroot.mc:215), or
--libs-dir DIR on every command that reads it (the --sysroot-dir precedent), so CI depends on
no HOME. Step 1 exists only when a lock was read (mc build, both halves); the single-file CLI
(mc x.mc) has no lock and therefore no step 1: <geo/geo.mc> there is refused with the step-3
miss, and --include=DIR remains the hand road. Never the working directory (a project cannot
shadow <float> by dropping a float.mc next to main.mc; that is the M15 stance and it is what
lets the answer be a function of (binary, lock, libs content) alone), never an unpinned
latest (<libs>/geo/ may hold v1.0.0/ and v1.2.0/; only the lock says which one, and a
directory that no lock names is never opened).
A4. The mc package and the layout of ~/.mc/libs/mc/v<version>/. The bundle's 75 entries
are ONE package, named mc, at the compiler's own version (C). On disk it keeps the REPOSITORY
layout -- lib/float.mc, src/core.mc, src/host_macos.mc, tests/mc/bundle/embed_demo.txt --
with bundle.list (the NAME<TAB>PATH manifest, tools/bundle.list) at the root as the name map.
Not a by-name layout, because the bundle's relative-include fallback (bundle_find_base,
src/bundle.mc: mc/driver -> "../lib/prelude.mc" -> last component prelude) has no
equivalent on a filesystem; in the repository layout src/driver.mc's #include "../lib/prelude.mc"
resolves through lex_find_path as a plain relative path, and src/core.mc's "arena.mc" lands on
src/arena.mc. The once-only key for a name served from disk is the normalised absolute path
(lex_include records paths, lex_include_bundled records canonical names; both go through
lex_seen, src/lex.mc:557), so <mc/host>, <mc/host_macos> and core.mc's own
"host_macos.mc" coincide on disk exactly as they coincide in the blob. Two consequences worth
writing down: the installed src/bundle_data.mc is the mode 1 text (#embed bundle_blob
"bundle.bin" + the index, bundle_emit(..., 1), src/bundle.mc) next to a real src/bundle.bin
(the blob, bundle_read(BUNDLE_BIN)), so a taught compiler built from an on-disk <mc/core> pays
one N_BLOB node and not ~45 000 u64 nodes (M21.5's arena argument, docs/build.md § M15), and
both forms produce the same object (that is what check-standalone measures today); and
tools/bundle.list itself is bundled as mc/bundle.list (one line in the manifest; its bytes do
not depend on its own content, so unlike bundle_data there is no recursion; last component
bundle.list collides with nothing), so the FULL binary can write the whole package to disk from its
own blob with no network (B3). Diagnostics from a disk-served <mc/core> print the absolute path
where the bundle printed the name (draft Risk 12, unchanged and now more visible).
A5. Collisions: the lock wins, except for mc. A registry package MAY carry a bundled name
(float, sys, i128 -- the draft's reserved-name rule is withdrawn for lib/'s names) and a
project that pins [deps] float = "1.3.0" gets <float> and <float/float_rt.mc> from the locked
tree instead of the blob. That is the extensibility the ruling asks for, and it is safe on both
counts the task names: determinism -- step 1 answers only from a lock row, the row pins a
content hash, mc build rehashes on every build (D7), so two machines with the same binary, lock
and tree bytes resolve the same bytes; a project with no [deps] float line is byte for byte what
it was; offline -- a locked float that is neither vendored nor fetched is mc: float 1.3.0
is not fetched / run: mc pkg sync --yes, exit 2, the same as any dependency, and a project with no
lock never leaves the blob. What stays reserved, in the registry and in [deps]: mc, every
mc/... name, deps, build. The mc package can never be pinned by a lock: <mc/core> is the
compiler's own source, check-standalone is an equality between a binary and ITS bundle, and a
taught compiler assembled from a foreign mc/core would be a different compiler than the one
running mc build -- the exact confusion M37's <mc/host> was designed out. mc pkg check refuses
an index row named mc; deps.mc refuses [deps] mc = ... at the key (mc: reserved, exit 1).
A6. The closure rule, restated for the new spelling. A file under a package root may include
or #embed: its own tree (quotes, relative), <...> names that resolve to the bundle or to the
installed mc package, and <dep/...> where dep is in its lock row's deps (or itself). The
edge list and lex_root_of (draft § 2, ~35 lines) survive as they were; roots are registered from
the lock rows by drv_apply_deps, not from a [include]-like list. The message and the #embed
case are unchanged.
A7. Rewrites, itemised. § 2 is replaced by A1-A3 and A6 (named roots gone; the "angle brackets
are the bundle, quotes are files" sentence becomes "angle brackets are libraries, quotes are my
files"); § 4's cache paragraph becomes ~/.mc/libs/<pack>/v<version>/ + <pack>/v<version>.toml
beside it, --pkg-dir becomes --libs-dir everywhere (mc pkg, mc build, mc install,
mc upgrade); vendoring keeps deps/<pack>/ and still wins over <libs>; § 6's example becomes
#include <geo/geo.mc> and modules = ["<geo/mc_geo.mc>", "user.mc"], and lib stops being
informational -- it is the target of a bare <geo> and is copied into the lock row; § 8's last
paragraph ("the bundle is the standard library and is not in the registry") becomes: the bundle is
the mc package, versioned with the compiler, shipped inside the full binary and installable next
to the slim one; its lib/ names are not reserved -- a registry package may replace one, lock-driven
-- and its mc/... names are. The M15 promise in docs/build.md § M15 and docs/reference/bundle.md
becomes: "the full binary alone is the toolchain; the slim binary plus one mc install is the
same toolchain".
B. The slim binary and mc install #
B1. Two release flavours per target. mc (full: today's binary, blob embedded -- the bootstrap
SEED on Linux and Windows, scripts/bootstrap-linux.sh:18-26, and the offline/CI default) and
mc-slim (the same compiler with an EMPTY bundle). Weight, from M41's measured table
(scripts/check-parts.sh, CLAUDE.md § M41): the assembly WITHOUT <mc/core_bundle> is
395 820 B on disk and mc itself is 759 875 B; the bundle part adds ~3.7 KB of __text
and ~368 KB of __data (the blob: check-bundle at M41 reports raw 776 601 -> lz 364 543, blob
365 449 B). A slim binary keeps the ~4 KB of reader code and drops the blob, so it weighs
~400 KB, 52% of the full one. The task's "~252 KB" is M15's figure (build/mc-exe 252 316 B
without the blob, when the compiler was nine milestones smaller); it is not what a slim binary would
weigh today and the spec should not promise it.
B2. How the slim flavour is assembled -- zero core lines. It is docs/reference/bundle.md
§ "Your own bundle" applied to an empty manifest: src/bundle_empty.mc (checked in, ~8 lines:
u64 bundle_blob[1]; i64 bundle_idx[BI_N]; #define BUNDLE_COUNT 0; bundle_end() already returns 0
for a zero count and bundle_cache is sized BUNDLE_COUNT + 2), src/core_bundle_slim.mc
(bundle_empty.mc + bundle.mc + bundle_glue.mc), src/bundle_glue.mc (the host_bundle_open
mc_bundle_initpair moved out ofsrc/core_bundle.mc, which then includes it too), andsrc/core_slim.mc= the five parts withcore_bundle_slim.mcin place ofcore_bundle.mc, plusmain.mc. Five entries of three lines each (src/mc_slim.mc,mc_linux_slim.mc,mc_linux_x86_64_slim.mc,mc_windows_slim.mc,mc_windows_x86_64_slim.mc: host file +core_slim.mc+user.mc, thesrc/mc_linux.mcshape) and four*-slim-obj.tomlconfigs (src/mc.linux-aarch64-obj.toml's shape,entrychanged) giverelease.yml's cross-compile steps the objects the Linux and Windows runners link with the samescripts/link-linux.sh/link-windows.sh.check-parts.sh's per-part case covers<mc/core_bundle_slim>for free. Themain()is the same file:mc_bundle_init()registers an opener that always misses.
B3. What a slim binary does BEFORE mc install. Everything that needs no <...>: compile a
program with no angle-bracket include (i64 main() { return 42; }, or a program with externs and
quote includes of its own tree); every --dump-*; mc --host, mc --version; mc build of a
project whose sources have no <...> AND no [compiler] section (a taught compiler is
#include <mc/host> + <mc/core>, src/driver.mc:414,425, so mc build with [compiler] needs
the mc package -- and that is the honest reason mc install must ship mc/core, mc/core_*,
mc/host* and every lib/*.mc: check-standalone's equality is the proof that <mc/core> IS
the compiler, and a slim binary cannot assemble one without it); mc sysroot, mc limits;
mc install itself. What it refuses, with which message, exit 2 in the M25 shape (src/sysroot.mc
sysroot_missing's tried:/run: block, docs/reference/cli.md § Exit codes gains one row):
mc: <float> is not installed
tried: ~/.mc/libs/mc/v0.10.3 (absent)
run: mc install --yes
(prog.mc:1: prefix as every include error carries; a stale ~/.mc/libs/mc/v0.10.2/ from another
version is not consulted and not mentioned -- per-version directories are the whole point.) The
full binary never prints it: step 2 answers first.
B4. mc install [--yes] [--from URL|DIR] [--libs-dir DIR] -- src/install.mc, in
<mc/core_build>, registered from mc_build_init() (MAXSUBCMD 16, src/hooks.mc: three used
today, seven after this milestone). It populates <libs>/mc/v<mc_version()>/ with the compiler's
own package at the compiler's version, from one of two sources:
- the binary's own blob (default for the FULL binary, no network, no downloader): read
<mc/bundle.list>(A4), for each entrybundle_readthe source andwrite_fileit under its path; writesrc/bundle_data.mcin mode 1 andsrc/bundle.binfrombundle_read(BUNDLE_BIN); writebundle.listat the root. ~50 lines, and it is the ONE definition of the on-disk layout:scripts/release-assets.sh --libs(B5) produces the release tarball by running exactly this into a staging directory, so "the same bytes the full binary embeds" holds by construction; - a release asset (default for the SLIM binary, which has no blob to read):
https://github.com/schivei/mc/releases/download/v<ver>/mc-libs-<ver>.tar.gzand its.sha256, throughfetch_get(draft § 4:sysroot_download,src/sysroot.mc:456, generalised), sha256 bysrc/sha256.mccompared to the first 64 hex characters of the.sha256file (thesha256sum -clinerelease-assets.shwrites) BEFORE anytarspawn,tar -xzf --strip-components=1into the version directory (sysroot_extract's one-spawn shape,:484), every path of the extractedbundle.listprobed withlex_readable, and only then<libs>/mc/v<ver>.toml([source] name = "mc" version url sha256+ one[[file]]per entry:sysroot_manifest's shape,:607, no date). On any failure:unlinkwhat was listed, no manifest, exit 2 with the M25 texts (the download failed,checksum mismatch for mc-libs-0.10.3.tar.gz,the archive did not carry src/core.mc).--from DIRreadsDIR/mc-libs-<ver>.tar.gz+.sha256as local files (the test road; also how an air-gapped machine is fed).
Without --yes it prints the plan (sysroot_plan's shape: source, size when known, sha256 when
known, destination) and nothing was downloaded: re-run with --yes, exit 0 -- M25 D10, same reason
(mc has no isatty). A directory whose manifest already exists is mc: mc 0.10.3 is installed
(~/.mc/libs/mc/v0.10.3), exit 0, unless --force. mc install never touches another package's
directory and never reads a lock: it is about the compiler's own libraries only (mc pkg add is how
a registry package arrives; an mc install NAME alias is not adopted, to keep the two meanings
apart).
B5. Release assets. scripts/release-assets.sh gains --slim (the archive is
mc-<ver>-<target>-slim.tar.gz, the binary inside still named mc/mc.exe -- the ad-hoc signature
identifier is the output basename, release.yml "Build the compiler" step) and --libs VERSION
BINARY (stages BINARY install --yes --from-bundle --libs-dir OUTDIR/.stage/mc-libs-VERSION, then
the same explicit-sorted-members / mtime 0 / ustar / gzip -n -9 rules the script already
imposes, producing mc-libs-<ver>.tar.gz + .sha256, host-independent). release.yml: the macOS
build job builds dist/mc AND dist-slim/mc (build/mc1 --exe src/mc_slim.mc -o dist-slim/mc,
after rm -f), runs scripts/test.sh with the full one, packages both, produces the libs tarball
from the full one, cross-compiles the four slim objects next to the four full ones (make
mc-linux-slim-obj etc.), and publishes a LATEST file (D2). build-linux / build-windows link
the slim object next to the full one and package it with --slim; the bootstrap proof keeps using
the FULL binary as its seed (E, risk). publish already globs dist/mc-*.tar.gz -- the slim and
libs archives ride along; the release body's install snippet gains the slim road (mc install
--yes as the second line). A release therefore carries eleven archives: five full, five slim,
one libs, each with its .sha256, plus LATEST.
B6. Does the full binary consult ~/.mc/libs first? Yes, lock-driven only: step 1 runs
before step 2 for names the lock pins (A3, A5); step 3 is reached only on a bundle miss, which for
the full binary means a name the binary does not ship -- and the installed mc package ships the
same names, so it is a miss there too. So the full binary's behaviour with no lock is byte for byte
today's, and with a lock it is today's plus the overrides the lock spells out. A full binary that
was ALSO mc installed has a ~/.mc/libs/mc/v<ver>/ it never reads for its own includes; what the
directory is for on that machine is mc pkg's cache root and the override trees.
C. A version for mc #
C1. Today. The compiler has no version string (docs/ci.md § Versioning: "nothing in the
working tree records it", the VERSION file was deleted so that the tags would be the only truth;
grep MC_VERSION src/ lib/ is empty). ~/.mc/libs/mc/v<X.Y.Z>/ and mc upgrade both need the
running binary to know it.
C2. The baked version. src/version.mc, checked in, six lines:
// the version this binary reports. 0.0.0-dev in the tree; scripts/set-version.sh
// rewrites it from the tag in release.yml and the result is never committed.
uptr mc_version() { return "0.0.0-dev"; }
Included by src/core_min.mc before cli.mc (it is <mc/core_min>'s: cli.mc prints it,
install.mc/upgrade.mc in <mc/core_build> consume it) and bundled as mc/version, because
<mc/core> includes it and the bundle must stay complete. mc --version prints it and a newline
(src/cli.mc:173, next to --host; --host's three lines stay byte-stable -- release.yml and
the bootstrap scripts sed them). scripts/set-version.sh X.Y.Z validates the shape with
scripts/next-version.sh (the one definition of a version), rewrites the one return line, and
runs make bundle so that mc/version inside the blob agrees with mc_version() -- otherwise a
taught compiler built by a release binary would say 0.0.0-dev and look for the wrong libs
directory. release.yml calls it after make mc1 and before every --exe, cross-compile and
--libs step; it does not commit.
C3. The goldens and check-standalone, thought through. src/mc.mc's bytes differ per release
-- in the working tree of the release runner only. The tag's TREE keeps 0.0.0-dev, and every
golden (tests/golden/mc2.sha256, the two Linux and the two Windows ones) is the hash of an object
compiled from the checked-in tree, so they do not move per release and are rewritten only when
src/ changes on purpose, as today. The release proofs still hold: scripts/bootstrap-linux.sh
takes the versioned binary as SEED, compiles the checked-out tag (dev) to mc1l, which compiles it
again to mc2l.o -- the dev object, against the dev golden -- and the script already allows the seed
to differ from mc1l ("the seed may legitimately be an older compiler", docs/bootstrap.md § The
Linux chain, point 3; what must agree is mc1l --dump-asm vs mc2l --dump-asm, both dev). The
macOS release job runs scripts/test.sh dist/mc, no golden. check-standalone compares a binary
against an object compiled from the SAME tree by the same mc1; on the dev tree both say
0.0.0-dev, on a release runner (if it were run there) both would say the tag; equality holds
either way. Two guards so the sentinel cannot leak: scripts/check-bundle.sh asserts
src/version.mc says 0.0.0-dev (a rewritten file fails make check with a message naming
set-version.sh), and .github/workflows/ci.yml runs make check on the untouched tree as it
does now. docs/ci.md § Versioning is amended, not reversed: the tag is still the only truth for
RELEASES; the tree carries a constant sentinel, not a version, and the objection to two sources of
truth does not apply to a constant.
C4. The dev build. mc_version() is the constant 0.0.0-dev -- not the commit hash, which
would move src/bundle_data.mc, build/mc2.o and all five goldens on every commit. Its libs
directory is ~/.mc/libs/mc/v0.0.0-dev/; a dev FULL binary populates it from its blob
(make check's slim proof uses --libs-dir in a temporary directory and never touches HOME).
For ordering, 0.0.0-dev compares as 0.0.0: every release is newer.
D. mc update and mc upgrade #
D1. Placement. install, update and upgrade are top-level subcommands
(subcommand() registrations, one usage line each, mc with no argument lists them after
build|limits|sysroot), consistent with mc build and mc sysroot fetch; the package-author and
maintenance verbs stay under mc pkg (sync add list vendor verify hash check), and the draft's
mc pkg update is REMOVED in favour of mc update [NAME] [DIR] [--config FILE] [--yes]
[--registry ...] [--libs-dir DIR], same semantics (go get -u: raise the [deps] minimum(s) to the
newest non-yanked index version, then sync). install and upgrade live in <mc/core_build>
(src/install.mc, src/upgrade.mc): a compiler without core_pkg -- the CI/consumer shape of
D12 -- still installs its own libraries and updates itself. update lives in <mc/core_pkg>
(src/pkg.mc, registered from mc_pkg_init() beside pkg): it needs the index and MVS. Semver
parsing/comparison moves from pkg.mc to deps.mc (<mc/core_build>), where upgrade can reach
it.
D2. mc upgrade [--yes] [VERSION] [--from URL|DIR] [--libs-dir DIR] -- the self-update, in
the M25 order (plan, --yes, download, verify BEFORE unpack, act, claim last):
- Which version. With no
VERSION: fetchhttps://github.com/schivei/mc/releases/latest/download/LATEST-- GitHub resolvesreleases/latest/download/<asset>to the newest release's asset by redirect, which-fLsS+--proto-redir =httpsalready follow (sysroot_download) -- a one-line text fileX.Y.Z\nthatrelease.yml'spublishjob attaches (gh release create ... dist/LATEST). Chosen overcurl -sI ... -w '%{redirect_url}': the-wform iscurl-only (thewgetfallback has no equivalent), and a one-line file is parsed by the samever_parsethat validates every version, whereas aLocation:header is a URL to be cut. The GitHub API is JSON, whichmcdoes not parse. With aVERSION: that one, validated. - Refusals, before any download:
mc: 0.10.3 is the newest release(latest == current), exit 0;mc: 0.10.1 is older than this binary (0.10.3): name it with --yes to downgrade(an explicit olderVERSIONwithout--yes), exit 2; aLATESTolder than the running version is refused even with--yes(a stale or tampered file, loudly, exit 2); a dev build (0.0.0-dev) refuses unless--yes VERSIONnames one (somake checknever clobbersbuild/mc-exe, and the acceptance test can). - The plan, then
--yes: current, target, the asset URL for this host pair and this FLAVOUR (BUNDLE_COUNT == 0means slim: fetch-slim), the destination path (the running binary's own path), andnothing was downloaded: re-run with --yes. - Download
mc-<ver>-<target>[-slim].tar.gz+.sha256into<libs>/mc/upgrade.<ver>/throughfetch_get;<target>is the release vocabulary (macos-arm64,linux-arm64,linux-x86_64,windows-arm64,windows-x86_64:host_os()+ anaarch64 -> arm64map of six lines, the "two vocabularies"docs/bootstrap.mdrecords). Verify the archive's SHA-256 before unpacking (src/sha256.mc, against the.sha256line), thentar -xzf F -C DIR --strip-components=1 mc-<ver>-<target>/mc-- one member,sysroot_extract's member-list shape. -
Swap, per host, by
host_self_path()(new in the host layer:_NSGetExecutablePathon macOS,readlink("/proc/self/exe")on Linux,GetModuleFileNameA(0, ...)on Windows -- oneexterneach, theGetEnvironmentVariableAprecedent insrc/host_windows.mc;argv[0]is a PATH lookup and not a path) andhost_retire(path):- macOS:
unlink(self), then write the new bytes to the SAME path withcreat(..., MODE_755). A new inode: the running process keeps the old one, and the kernel's cached-signatureKilled: 9(CLAUDE.md§ M12) happens only when a signed file is overwritten IN PLACE -- which is whydrv_compileunlinks first (src/driver.mc:355, "never rewrite a signed binary in place") and whyrelease.ymldoesrm -f dist/mc. The downloaded binary carries its own ad-hoc signature from the release build and was written asmc; nocodesignis run and no quarantine attribute is set (the file is written bymc, not by a browser). - Linux: the same
unlink+ write (renaming over is also fine; not needed). - Windows: a running
.execannot be deleted (docs/bootstrap.md§ The Windows chain;lib/sys_windows_host.mc:280'sunlinkisDeleteFileA) but it can be RENAMED:MoveFileExA(self, self + ".old", MOVEFILE_REPLACE_EXISTING)-- one more kernel32 name inscripts/sysroot-windows.sh's list (:79-82) and oneexterninsrc/host_windows.mc-- then write the new bytes toself. The nextmc upgradeunlinks a leftovermc.exe.oldfirst;mc --versiondoes not.
- macOS:
- Then the libraries: spawn
<self> install --yes [--libs-dir DIR]-- the NEW binary, throughdrv_spawn_ok(posix_spawnp+waitpid), because the running process is the old version and itsmc_version()is the wrong directory. A full binary installs from its blob (no second download); a slim one fetchesmc-libs-<ver>.tar.gz. Exit is the child's. - Print
mc 0.10.2 -> 0.10.3 (/usr/local/bin/mc); remove the download directory.
D3. The supply-chain gap, named. Every check above is against a .sha256 fetched from the
SAME origin as the archive: that proves the bytes arrived intact, not that they were published by
the owner. Anyone who can serve github.com/schivei/mc/releases/... to this machine can serve a
matching checksum. The priced follow-up is a signing key -- minisign or ssh-keygen -Y sign over
the .sha256 files, the public key baked into src/version.mc next to the version and rotated
with a release, ~90 lines of verification in .mc (Ed25519, no libc) or one more spawned tool --
and mc upgrade refusing an unsigned release once a key ships. Until then packages.md and
docs/ci.md say in one sentence what the .sha256 does and does not prove; mc pkg's tree hashes
are a different matter (they are pinned in the lock by the developer who reviewed the tree).
E. Files, acceptance, risks -- reworked #
Files and estimated deltas (replaces the draft's table; unchanged rows kept for completeness):
| file | lines | what |
|---|---|---|
src/version.mc | ~6, new | mc_version(); the 0.0.0-dev sentinel |
src/core_min.mc | +1 | version.mc before cli.mc |
src/cli.mc | +4 | --version; the usage line |
src/lex.mc | +60 | lopen_fn/lex_set_libs; lex_include_name = three steps + .mc strip; tok_add(".", 1) last in tok_init; lex_root_of + edges + the closure test in lex_include/#embed; the post-parse files accessor. The draft's named-root branch in lex_find_path_from is NOT added |
src/fetch.mc | ~150, new | fetch_get (https via the host's downloader, else local copy), fetch_extract, fetch_sha256_line (parse a .sha256 file), hex64 |
src/sysroot.mc | -75 / +12 | delegates to fetch_* |
src/deps.mc | ~400, new | name rule + reserved mc; [deps]/[replace]/[registry]; lock reader (rows carry lib); tree hash; libs_open (steps 1 and 3 of A3, the bundle.list map, lazily read once); semver; edge registration; the § 3 refusals; the B3 message |
src/install.mc | ~260, new | install_cmd: plan, --yes, --from, --libs-dir, --force; the from-blob road (the one definition of the layout); the tarball road; the manifest |
src/upgrade.mc | ~250, new | upgrade_cmd: LATEST, refusals, plan, download + verify + one-member extract, the per-host swap, the spawned install |
src/pkg.mc | ~700, new | index, MVS, lock writer, archive fetch + hash + manifest, vendor, [deps] edit, hash, check, list, verify, pkg_cmd; update_cmd |
src/core_pkg.mc | ~24, new | core_build.mc + pkg.mc; subcommand("pkg", ...), subcommand("update", ...) |
src/core_build.mc | +6 | fetch deps install upgrade in the list; install/upgrade registrations |
src/bundle_glue.mc, src/core_bundle.mc | ~14 new / -10 +1 | host_bundle_open + mc_bundle_init shared by the two bundle parts |
src/bundle_empty.mc, src/core_bundle_slim.mc, src/core_slim.mc | ~8 / ~12 / ~14, new | the slim assembly (B2) |
src/mc_slim.mc + 4 host slim entries; 4 *-slim-obj.toml | ~5 each / ~12 each, new | the ten release flavours' entries |
src/host_macos.mc, host_linux.mc, host_windows.mc | +10 / +10 / +16 | host_self_path, host_retire; MoveFileExA, GetModuleFileNameA externs |
scripts/sysroot-windows.sh, lib/sys_windows_host.mc | +2 / 0 | the two kernel32 names |
src/core.mc, src/main.mc | +1 / +1 | the sixth part; mc_pkg_init() |
src/toml.mc | +38 | toml_push/toml_pop, toml_occurrences |
src/driver.mc | +45 | --libs-dir; drv_apply_deps for both halves; <...> modules emitted verbatim; the post-parse files check |
tools/bundle.list, src/bundle_data.mc | +13, regenerated | mc/version, mc/fetch, mc/deps, mc/install, mc/upgrade, mc/pkg, mc/core_pkg, mc/bundle_glue, mc/bundle_empty, mc/core_bundle_slim, mc/core_slim, mc/bundle.list, the five slim entries are NOT bundled (they are entries, like mc/main... mc/main is; add mc/mc_slim only if a recreated slim compiler wants it -- not this milestone) |
Makefile | +30 | mc-slim, mc-linux-slim-obj, mc-linux-x86_64-slim-obj, mc-windows-slim-obj, mc-windows-x86_64-slim-obj, libs-tarball, check-pkg, check-slim, check-upgrade inside check |
scripts/set-version.sh | ~40, new | rewrite src/version.mc from a tag, make bundle |
scripts/release-assets.sh | +50 | --slim suffix; --libs mode via BINARY install --yes --from-bundle |
scripts/check-bundle.sh | +6 | the 0.0.0-dev sentinel guard |
scripts/check-slim.sh | ~190, new | Acceptance 20-23 |
scripts/check-upgrade.sh | ~160, new | Acceptance 24-26 |
scripts/check-pkg.sh | ~340, new | Acceptance 1-19 |
scripts/check-parts.sh, check-docs.sh | +10 / +3 | <mc/core_pkg>, <mc/core_bundle_slim>; the new families and flags |
.github/workflows/release.yml | +80 | set-version.sh; slim builds + cross objects; --libs; LATEST; slim links on the Linux/Windows runners; the install snippet |
.github/workflows/ci.yml | +2 | build/mc-slim in the uploaded artifacts |
tests/pkg/... fixtures, tests/golden/pkg-list.txt, .gitattributes | as in the draft | spelling <geo/geo.mc>; the float override fixture (tests/pkg/src/float-1.3.0/) |
docs/reference/packages.md | ~500, new | everything above: the resolution order, the layout of ~/.mc/libs, install/update/upgrade, the slim binary, the version, the trust model |
docs/reference/cli.md | +70 | install, update, upgrade, pkg, --version, --libs-dir; the exit-2 rows |
docs/reference/bundle.md, docs/build.md | +45 / +30 | "the full binary alone"; the mc package; mc/bundle.list; the slim assembly |
docs/ci.md | +90 | § Versioning amended; the eleven assets; LATEST; set-version.sh; what .sha256 proves |
docs/bootstrap.md, tests/golden/README.md, docs/determinism.md | +15 / +10 / +12 | the seed stays full; the dev sentinel and the goldens; the lock and the libs dir as inputs |
docs/guide/25-packages.md, docs/reference/toml.md, docs/reference/diagnostics.md | ~270 / +80 / +75 | as in the draft, plus the three verbs |
schivei/mc-registry | as in the draft | check.yml refuses name = "mc" and any mc/... |
stage0/ | 0 |
Net new src/ lines ~1 700 (the draft's ~1 350 plus install, upgrade, the slim assembly and the host
additions). The five goldens move once per gated commit, under M41's note 9.
Acceptance (ordered; scripts/check-pkg.sh, check-slim.sh, check-upgrade.sh; no network
anywhere; a curl shim on PATH that fails if invoked, for every step that must not download):
1-4. As in the draft (the fixture registry built by the script; the hash stable across hosts; the
plan printed and nothing fetched; MVS not "latest", the lock byte-identical), with --libs-dir and
$tmp/libs/<pack>/v<version>/.
<geo/geo.mc>resolves to the locked version:mc build tests/pkg/app --libs-dir $tmp/c1builds the taught compiler from<teach/mc_teach.mc>+user.mc, compilesmain.mc(which includes<geo/geo.mc>, whose#include <mathx/mathx.mc>resolves through the closure;<geo>alone resolves togeo.mcthrough the lock'slib), and the binary's stdout/exit matchmain.mc's header. With$tmp/libs/geo/v1.0.0/ALSO present, the object iscmp-identical to one built with onlyv1.2.0/present: the unlocked directory was never opened. 6-9. As in the draft (byte-identical objects from two fetches; the tampered source refused, exit 2; stale lock / stale tree, nocurl; vendoring is the offline road,deps/<pack>/wins).- A package is closed, spelled with
<bad/bad.mc>; the#embedcase;not declared in geo's [package].files. - Names:
[deps] mc = "1.0.0"->mc: reserved;Geo = ...->invalid package name; exit 1 at the key.[deps] float = "1.3.0"is ACCEPTED (see 18). 12-17. As in the draft (majors refused; a failed fetch leaves no claim;mc pkg addedits one line;mc pkg checkis the gate, and refusesname = "mc"; parts -- a probe compiler withoutcore_pkgbuilds the vendored app and prints a usage with nopkg/updateline but WITHinstall/upgrade; inert --check-inertclean,check-standalone,check-obj32/32,check-build,check-sysrootsunchanged,mc sysroot fetch linux-aarch64 --yesin CI still writes the four files). - A bundled name pinned in
[deps]overrides the bundle, byte-checked: the fixture packagefloat1.3.0 (a copy oflib/float.mc+float_rt.mcwith one visible change:putf64prints a!suffix) locked bytests/pkg/app-float/;mc buildwith the FULL binary produces a program that prints the!; the object iscmp-identical between the cache road and the vendored road; one byte appended to$tmp/libs/float/v1.3.0/float_rt.mc-> exit 2float 1.3.0: float_rt.mc does not match mc.lock; removing the[deps] floatline and re-syncing gives an objectcmp-identical to one built by a checkout with notests/pkgat all -- the override leaves nothing behind. - No lock, no network, the FULL binary resolves
<float>:build/mc-exe --exeonlib/mc_float.mc's shape in an empty directory withHOMEunset and thecurlshim -- the existingcheck-standalonesteps, re-run with the shim, plus<float>. - The SLIM binary before
mc install:build/mc-slim(fromsrc/mc_slim.mc, ~400 KB, asserted< 60%ofbuild/mc-exe's size) compilesi64 main() { return 42; }to an objectcmp-identical tobuild/mc-exe's;--dump-asmofsrc/arena.mcidentical between the two;mc-slim --exe hello.mc(<sys>+<prelude>) exits 2 with<sys> is not installed/tried:/run: mc install --yes;mc-slim install --libs-dir $tmp/libs(no--yes) prints the plan andnothing was downloaded, and$tmp/libsholds no manifest. -
mc installfrom a local tarball, and the standalone proof extended:make libs-tarball(release-assets.sh --libs 0.0.0-dev build/mc-exe dist) writesdist/mc-libs-0.0.0-dev.tar.gz.sha256, reproducibly (two runscmpequal);mc-slim install --yes --from dist --libs-dir $tmp/libspopulates$tmp/libs/mc/v0.0.0-dev/and writesv0.0.0-dev.tomllast; then, in an empty directory with the shim onPATH,mc-slimruns every step ofcheck-standalone.shwith--libs-dir $tmp/libs:helloruns (exit 42), the taught compiler from<mc/host>+<mc/core>+<user_syntax_demo>is built and signed, compiles<syntax_demo_test>, and<mc/host>+<mc/core>+<user_default>compiles to an objectcmp-identical tobuild/mc2.o-- the slim binary plus the installed package is the compiler, byte for byte. A tarball with one flipped byte in its.sha256->checksum mismatch for mc-libs-0.0.0-dev.tar.gz, no directory, no manifest; a secondinstallafter success saysis installed, exit 0.
- The FULL binary's
mc installneeds no network:build/mc-exe install --yes --libs-dir $tmp/libs2with the shim; the tree isdiff -r-identical to the one step 21 unpacked (the tarball came from the same road, so this is a reproducibility check of the road itself);src/bundle_data.mcin it is the mode-1 form andsrc/bundle.binhasbundle_bin_size()bytes. mc --versionprints0.0.0-devforbuild/mc-exeandbuild/mc-slim; a tree copied to$tmp/treewithscripts/set-version.sh 9.9.9applied (in the copy) and compiled withbuild/mc1 --exe $tmp/tree/src/mc.mc -o $tmp/rel/mcprints9.9.9;scripts/check-bundle.shon the COPY fails naming the sentinel, on the tree passes.mc upgradeagainst a local release directory, no network:release-assets.sh 9.9.9 macos-arm64 $tmp/rel/mc $tmp/reland--libs 9.9.9 $tmp/rel/mc $tmp/relproduce the two archives;cp build/mc-exe $tmp/bin/mc;$tmp/bin/mc upgrade --from $tmp/rel --libs-dir $tmp/libs3(no--yes) prints the plan naming$tmp/bin/mcand downloads nothing;$tmp/bin/mc upgrade --yes 9.9.9 --from $tmp/rel --libs-dir $tmp/libs3(the explicit form, since the running binary is a dev build) swaps the file, spawns the new binary'sinstall, and afterwards$tmp/bin/mc --versionprints9.9.9,$tmp/libs3/mc/v9.9.9.tomlexists, and$tmp/bin/mccompileshello.mcthrough--libs-dir $tmp/libs3.- The macOS in-place-overwrite hazard does not occur:
stat -f %i $tmp/bin/mcbefore and after step 24 differ (a new inode),codesign --verify --verbose=4 $tmp/bin/mcpasses, and$tmp/bin/mc --versionexits 0 -- notKilled: 9. On the Windows CI legs the same script assertsmc.exe.oldexists after the swap and is gone after a secondupgrade(a no-op one,is the newest release). - Refusals:
upgrade --yes 0.0.1 --from $tmp/relon the 9.9.9 binary without--yes... as specified: an older explicitVERSIONwithout--yes->older than this binary, exit 2, no file touched (inode unchanged); aLATESTfile saying0.0.1against the 9.9.9 binary -> refused even with--yes; a tarball whose.sha256does not match ->checksum mismatch, the binary untouched. - CI builds both flavours and
make checkis green withcheck-pkg,check-slim,check-upgradeinside it;check-partsshows<mc/core_min>+<mc/core_pkg>and<mc/core_min>+<mc/core_bundle_slim>compile alone; goldens rewritten once per gated commit after an empty--dump-asmdiff andcmp build/mc2.o build/mc3.o;make check-docsgreen.
Risks (in addition to the draft's 1-12, which stand; 10 is closed by C):
- A stale
~/.mc/libsfrom another version. Closed by construction: themcpackage lives underv<mc_version()>/and nothing else is consulted; the B3 message names the directory it looked for. What is NOT closed: disk growth across versions --mc install --prune(remove everymc/v*but the running one) is ~25 lines and optional. - The seed of the bootstrap must stay the FULL binary.
bootstrap-linux.sh/-windows.shdownloadmc-<VER>-<target>.tar.gz(the unsuffixed name) and stay as they are; the chain proper (src/mc_linux.mchas no<...>) would in fact close with a slim seed, but the release gate runs the whole suite with the seed and must not depend on~/.mcor on a second download.docs/bootstrap.mdsays so in one paragraph. - CI must build both flavours or a slim-only breakage (the empty-bundle assembly,
lopen_fnstep 3) ships unnoticed;check-slimis insidemake checkandrelease.ymllinks the slim object on every runner. Cost: five more cross-compiles and five more links per release, ~1 minute. - The slim binary and
check-standalone. The original script keeps proving the FULL binary; the slim proof (step 21) is a second script because its precondition (a libs tarball) is not the empty directory the original insists on. A drift between the two proofs is a documentation bug to watch: both compare againstbuild/mc2.o. - The
.token. Appended last intok_init, so no id shifts; but a taught module that relied on.being ABSENT (a#token "."of its own is fine --tok_addis idempotent -- but asyntax_lit/on_stmtthat saw1.5as three tokens on purpose is not) would change behaviour. No module in the tree does;check-lang,check-float,check-surfaceare the net. mc upgradeis the widest new attack surface: it writes over the compiler. D3 names what the.sha256proves; until a signing key ships,packages.mdsays "mc upgradetrusts the release host".--fromwith a local directory is also how a reviewer can stage an upgrade.set-version.shleaking into a commit moves the goldens and the bundle silently;check-bundle's sentinel guard makes it a redmake check.- Windows
mc.exe.old:MoveFileExAon a running executable succeeds, but an antivirus holding the file open can make the rename fail; the message names the file and asks the user to re-run, nothing is half-written (the new bytes are written only after the rename returns). host_self_path()and symlinks:/usr/local/bin/mc -> /opt/mc/0.10.2/mcgets the resolved TARGET replaced (that is what_NSGetExecutablePath/readlinkreturn), not the link; documented, not resolved. A binary on a read-only path fails atcreatwithcannot writenaming the path, exit 2.- A taught compiler built by the slim binary carries the full blob (its
src/bundle_data.mcon disk is the wholemcpackage): expected -- it is the same object the full binary builds (step 21) -- but a user who chose slim for size gets full-size taught compilers unless the project sayscore = "<mc/core_slim>", which is whycore_slimis bundled.
F. Amended decisions -- to ratify with the owner #
- D1' -- import spelling:
#include <pack/file.mc>, angle brackets, one resolution model (lock, bundle, installedmcpackage; never the working directory);<pack>alone is the package'slibentry;.becomes a core lexeme appended last intok_init, and a trailing.mcis stripped from every<...>name. Rejects: the draft's named roots for the quote form (two spellings for one thing) and<pack/file>without the extension as the only form (the owner's spelling, one token away). - D2' -- identity is a registry name;
[deps] name = "min.version";mc,mc/...,deps,buildreserved;lib/'s bundled names are NOT reserved. Rejects: the draft's blanket reservation of every bundled name (it forecloses the extensibility the ruling asks for). - D10' -- the cache is
~/.mc/libs/<pack>/v<version>/with<pack>/v<version>.tomlbeside;--libs-dir DIRon every reader;deps/<pack>/vendoring wins when present. Rejects:~/.mc/pkg/<name>/<version>/and--pkg-dir. - D15' -- the bundle is the
mcpackage: versioned with the compiler, embedded in the full binary, installable beside the slim one under~/.mc/libs/mc/v<version>/in the repository layout withbundle.listas the name map; a bundledlib/name pinned in[deps]overrides the blob, lock-driven and byte-checked; themcpackage itself can never be pinned. Rejects: a by-name layout (breaks../lib/prelude.mc), and refusing collisions (loses the override with no gain: the lock already pins content). - D18 -- two release flavours per target:
mc(full, ~760 KB, the seed and the default) andmc-slim(~400 KB by M41's table, an empty bundle assembled from<mc/core_bundle_slim>at zero core lines); eleven archives per release plusLATEST. Rejects: slim as the only flavour (the bootstrap seed and offline CI need the blob) and a slim binary that carrieslib/but notmc/core(a[compiler]section would then fail, andcheck-standalone's equality is the property being sold). - D19 --
mc install [--yes] [--from URL|DIR] [--libs-dir DIR], top-level, in<mc/core_build>: the compiler's own package at the compiler's version, from the blob (full binary, no network) or frommc-libs-<ver>.tar.gz+.sha256(verified before unpack, manifest written last);release-assets.sh --libsis that same road into a staging directory. Rejects: a hand-maintained member list in the script (two definitions of one layout) and anmc install NAMEalias formc pkg add. - D20 -- the version is baked:
src/version.mc(mc_version(), in<mc/core_min>, bundled asmc/version),0.0.0-devin the tree, rewritten byscripts/set-version.sh+make bundleinrelease.ymland never committed; the goldens are recorded for the dev tree and do not move per release;mc --version;check-bundleguards the sentinel. Rejects: the commit hash as the dev version (moves every golden every commit), a version outside the bundle (a taught compiler would report and install the wrong one), and aVERSIONfile (the reason it was deleted stands). - D21 --
install,update,upgradeare top-level subcommands;mc pkgkeepssync add list vendor verify hash check;mc pkg updateis removed;updateis<mc/core_pkg>'s, the other two are<mc/core_build>'s. Rejects: three moremc pkgverbs (the user-facing ones read likemc build, not like maintenance). - D22 --
mc upgrade [--yes] [VERSION] [--from URL|DIR]: the newest version from a one-lineLATESTasset atreleases/latest/download/LATEST; the host pair's archive of the running flavour plus.sha256, verified before unpack; the swap isunlink+ write on macOS and Linux (a new inode, never in place) andMoveFileExAto.old+ write on Windows; then the NEW binary'sinstall --yesis spawned; a downgrade needs--yes VERSION, a dev build needs the same, a staleLATESTis refused. Rejects: the GitHub API (JSON),curl -w '%{redirect_url}'(curl-only), in-place overwrite (Killed: 9), and running the old process's own install logic for the new version's libraries. - D23 -- the host layer gains
host_self_path()andhost_retire(path)(three externs:_NSGetExecutablePath,readlink,GetModuleFileNameA/MoveFileExA);rename,opendirandgetenvare still not added. Rejects:argv[0]as the binary's path (it is aPATHlookup). - D24 -- the full binary consults the lock before its bundle (the override) and never consults
the installed
mcpackage for its own names; with no lock its behaviour is byte for byte today's. Rejects: an unconditional~/.mc/libslookup (the working-directory-independence argument of M15, one level up). - D25 --
.sha256from the release origin is integrity, not authenticity; a signing key (minisign orssh-keygen -Y, public key insrc/version.mc) is the priced follow-up and not in this milestone. Rejects: shippingmc upgradewithout saying so inpackages.mdandci.md.
Architect's additions, extended: (d) step 24's local-release test is also the documented road for an
air-gapped upgrade (--from DIR), and packages.md shows it; (e) the release body's install snippet
lists the slim road second, never first -- the full binary stays what a newcomer downloads; (f) the
first commit of this milestone is src/version.mc + --version + the sentinel guard alone, so the
goldens move once for a six-line change and every later commit can be checked against a versioned
binary.
Implementation notes #
Step 1 — src/version.mc, mc --version, the sentinel guard (D20) #
The architect's addition (f): the first commit of this milestone is D20 and nothing else, so the
five goldens move once for a nine-line change and every later step can be checked against a
binary that knows what it is. Nothing of the slim binary, mc install, mc upgrade, LATEST or
[deps] is in it.
1. What was written. src/version.mc (33 lines, one of them code):
uptr mc_version() { return "0.0.0-dev"; }
src/core_min.mc includes it between hooks.mc and cli.mc — before cli.mc, which calls it,
and after everything it does not need, which is everything: the file names no other function.
src/cli.mc gained dump_version() (four lines), one else if in mc_main's argument loop next
to --host, and one line in usage(). tools/bundle.list gained mc/version (87 entries).
Measured cost in src/, excluding the generated src/bundle_data.mc: 53 added lines, 9 of them
neither comment nor blank — version.mc 33/1, cli.mc +17/7, core_min.mc +3/1. The spec's
§ E table estimated ~6 + 1 + 4 = 11 code lines; the difference is this repository's comment
density and the one-line usage entry.
2. The format is mc <version> and a newline, decided from docs/ci.md § Versioning with
--host as the shape. --host is the precedent for the mechanism — a one-shot informational
flag, answered before any source is read, stdout, exit 0 — but not for the layout: it prints
key value lines because it answers three questions. --version answers one, so it prints the
conventional single line, with the program name on it because that is what gets pasted into a bug
report and what every other --version on the machine prints. The version carries no v:
docs/ci.md says the v belongs to the tag name and nothing else, and the string is exactly what
scripts/release-assets.sh takes as its first argument and what a [deps] minimum will carry in
step 2.
3. scripts/set-version.sh (67 lines). It validates the X.Y.Z half by calling
scripts/next-version.sh "$core" patch and throwing the answer away — so there is still exactly
one definition of what a version number is — and validates a -suffix itself against
[0-9A-Za-z.-]+. The suffix is not decoration: 0.0.0-dev is one, so
scripts/set-version.sh 0.0.0-dev is the documented way back after a local experiment, and a
hand-pushed pre-release tag (the only kind docs/ci.md allows) goes through unchanged. A leading
v is stripped, as next-version.sh strips it.
The rewrite is awk over the whole file plus mv, not sed -i: BSD and GNU sed disagree about
-i's argument, and every other script here avoids it. It asserts there is exactly ONE
^uptr mc_version() { return " line first, so a future edit to src/version.mc that splits the
literal fails loudly instead of silently baking nothing. Its last act is make bundle.
4. make bundle is not a convenience there, it is the point of D20. src/version.mc is
bundled as mc/version and <mc/core_min> includes it, so a taught compiler is built out of the
blob's copy. Measured, both directions:
$ build/mc-exe --version # the dev binary
mc 0.0.0-dev
$ ./taughtdev --version # <mc/host> + <mc/core> + <user_default>, built by it
mc 0.0.0-dev
$ $tmp/rel/mc --version # the same tree with set-version.sh 9.9.9
mc 9.9.9
$ ./taught999 --version # the taught compiler THAT binary builds
mc 9.9.9
Without the make bundle, the last line would read mc 0.0.0-dev — a release binary building
compilers that lie about their version and, from step 3, look in the wrong
~/.mc/libs/mc/v<version>/. That is the whole argument for the version living in the bundle, and
it is now a measurement rather than a claim.
5. The sentinel guard. scripts/check-bundle.sh gained six lines asserting the exact literal
^uptr mc_version() { return "0.0.0-dev"; }$. It lives there, and not in a check of its own,
because set-version.sh's second act is make bundle: the two artefacts a release rewrite
touches are src/version.mc and src/bundle_data.mc, and this script already owns the second.
Measured: scripts/check-bundle.sh exits 1 on the 9.9.9 copy with
FAIL: src/version.mc does not carry the 0.0.0-dev SENTINEL.
It says: uptr mc_version() { return "9.9.9"; }
scripts/set-version.sh ran in this tree. A release build is never
committed: restore it with 'scripts/set-version.sh 0.0.0-dev'
(docs/ci.md § Versioning).
and 0 on the tree.
6. Where the tag reaches release.yml. The build job (macOS) derives VERSION from
inputs.tag || GITHUB_REF_NAME, validates its shape with next-version.sh and exports it through
$GITHUB_ENV; the build-linux and build-windows jobs re-derive the same string without
re-validating, and they link objects cross-compiled by the build job. So the version has to
be baked in exactly one place, and it is: the old "Build the compiler" step was split into
Build the seed compiler—make mc1(set-version.sh'smake bundleneeds a working compiler);Bake the release version into the tree—scripts/set-version.sh "$VERSION";Build the compiler— the--exe, plustest "$(dist/mc --version)" = "mc $VERSION".
Everything after that step in the same job — the two Linux ELF objects, the two Windows COFF
objects, mcrt/winstart — is compiled from the rewritten tree, so all five shipped binaries
carry the tag. The make targets that produce them rebuild build/mc1 first (its prerequisite
src/bundle_data.mc is now newer), which is correct and costs a second.
7. What D20 said that did not survive contact with the tree, and what changed instead:
src/version.mcis six lines in the spec and 33 here. The extra 27 are comment: why a constant is not a second source of truth, what regenerates the bundle, and the two rejects. The code is the one line the spec wrote.- The spec put
--version"atsrc/cli.mc:173, next to--host".dump_version()is next todump_host()and the dispatch is the line after--host's, but the file is 20 lines longer than when the spec was written; line numbers were not usable and are not quoted here. set-version.sh"validates the shape withscripts/next-version.sh" reads as delegation of the whole string. It cannot be:next-version.shrejects0.2.0-rc1on purpose, and the sentinel0.0.0-devis itself suffixed — the script would refuse to write the value the repository must hold. Split as in note 3:next-version.shownsX.Y.Z,set-version.showns the suffix.- The
check-bundleguard's placement. The spec says only that the script asserts the sentinel. It is placed after thesrc/bundle_data.mcfreshness comparison and before the<mc/bundle_data>shape check, so a tree that is BOTH stale and versioned is reported as stale first — the actionable failure. mc_version()is not required bymake check-docs. The coverage list is a set of name prefixes (p_,type_,machine,host_, …) andmc_is not one of them, so nothing forced a reference entry. It is documented anyway, indocs/reference/bundle.md§<mc/version>; what the gate DID require is the flag,--versionindocs/reference/cli.md, because it is matched as astr_eq(a, "--version")literal (34 flags now, was 33).- Nothing had to move for the include to work. The spec's § E table budgets no change beyond
core_min.mc+1, and none was needed:version.mcnames no other function, so it can sit anywhere afterarena.mcand beforecli.mc, and the relative include inside a BUNDLED<mc/core_min>resolves by last component tomc/versionwith no entry in the resolver.
8. Acceptance 23, first half, measured on this tree (the rest of 23 needs build/mc-slim,
which is step 2):
| claim | result |
|---|---|
build/mc-exe --version | mc 0.0.0-dev |
a copy with set-version.sh 9.9.9, compiled by the TREE's build/mc1 | mc 9.9.9 |
check-bundle.sh on the copy | exit 1, names the sentinel |
check-bundle.sh on the tree | exit 0, ok src/version.mc carries the 0.0.0-dev sentinel |
| a taught compiler reports the version of the binary that built it | both directions, note 4 |
9. Inertness and the numbers. scripts/check-inert.sh build/mc1.pre build/mc1 (pre = a mc1
built from main at 9552719): 33 objects identical (tests/*.mc and src/mc.mc) plus
byte-identical artefacts for examples/api, examples/lang, examples/conc,
examples/desktop and examples/kernel through the taught compiler each side builds.
make check RC 0, zero FAIL, 5m57s; check-obj 32/32 identical to the frozen seed;
check-limits 17/17 under 90%, with globals unchanged at 432/512 (84%) — the version is
a string literal, not a global, so the step added none — and strings 1247/2048 (60%).
make check-linux-host RC 0 over all four cells. The five goldens were rewritten once, each
after its own criterion (§ Implementation notes of M41, note 9), and the values are in
CLAUDE.md § State.
Step 2 — the resolution model, mc.lock and the closure rule (A1-A3, A5, A6) #
The half of packages that has no network in it: what answers #include <pack/file.mc>, what
mc.lock says, what is rehashed on every build, and what a package may read. No fetcher, no lock
writer, no mc pkg, no slim binary, no install/update/upgrade — those are steps 3 to 5.
1. What was written.
| file | added | of which code | what |
|---|---|---|---|
src/deps.mc | 662 | 478 | new: the name rule, [deps]/[replace]/[registry], the lock reader, the tree hash, the cache manifest, libs_open, semver, the refusals, the post-parse files walk |
src/lex.mc | +217 | 123 | tok_add(".", 1); lopen_fn/lex_set_libs; lex_include_libs; lex_include_name as three steps plus the .mc strip; the package roots, the edge list, lex_root_of, lex_closed; lex_inc_count/lex_inc_at |
src/toml.mc | +65 | 47 | toml_push/toml_pop/toml_occurrences |
src/driver.mc | +38 | 21 | --libs-dir, forwarded to the spawned child; deps_apply in drv_parse for BOTH halves; deps_check_files after the parse; a <...> module emitted verbatim |
src/core_build.mc | +9 | 3 | deps.mc in the list; lex_set_libs(&libs_open) in mc_build_init |
src/parse.mc | +8 | 5 | the closure test on #embed's resolved path |
999 added lines in src/, 677 of them neither comment nor blank, against the amended § E
table's ~400 (deps.mc) + 60 (lex.mc) + 38 (toml.mc) + 45 (driver.mc) = ~543. The excess is
almost entirely deps.mc: the spec's line named "cache/vendor/path resolution, per-file
verification" in six words, and the cache manifest reader, the two-implementation hash and the
bundle.list reader for step 3 of A3 are each real code. stage0/ is untouched (2848/3000);
git diff origin/main -- stage0/ is empty.
Outside src/: scripts/check-pkg.sh (330), scripts/pkg-hash.sh (60, the tree hash in shell),
tests/pkg/ (six fixture packages, five consumer projects, three locks), one line in Makefile
and one in .gitignore.
2. . is a lexeme, and it cost one lex-skip. tok_add(".", 1) is appended after =>, so no
id moves: K_U8..K_EXTERN are still 256..269 and the new K_DOT is 301. check-lex compares
mc0 --dump-tokens against src/lexdump.mc over the whole corpus and it went from 136/136 to
135 identical + one FAIL: lib/syntax_demo_test.mc uses the taught .+ operator, and
--dump-tokens does not process directives, so the new lexer reads . + where the frozen seed
says unexpected character. Under a real compile the #infix has registered .+ and the longest
match takes it on both sides — which is why check-asm and check-ast still compare that file
byte for byte.
The escape is a header of its own, // lex-skip:, and deliberately not seed-skip::
seed-skip is read by check-asm.sh and check-ast.sh too, and using it here would have dropped
the file from two gates that can still compare it. check-lex now reports 135/135 files
identical (3 skipped) and names the reason. This is M44 risk 17, measured.
3. lex_include_name is three steps, and the guard is what makes the override safe. Step 1
(the lock) is consulted only when the file asking is allowed to reach that package — a file inside
geo that writes <float> gets the BUNDLE's float, not a project's override it never asked for.
A name whose first component IS a locked package the file may not reach is not silently downgraded
either: step 1 is skipped, and if nothing else answers, the refusal is the closure message and not
unknown bundled include.
4. Two path bugs that only a real fixture finds, both worth writing down because they were silent:
path_normdrops a trailing slash, sopath_join(cfg, "deps/geo/")came back as.../deps/geoand the next join cut it off as if it were a file name — the vendored road looked fordeps/mc.toml. The directory is now normalised once, indp_set_dir, and the/is re-attached after.- macOS
TMPDIRends in/, so--libs-dir "$TMPDIR/x/libs"carries a//. The package roots are compared against paths that came out ofpath_join, which normalises, so the root was a prefix of nothing:lex_root_ofanswered -1 for every file and the closure rule silently never fired. Same fix, same line. Both are the reasondp_set_dirnormalises rather than storing what it was given.
5. Deviations from the amendment's text, on record.
- The per-file attribution needs the cache manifest, and a vendored tree has none. The lock
pins ONE hash per package (D4), so a mismatch says "something moved" and nothing more; what names
the file is
<libs>/<pack>/v<version>.toml, the[[file]]list the fetch writes last. The draft § 3 table assumes it. A vendored tree has no manifest, so its refusal isgeo 1.2.0: the tree does not match mc.lock— a third row the draft did not have. The reader is ~40 lines and it was implemented now rather than in step 3, because the manifest is also what makesis not fetchedexact: a half-extracted directory has no manifest and is reported as not fetched instead of as does not match. mc.lock is stalecarries the M25run:block, not the draft's inlinemc.lock is stale: run mc pkg sync --yes. Every exit-2 refusal in this compiler has the same shape (src/sysroot.mc,docs/reference/cli.md§ Exit codes) and having one of them spellruninside its own first line would be the odd one out. The text ismc: mc.lock is stale: geoplusrun: mc pkg sync --yes.- The once-only key for a file served from disk is its NORMALISED path, not its absolute path.
mchas nogetcwdand does not add one for this: what matters is that the key is the same onelex_includewould record for the same file, and both go throughpath_join/path_norm. It is absolute exactly when the config's path is. libs_openre-attaches the.mcthe lexer stripped.<geo/geo.mc>and<geo/geo>are one name (A2), but a bundled name never carries an extension while a file on disk always does, so the disk road triesrestand thenrest + ".mc". That also leaves<pack/table.txt>— an#embedpayload — findable under the name it was written with.- No arena tag for either new table. The package roots and the edge list are sized ONCE, from
the lock, before the first root is registered (
lex_pkg_reserve); the per-package file list doubles insidedeps.mc. None of the three scales with the program, and a growth event on a table whose size the developer wrote inmc.lockcarries nothing anybody could act on — the M17 argument forMAXTARGETS, with an exact count instead of a ceiling.mc limitstherefore gains no row andarena.mcis untouched. [registry]is read and unused.deps_registry()exists so the key is validated and documented; nothing in<mc/core_build>reaches the network.
6. How step 3 of A3 is tested at all. A full binary answers <mc/core> out of its own blob and
never reaches the installed mc package, so the road would have been dead code until the slim
binary of § B2 exists. tests/pkg/nobundle.mc (26 lines) is the probe: every part of mc except
<mc/core_bundle>, with a main() that does not call mc_bundle_init(). bopen_fn is never
registered, every bundled name misses, and lopen_fn answers. That file is, line for line, what
mc-slim will be once the empty bundle and its glue are checked in.
The measurement: check-pkg.sh lays <libs>/mc/v0.0.0-dev/ out by hand in the REPOSITORY layout
(every path tools/bundle.list names, plus src/bundle_data.mc, which the blob cannot carry) with
bundle.list at the root as the name map, and the probe compiles
#include <mc/host> + <mc/core> + <user_default> into an object cmp-identical to
build/mc1 src/mc.mc. That is check-standalone's equality with the blob replaced by a
directory, and it is the property step 4 sells.
7. The gate: scripts/check-pkg.sh, 31/31, inside make check. A curl, a wget and a tar
that exit 97 sit on PATH for the whole run, so "mc build never downloads" is proved and not
asserted (architect's addition (a)). What it measures, in order:
ok tree hash geo/mathx/teach/float/bad: mc.lock and scripts/pkg-hash.sh agree
ok <geo/geo.mc> + <geo> + <mathx/mathx.mc> + <teach/mc_teach.mc>: exit 42, 'geo 120'
ok geo v1.0.0 beside v1.2.0 changes nothing: the two objects are identical
ok tampered vec.mc: mc: geo 1.2.0: vec.mc does not match mc.lock
ok tampered geo mc.toml: mc: geo 1.2.0: mc.toml does not match mc.lock
ok stale lock: mc: mc.lock is stale: geo
ok unfetched package: mc: mathx 1.0.0 is not fetched (+ the run: line)
ok deps/ wins and gives the same object as the cache road, byte for byte
ok closure, #include / #embed: package bad reaches outside its tree: ...
ok undeclared file: geo/extra.mc:1: not declared in geo's [package].files
ok reserved package name: deps.mc / invalid package name: deps.Geo / deps.deps
ok <float> and <float/float_rt.mc> come from the locked tree ('1.500000!')
ok the override is the same object vendored or installed
ok tampered float_rt.mc: mc: float 1.3.0: float_rt.mc does not match mc.lock
ok with no [deps] the installed float cannot change a byte
ok the override does change the object, and removing it puts it back
ok a float.mc beside the entry does not shadow <float_rt>
ok the single-file CLI refuses <geo/geo.mc>: unknown bundled include: geo/geo
ok <mc/host> + <mc/core> + <user_default> served from <libs>/mc/v... == src/mc.mc
ok the probe with no installed package: unknown bundled include: mc/host
ok check-standalone is green with curl/wget/tar refusing to run
check-pkg: 31/31
scripts/pkg-hash.sh is a SECOND implementation of D5's hash, in shell, and every lock hash
checked into tests/pkg is compared against it on every run — so a divergence between deps.mc
and the specification is a red make check rather than a surprise at a consumer. It is also how
the fixture locks and the cache manifests are produced, which is what mc pkg sync|hash will do in
step 3. .gitattributes already carries * -text (M38), so risk 8 needed no line.
8. Inertness and the numbers. scripts/check-inert.sh /tmp/m44pre/build/mc1 build/mc1 (pre = a
mc1 built from the step-1 commit): 33 objects identical (tests/*.mc and src/mc.mc) plus
byte-identical artefacts for examples/api, examples/lang, examples/conc, examples/desktop
and examples/kernel through the taught compiler each side builds. A project with no [deps]
reads no lock, registers no root and takes no extra open — D24 holds by construction, since
deps_apply returns before touching the disk.
make bundle re-run BEFORE bootstrapping (tools/bundle.list gained mc/deps): 88 files,
src/bundle_data.mc 1 208 008 bytes, <mc/bundle_data> one #embed node plus the 352-value
index, lz round trip 112 cases.
make check green end to end (RC 0, zero FAIL): budget 2848/3000, test 32/32, check-lex
135/135 (3 skipped), check-ast 137/137, check-asm 137/137, check-obj 32/32 identical to the
frozen seed, check-bundle (reproducible + fresh), bootstrap at a fixed point
(cmp build/mc2.o build/mc3.o, 1 154 264 bytes; the --dump-asm diff between mc1 and mc2 is
empty), check-surface 32/32 + inert, test-exe 32/32, check-mc 15/15,
check-standalone, check-parts, check-toml 10/10, check-build 53/53, check-pkg 31/31,
check-sysroots (13 rows), check-stubs 9/9, check-limits 17/17 under 90%,
check-minimal, test-linux 41/41, test-linux-x86_64 38/38, test-linux-exe 44/44 musl +
44/44 gnu, test-linux-x86_64-exe 41/41 musl + 41/41 gnu, test-windows 42/42 objects,
test-windows-x86_64 40/40, check-examples, check-lang, check-conc, check-desktop,
check-float, check-wide, check-kernel, check-avr, check-docs (196 symbols, 35 flags,
24 TOML keys, 10 directives, 51 samples, 349 links), site + check-site, test-sandbox
55 ok. make check-linux-host RC 0 over all four cells.
The globals budget, which is the one seed limit this milestone could have broken: globals
went from 432/512 (84%) to 439/512 (85%) — seven, and check-limits fails at 460. The state is
in arena records with accessors: deps.mc costs two (dp, the whole package/file/bundle.list
state behind dp_state(), and dp_libs_opt) and lex.mc five (lopen_fn, and the two tables
with their counts).
9. The five goldens rewritten once, each after its own criterion:
tests/golden/mc2.sha256 149d6a06…06b2ce ->
9e00398d7338ad9b53654c7f07e0d16ff21319e79c915b2ac473d20e1411420a (after the empty --dump-asm
diff and cmp build/mc2.o build/mc3.o); the Linux pair deleted and re-recorded by
make check-linux-host (Docker, both architectures, both libcs, each after its own fixed point and
with the cross proof green) — mc2-linux-arm64.sha256
67f062a4a5363b850fef4410f36408942c9be177801d76c4c4d273c1c47a02fd,
mc2-linux-x86_64.sha256 3c93e81f4745391de7316aee7a941574258e8961652dd6a28ffab73488debe70; the
Windows pair cross-computed per tests/golden/README.md — mc2-windows-arm64.sha256
2858b236d25352c4064f3620f0d78e2262bc62a7228061fdd860e71fa03f5959 (1 177 344 B),
mc2-windows-x86_64.sha256
c75e23fdee44b58f85313997196e799b20419a1ae688ef5e525ec09edb909322 (1 207 508 B).
10. What step 2 deliberately did NOT do, so the next step knows where the seam is: there is no
src/fetch.mc, no src/pkg.mc, no <mc/core_pkg> and no subcommand — mc pkg sync|add|list|
vendor|verify|hash|check and the registry are step 3; the lock and the cache manifests are written
by scripts/pkg-hash.sh and scripts/check-pkg.sh in their place, in exactly the format
packages.md documents. MVS is not implemented and no fixture needs it: the locks are hand-written
and the [deps] check is a minimum comparison (ver_cmp), which is all a build has to do. The
mc package on disk (A4) has a READER and no writer — mc install is step 4.
Docs: docs/reference/packages.md (new, ~230 lines), docs/guide/25-packages.md (new, ~140),
docs/reference/toml.md ([deps], [replace], [registry], [package]),
docs/reference/diagnostics.md § 13 (thirteen rows), docs/reference/cli.md (--libs-dir, the
exit-2 rows), docs/reference/hooks.md § 4 (lex_set_libs, lex_root_of, the once-only
accessors, and lex_include's new closure clause), docs/reference/bundle.md and docs/build.md
(the M15 promise restated per § A7), docs/README.md.
Step 3 — the registry, MVS, the lock writer and mc pkg (§ 3-§ 8, D21) #
The write and network side. What answers #include <pack/file.mc> was step 2 and did not move;
what this step adds is how a lock and a tree get onto the disk in the first place. No slim binary,
no mc install, no mc upgrade, no LATEST — those are steps 4 and 5.
1. What was written.
| file | added | of which code | what |
|---|---|---|---|
src/pkg.mc | 1387 | 1106 | new: the index reader, MVS, the lock writer, the archive fetch, the cache manifest, vendor, add, list, verify, hash, check, pkg_cmd and update_cmd |
src/fetch.mc | 169 | 105 | new: fetch_get, fetch_extract, fetch_tar_flag, fetch_basename, fetch_ends, fetch_is_url, fetch_sha256_line |
src/core_pkg.mc | 30 | 8 | new: core_build.mc + pkg.mc, and the two subcommand() registrations |
src/sha256.mc | +29 | 20 | hex64 and sha256_file moved here |
src/deps.mc | +34 / -33 | 16 | dep_scan split into dep_hash_tree(dir, pk) + a two-line caller |
src/sysroot.mc | +16 / -122 | 9 | delegates: sysroot_extract is four lines, the downloader and the hex printer are gone |
src/core.mc, src/main.mc, src/core_build.mc, src/stubs.mc, src/driver.mc | +8 | 4 | the seventh part, mc_pkg_init(), fetch.mc in the list, two renamed calls |
1673 added lines in src/, 1268 of them neither comment nor blank, against 157 removed —
the amended § E table's ~150 + ~700 + ~24 + 6 + 2 = ~880. The excess is all in pkg.mc and it is
itemisable: the cache manifest writer (§ 4 asks for one and the table did not price it), the
immutability half of check (which has to read the registry's published copy and compare it row
by row), vendor's copier, the plan table, and this repository's accessor style — the state is one
arena record, so every field costs a one-line reader.
Outside src/: scripts/check-pkg.sh +374/-7, scripts/check-parts.sh +6, tools/bundle.list
+3 (mc/fetch, mc/pkg, mc/core_pkg), five new fixture packages and three consumer projects
under tests/pkg/, and two goldens (tests/golden/pkg-list.txt,
tests/pkg/sync/mc.lock.expect). stage0/ untouched, 2848/3000.
2. hex64 went to src/sha256.mc, and that is what made the ordering work. The plan was
hex64 in fetch.mc; it cannot be, because src/deps.mc prints a tree hash and is included
BEFORE src/driver.mc, while fetch.mc needs drv_spawn_ok and DRV_MAXARG and must come
after it. The choice was a forward prototype (the language has them — measured) or a better home.
The digest's TEXT form belongs beside the function that produces the bytes: hex64 and
sha256_file are sha256.mc's now, dep_hex/dep_file_hash/sysroot_hex are gone, and there
is one spelling of a digest in the compiler instead of three.
3. The tree hash has exactly one implementation, and it is the one mc build rehashes with.
dep_scan(pk) became dep_hash_tree(dir, pk): pk is the package's index for a locked tree and
-1 for a directory nobody locked, which is what mc pkg hash DIR, every tree sync unpacks
and every row check verifies pass. Only the file names THIS call recorded are hashed (the walk
starts at the table's length on entry), so the same table serves both and a second call cannot
pick up the first one's names. scripts/pkg-hash.sh is still the second implementation, in shell,
and check-pkg compares them on every run — now including mc pkg hash's own output.
4. Deviations from § 5 and § 7, on record.
- The default registry is
https://minicompiler.dev/registry, notraw.githubusercontent.com/schivei/mc-registry. The owner decided (2026-09-04) that a package SERVER at that host will produce the § 5 layout —index/<name>.toml, immutable rows — out of the git repositories registered with it. The compiler gains nothing for it: the reader is the same file-at-a-time reader a directory registry uses, and the constant is one function,pkg_default_registry(). Everything the spec says about a tap still describes the FORMAT. mc pkg updateismc update(D21), and it stays inside its major.go get -udoes not cross a major either, and raising a minimum across one is exactly the case D6 refuses to solve — somc updatewould otherwise be a command whose whole job is to producedifferent majors: no solver.mc pkg add NAMEwith no@still takes the newest non-yanked row of any major: a name nothing requires yet has no major to stay inside of.syncwith nothing to download does not need--yes. The plan is about downloads; when the build list is already on the disk,syncwrites the lock and says so. With anything to fetch it prints the plan and stops, which is D13 as written.- A
[replace]d package gets apathrow and nosha256, andsyncprintsreplaced geo: ../geo -- not pinned by mc.lock— the same sentencemc buildprints, from a different file.src/deps.mcneeded no change for it: a row with no hash is not checked. checkcompares against the registry's published copy, which is the only way "a row never changes" can be enforced by a program that is handed one file. With a DIR registry the comparison is free; with a URL one it needs--yes(there is nothing to compare against otherwise) and it is skipped, loudly, by doing nothing.checkdoes not cross[package].filesagainsttar -t(§ Risks 7, priced as optional). What it does compare is the tree hash, the package name and the set of[deps], which is what a wrong row actually gets wrong.
5. The fixture registry is BUILT by the script, and it is a directory. scripts/check-pkg.sh
tars each tests/pkg/src/<name>-<version>/ into $tmp/archives/ with the top directory GitHub's
tag archives have (strip = 1) and writes $tmp/registry/index/<name>.toml with url pointing at
those files and sha256 from scripts/pkg-hash.sh. Generating the index instead of checking it in
(the draft's § Acceptance 1 says "hashes hard-coded") keeps ONE source of truth for the fixture
hashes — the trees — while still crossing the two implementations, since the index is written by
the shell one and read by the compiler's. The locks in tests/pkg/*/mc.lock stay checked in and
stay compared.
The tar shim had to move. Step 2 put curl, wget and tar on PATH as programs that
fail if invoked, for the whole run. mc pkg sync UNPACKS an archive, so tar cannot be a stub for
it: there are two shim directories now, and pkg()/upd() in the script are the one place the
difference lives. The two downloaders still refuse in both, so nothing in the run can reach the
network — including every mc pkg command, since a DIR registry and a local url never spawn one.
6. New fixtures, and what each one is for.
| fixture | why it exists |
|---|---|
src/mathx-1.1.0 | the version MVS must select: higher than the project's minimum, lower than the newest |
src/mathx-2.0.0 | the other major |
src/mathx-2.0.1 | registered and yanked, and the NEWEST row — which is what makes mc pkg add mathx answering 2.0.0 a proof and not a coincidence |
src/plot-1.0.0 | requires mathx 1.1.0: the transitive minimum that wins |
src/heavy-1.0.0 | requires mathx 2.0.0: the two-majors refusal |
sync/ | the consumer mc pkg sync resolves, builds and runs (plot 110, exit 42) |
major/ | the project whose resolution is refused |
add/ | the file mc pkg add edits, with a comment between the sections so "every other byte" means something |
nopkg.mc | a compiler with every part but <mc/core_pkg> |
tests/pkg/app and the four step-2 consumers were not touched: their locks, their hashes and their
geo -> mathx 1.0.0 edge are unchanged, so the step-2 half of the gate still measures what it
measured.
7. The gate: scripts/check-pkg.sh, 63/63 (was 31/31), inside make check. The 32 new lines,
in order:
ok mc pkg hash geo/mathx/plot: mc and scripts/pkg-hash.sh agree
ok the plan names 2 archives with their hashes and nothing was fetched
ok an https registry with no snapshot: the plan is its index files, nothing spawned
ok sync --yes: mc.lock is mc.lock.expect, mathx at 1.1.0 (MVS, not 1.0.0 and not 2.0.0)
ok two manifests exist, written last, with [[file]] rows
ok no unselected version was fetched (1.0.0, 2.0.0 and the yanked 2.0.1 stayed put)
ok a second sync downloads nothing and rewrites the same lock, byte for byte
ok <plot> through the lock's lib entry: exit 42, 'plot 110' (mathx 1.1.0)
ok a second fetch into another cache writes the same lock
ok two fetches, two caches, byte-identical objects
ok vendor: deps/plot and deps/mathx hold mc.toml plus [package].files
ok the vendored tree gives the same object as the cache road, byte for byte
ok mc pkg list: every row 'vendored', and it matches tests/golden/pkg-list.txt
ok mc pkg verify: verified 2 packages against mc.lock
ok majors: mc: mathx: 1.0.0 and 2.0.0: different majors: no solver
ok a failed fetch: mc: cannot open: ..., no manifest
ok a wrong hash: checksum mismatch, the listed files unlinked, no manifest
ok a build over the debris: mc: mathx 1.1.0 is not fetched
ok mc pkg add mathx@1.0.0: add.toml.expect byte for byte (3 lines added)
ok mc pkg add mathx: the newest NON-YANKED row, 2.0.0, never the yanked 2.0.1
ok mc update mathx: 1.0.0 -> 1.1.0, the newest of ITS major (2.0.0 is not an update)
ok mc pkg check geo.toml --yes: both rows re-derived from their archives
ok mc pkg check without --yes: what it would check, and nothing downloaded
ok mc pkg check refuses name = "mc" / a wrong hash (exit 2) / an edited published row
ok the pkg-less probe compiler builds
ok no <mc/core_pkg>: no 'mc pkg' and no 'mc update' usage line
ok no <mc/core_pkg>: 'mc pkg' is an ordinary file name, exit 1
ok a compiler with no fetcher builds the vendored project, same object
check-pkg: 63/63
scripts/check-parts.sh covers <mc/core_min> + <mc/core_pkg> (the part stands on its own —
it is the one part that names another, core_build.mc, and the once-only include is what makes
that legal) and the seven-part spelling still cmps equal to <mc/core>.
8. Inertness and the numbers. scripts/check-inert.sh /tmp/m44pre/build/mc1 build/mc1 (pre =
a mc1 built from origin/main 8c31a0e): 33 objects identical (tests/*.mc and src/mc.mc)
plus byte-identical artefacts for examples/api, examples/lang, examples/conc,
examples/desktop and examples/kernel through the taught compiler each side builds. Nothing
this step adds is reachable from a source: mc build calls no function in src/pkg.mc.
The globals budget, the one seed limit this step could have broken: 439 -> 440 of 512
(85%), and check-limits fails at 460. src/pkg.mc costs exactly one (pk, the whole registry,
build list, plan and option state behind pk_state()), src/fetch.mc none. mc limits gains no
row either: the four tables in pkg.mc scale with the dependency graph, which the developer wrote,
so they double in place — the same argument src/deps.mc's file table and M17's MAXTARGETS make.
make bundle re-run BEFORE bootstrapping: 91 files, raw 1133380 -> lz 528927, blob 530054 B.
make check green end to end (RC 0, zero FAIL): budget 2848/3000, test 32/32, check-lex
143/143 (3 skipped), check-ast 144/144, check-asm 144/144, check-obj 32/32 identical to the
frozen seed, check-bundle (reproducible + fresh, lz round trip 115 cases), bootstrap at a
fixed point (cmp build/mc2.o build/mc3.o, 1228304 B; the --dump-asm diff between mc1 and
mc2 is empty), check-surface 32/32 + inert, test-exe 32/32, check-mc 15/15,
check-standalone, check-parts, check-toml 10/10, check-build 53/53, check-pkg 63/63,
check-stubs 9/9, check-sysroots (13 rows), check-limits 17/17 under 90%,
check-minimal, test-linux 41/41 + 38/38, test-linux-exe 44/44 musl + 44/44 gnu,
test-linux-x86_64-exe 41/41 musl + 41/41 gnu, test-windows 42/42 + 40/40 objects,
check-examples, check-lang, check-conc, check-desktop, check-float (13/13 on each of the
five legs), check-wide, check-kernel, check-avr, test-sandbox 55 ok, check-docs
(197 symbols, 36 flags, 27 TOML keys, 10 directives, 51 samples, 358 links), site 89
pages + check-site 0 link problems. make check-linux-host RC 0 over all four cells.
9. The five goldens rewritten once, each after its own criterion: tests/golden/mc2.sha256
9e00398d…11420a -> cede0b38995e99c0a9c33b08f48daadc283189f3c9a4f43fce7368c75b407284 (after the
empty --dump-asm diff and cmp build/mc2.o build/mc3.o); the Linux pair deleted and re-recorded
by make check-linux-host (Docker, both architectures, both libcs, each after its own fixed point
mc2l.o == mc3l.o and with the cross proof green) — mc2-linux-arm64.sha256
e4c876dd3aca86ded5d62fb6639d4be6da7dbb981b45bf91107231c75fcdbc02, mc2-linux-x86_64.sha256
3f036b4d627ba54904729b3385033307c86113670368a0422bfcd429ff2d2012; the Windows pair cross-computed
per tests/golden/README.md and reproduced byte for byte by build/mc2 —
mc2-windows-arm64.sha256 70a2259d0152839d64f86a41e86d911b4c4827a13b54dc5ba0b35d010e268309
(1254075 B), mc2-windows-x86_64.sha256
98cf8605af38b44720ddbe43c61faeab3e53724f3615ea920cc611c11ca4fde5 (1287987 B).
10. What step 3 deliberately did NOT do, so step 4 knows where the seam is: there is no
src/install.mc, no src/upgrade.mc, no src/bundle_empty.mc and no slim entry — mc install,
mc upgrade, LATEST, the two release flavours and scripts/release-assets.sh --libs/--slim are
steps 4 and 5. The mc package on disk (A4) still has a reader and no writer: tests/pkg/nobundle.mc
plus a <libs>/mc/v<version>/ laid out by the check script is how step 3 of the resolution order
is still exercised. Nothing here writes into <libs>/mc/, and mc is refused as a package name
everywhere it could be asked for.
Docs: docs/reference/packages.md § 9-10 (the registry and the server that produces it, MVS, the
fetch, the lock writer, vendoring, check), docs/reference/cli.md § 3d, docs/reference/toml.md
([registry].url is read now; package.repo), docs/reference/diagnostics.md § 13 (seventeen
more rows), docs/reference/bundle.md (the seventh part and the three names),
docs/guide/25-packages.md (add, update, vendor, and the author's road ending in
mc pkg check), docs/build.md § M44.
Implementation notes — the supply-chain review (post-step-3 batch) #
Six findings, ranked by the reviewer; the first three were this milestone's own claims failing.
Everything below was reproduced on the pre-batch compiler first, then fixed, then re-reproduced as
a refusal. The whole code change is src/deps.mc, src/fetch.mc, src/pkg.mc and one line of
src/sysroot.mc; stage0/ is untouched.
1. CRITICAL — [package].files entries were never contained #
What reproduced. A files entry is an arbitrary string in an mc.toml that arrived inside a
downloaded tree, and every consumer joined it with path_join/path_norm and used it unchecked.
path_join DISCARDS its base when rel is absolute and path_norm resolves .. with no floor,
so the entry was a free path.
- Read, on every
mc build:dep_line/dep_hash_tree. A package installed withfiles = ["evil.mc", "../../../payload.txt"]hashed a file three directories above its own tree and the build came outrc=0. - Write,
mc pkg vendor:pkg_copy_file/pkg_copy_treejoined the same entry againstdeps/<pack>/and copied. With the project two directories deeper than the installed tree, the source resolved to<libs>/../payload.txtand the destination to a file OUTSIDE the project — measured:/tmp/m44repro2/a/b/payload.txt, contentPAYLOAD, written bymc pkg vendor, exit 0. - Delete,
mc pkg sync --yes: on a hash mismatchpkg_unblessre-read the just-extracted, untrustedmc.tomlandunlinked everything it listed. A registry row with a deliberately wrongsha256and an archive whosefilescarried../../../victim.txtprintedmc: checksum mismatch for evil 1.0.0— andvictim.txtwas gone. The attacker never needs a hash that passes: the wrong hash IS the trigger.
The rule now. dep_rel_ok(rel, dirok) + dep_under(dir, rel) in src/deps.mc, behind one
refusal, dep_file_check(dir, rel, what) →
<pack> <ver>: files entry escapes the package: <entry>, exit 2. An entry may not be empty or
absolute, may not carry a ., .. or empty component, may not hold a backslash or a byte below
0x20, and its normalised join must still start with the package directory plus a separator (the
component walk and the arithmetic, both, on purpose).
The five copies of "read [package].files into an array" became one, dep_read_files(dir,
what, pnames), which is where the check lives — so dep_hash_tree, pkg_write_manifest,
pkg_copy_tree and mc pkg hash cannot drift apart again, and dep_manifest_bad (which reads
the CACHE manifest, a different file) got the same test inline. The check runs after the
toml_pop, so a refusal cannot leave the project's own table swapped out (§ Risks 6).
pkg_unbless no longer reads that list at all. It deletes what the EXTRACTION wrote: the
member table src/fetch.mc fills while validating the archive (finding 2), plus mc.toml. mc
has no rmdir, so directories stay — a directory with no mc.toml is is not fetched, which is
the outcome that was wanted.
2. HIGH — fetch_extract trusted tar #
What reproduced. A tag archive whose member link-1.0.0/secret.mc is a symlink to /etc/hosts
extracted as that symlink; the tree then hashed and vendored the LINK TARGET, so
mc pkg vendor wrote the host's /etc/hosts into deps/link/secret.mc — measured, exit 0. (The
hash makes this a targeted rather than a blind read: the attacker has to know the victim file's
content to publish a row that verifies, or use a [replace]d or vendored tree, which carries no
hash at all.) A .. member is refused by the bsdtar on this host — which is exactly the point:
the guarantee was the tool's, not mc's, and busybox tar, an old GNU tar or a Windows tar.exe are
not the same program.
The rule now. fetch_check_members(archive, dest, strip) runs before every extraction. Two
listings, because one cannot answer both questions: tar -t...f gives the names one per line
(the only way to read a name containing a space) and tar -tv...f gives the ls-style type
character in column 1 (the only way to see that a member is a link). Same order, so line k
describes line k; a disagreement in the number of lines is itself a refusal. A member is refused
when it is a link (l/h), when dep_rel_ok says no before the strip, or when it leaves dest
after it — <archive>: member escapes the archive: <member>, exit 2, and the archive is unlinked.
The listing output is itself capped at 1 MiB. After the extraction every listed regular file has
to be openable under dest, which is what says tar wrote what tar said it would; that check is
skipped when the caller named a subset of members, which only src/sysroot.mc does.
Not done, on record: the extraction still names no members. Handing the validated list back to
tar means writing it into fetch_extract's space-separated members argument, which no member
whose name contains a space survives — and the archive between the listing and the extraction is a
file mc has just written and does not re-fetch.
3. MEDIUM — the hash line format was not injective #
<hex><two spaces><path><LF> is Go's dirhash.Hash1 verbatim, and a path carrying a newline
writes two lines. Honest limit of the finding, measured: a forgery is not constructible, because
the FIRST line of the stream is the digest of mc.toml and mc.toml is where the file list is
written — two different lists mean two different first lines unless sha256 collides. The format
is still unable to say which list it hashed, and the primitive (a newline in a path) is
attacker-controlled, so both halves were closed: dep_rel_ok refuses any byte below 0x20
outright, and the line became
<64 hex><space><decimal byte length of the path><colon><path><LF>
Every hash moves, which costs nothing today (no package is published): scripts/pkg-hash.sh — the
second implementation, in shell, that check-pkg crosses against the compiler on every run —
follows the same rule, and the five fixture locks, mc.lock.expect and tests/golden/pkg-list.txt
were re-recorded from it. pkg-hash.sh --files keeps printing <hex> <path> pairs (that is what a
cache manifest's [[file]] rows are made of, not hash lines) and gained --lines, which prints
the canonical lines so check-pkg can read the format itself.
4. MEDIUM — pkg_check_immutable had two bypasses #
Against a URL registry it returned — silently answering "immutable" — both when --yes was
absent and when the fetch of the published copy failed for ANY reason, so a network hiccup, a
proxy error or a 500 switched off the one rule the registry has. Now: without --yes,
check needs --yes to compare against the published index: <name>, exit 2 (it refuses to answer
rather than answering by doing nothing). With --yes, only curl -f's 22 and wget's 8 — "there
is no such file", i.e. a new package — return; every other code is
check: cannot read the published index for <name>: <reason>, exit 2, with pkg_fetch_reason
turning the code into words.
5. LOW-MEDIUM — a refusal inside the hash left the tree behind #
dep_line's a file the package lists is missing called dep_die directly, so an archive that
lists a file it does not ship exited from inside dep_hash_tree with the extracted tree still on
the disk and no pkg_unbless. Finding 1's new refusal would have had the same shape.
Which road was taken: "collect the error". dep_hash_soft(1) puts src/deps.mc in soft mode
— dep_soft_die records the first problem in the state record instead of exiting, and
dep_hash_tree answers 0 — and pkg_fetch_one reads it back with dep_hash_err() /
dep_hash_errdet(), unblesses, and dies with the SAME message it would have printed. Clearing the
directory before a re-fetch was the alternative and is weaker: it leaves the debris until somebody
re-fetches. dep_hash_label() goes with it, so a refusal during a fetch says evil 1.0.0 and not
the cache directory's path — the tree is not in any lock yet, so dep_pkg_what has nothing else to
call it.
6. LOW — no size cap #
fetch_get gained a cap argument (0 = none) and FETCH_TOOBIG: 64 MiB for an archive
(FETCH_MAXARCHIVE), 1 MiB for a registry index file (FETCH_MAXINDEX), applied to the local-path
branch before the copy and to a downloaded body after it (neither curl nor busybox wget has a
size limit mc can rely on), with the file unlinked. mc: larger than the cap of 67108864 bytes:
<source>, exit 2. src/sysroot.mc's one call site passes the archive cap and reports it before
its own rc < 0 branch, which would otherwise have called it "no downloader".
The gate #
scripts/check-pkg.sh 63/63 → 78/78, fifteen new lines: the four shapes of an escaping files
entry (../, absolute, a/../../x, a newline) each refused at exit 2 with a canary asserted
untouched; mc pkg vendor refusing one, with the project placed two directories deeper than the
installed tree so the source and the destination resolve the entry to DIFFERENT files and a copy
that happened is a file that appeared; the wrong-hash unbless with a ../ entry, canary alive and
no manifest; three crafted archives (symlink, hard link, .. member) and one ordinary one that
must still extract; mc pkg check without --yes and with an unreadable index; a 68 MB archive
over the cap; and the hash line read for its shape. The three crafted-archive cases self-skip with
an ok line when the filesystem or the local tar cannot produce the fixture. rtar() was added
because the script's own tar is a shim that fails if invoked — that is how mc build is PROVED
never to download, and the fixtures have to be built around it.