Projects and builds

This page follows a project from its .queen file to the executable. The user reference for the file is Project files (.queen); here the subject is what larva does with it. The code lives in two places:

Larva.Build knows nothing about languages: it gets the list of source extensions from the installed language packs (QueenProject.SourceExtensions) and hands file lists to the CLI. Packages are resolved through a callback (QueenPackageResolver) that Hive provides, so Larva.Build does not depend on Hive either; see Hive internals.

Reading a .queen file

The grammar

QueenParser is a hand-written, single-pass reader with line and column tracking. It never throws: every problem becomes a QueenDiagnostic with a QNxxx code, and it recovers by skipping to the end of the line.

document  := ws "queen" version ";" chunk*
chunk     := "chunk" word ":" entry*            (up to the next "chunk" keyword or the end)
entry     := word ":" value [","]               (an entry also ends at the end of its line)
value     := list | quoted | raw
list      := "[" (value ("," value)*)? "]"       (may span lines)
quoted    := '"' text '"' | "'" text "'"        (one line; \" or \' escapes the quote)
raw       := characters up to "," or end of line (or "]" inside a list), trimmed;
             may contain $[ expression ] parts
word      := letters, digits, "_", "-", "."
comment   := "//" to end of line, or "/* ... */", only at the start of a line or after whitespace

The rule for comments is why Tests/*.ant and https://www.antril.org/hive-api are plain values: a / that does not follow whitespace never starts a comment.

Values

Unquoted text is classified by QueenParser.Classify, in this order:

Each value keeps its line and column, which is how diagnostics point at the exact value (App.queen(8,18): error QN124 …).

$[…] is an expression in the project's language: QueenProject.Load takes a QueenEvaluator that would run all of a file's expressions in one go and re-classify the results (a value that is only $[…] takes the kind of its result). Every caller in larva passes no evaluator today, so a $[…] in a project is reported as QN110 ("needs a frontend to run"). Not implemented yet: evaluating $[…] in larva build.

Repeatable keys

A key may appear once per chunk, except the ones QueenParser.Repeatable allows; a second one is QN011:

A chunk name may appear only once in a file (QN008).

Diagnostics

Errors stop the load (QueenProject.Load returns null); warnings are printed and the build goes on. A chunk with a name larva does not know, such as chunk NOTES:, is kept in QueenDocument.Chunks for other tools and is not checked.

Loading a project

QueenProject.Load(path, evaluator, diagnostics, packages) does, in order:

1. Parse the file (or, in the editor, the unsaved buffer given to LoadText).
2. Evaluate $[…] (see above).
3. Read PROJ, BUILD and HIVE into typed properties (Name, Kind, Version, Gc, Backend, Target, Optimization, Cpu, Reflection, BorrowCheck, AllowUnsafe, Generics, DefaultAccess, Output, Hive…). If this produced errors, stop.
4. Find sources (SOURCES).
5. Expand FILES and RESOURCES into concrete files.
6. Follow DEPS, loading each dependency project recursively. A list of the files being loaded detects cycles.
7. Resolve packages through the QueenPackageResolver, and add what it returns as dependencies, libraries or native libraries.

Finding sources

With scan not false, FindSources walks the project's folder in ordinal order and takes every file whose extension belongs to an installed language. It does not go into:

• hidden folders (a name starting with .), bin and obj;
• folders that are links (reparse points), so a link cannot pull another tree in;
• folders that contain a .queen file of their own: a nested project is a separate project;
• folders it may not read;
• anything matching ignore (BUILD) or exclude (SOURCES).

Then each include is expanded and added (if it is not excluded and not already there). Patterns are turned into regular expressions: **/ is "any number of folders", ** is "anything", * and ? stay within one folder, and matching ignores case. Everything is sorted ordinally so that two machines compile files in the same order.

AllSources() returns the dependencies' sources first, then the project's own, each file once. That list is what the language's frontend compiles as one assembly.

FILES and RESOURCES

ReadFiles expands both chunks once, at load time:

• FILES: each copy pattern gives files; each keeps its path relative to the project, under into if given. A file from outside the project's folder goes to the top, by its file name. Duplicate targets are dropped. The result is CopiedFiles, a list of (Source, Target).
• RESOURCES: embed patterns name each file by its path under the project (data/greeting.txt). A key other than embed names one file by that key (logo: images/logo.png is the resource logo). The result is Resources, a list of (Name, File) in a stable order; AllResources() adds the dependencies' resources, and the project's own win on a name clash.

Resources go into the program's HIR assembly: the build reads each file's bytes into an HirResource and sends them, base64-encoded, in the compile request to the language pack. System.Resources reads them back at run time from the assembly. Copied files and native libraries go next to the build after it succeeds (CopyNextTo), and only when the target is missing, a different size, or older than the source.

DEPS

Each DEPS entry is resolved relative to the project's folder:

A dependency project contributes its sources (compiled together with the program), its libraries, its native libraries and its resources. The All…() methods walk the dependency tree and return each item once.

Settings: who wins

Four layers decide a setting, the first that says something wins:

   larva build --backend x64 -O2 ...        1. the command line
              │
   chunk BUILD: backend: llvm               2. the project's BUILD chunk
              │
   %APPDATA%\Larva\defaults.json            3. the machine's defaults (larva config)
              │
   LarvaDefaults.BuiltIn, SourceBuild       4. larva's own

The machine's defaults

LarvaDefaults.ConfigPath is LARVA_CONFIG when set, else %APPDATA%\Larva\defaults.json on Windows and $XDG_CONFIG_HOME/larva/defaults.json (or ~/.config/larva/defaults.json) elsewhere. It is a flat JSON object of strings:

{
  "backend": "llvm",
  "optimization": "0"
}

Only the keys in LarvaDefaults.Keys are read: the BUILD keys that make sense for a whole machine (gc, backend, target, optimization, cpu, reflection, borrowcheck, unsafe, generics, access.default) plus hive.provider, languages.provider and languages.trust. Paths and lists (output, ignore, stdlib) stay per project. A missing or unreadable file means "no defaults"; a build never fails because of it. larva config <key> <value> validates the value against QueenSchema.Build (LarvaDefaults.Problem) before writing.

LarvaDefaults.BuiltIn holds what applies when the file is silent: borrowcheck: off, and the default languages registry. The borrow checker is off unless a project or the machine turns it on.

How the layers are merged

The project reader does not special-case defaults. LarvaDefaults.Entries turns the effective defaults into the text of a synthetic chunk BUILD:, parses it with the same parser, and QueenProject.Read walks that chunk before the project's own BUILD chunks. Each key simply overwrites the property, so the project's entry, read later, wins. QueenProject.Sets(key) answers "did the project or the machine say this?", and the build uses it to tell a chosen value from a field's initial one.

The command line is applied last, in SourceBuild.Build: an option given there (--backend, --target, -O0…-O3, --borrowcheck, --cpu) is never replaced by the project. For the project's values the build reads only what Sets confirms; otherwise it keeps its own built-in choice: the LLVM back end, a native target, -O3.

A run with a private LARVA_CONFIG shows the layers. The project's BUILD chunk sets nothing; the machine says LLVM at -O0:

$ larva config optimization 0
optimization: 0
$ larva config backend llvm
backend: llvm
$ larva build -o bin/App.exe
bin/App.exe (llvm back end, O0, entry App::App.$Module.Main)
$ larva build -O2 -o bin/App.exe
bin/App.exe (llvm back end, O2, entry App::App.$Module.Main)
$ larva config --unset backend
backend: -
$ larva config --unset optimization
optimization: -
$ larva build -o bin/App.exe
bin/App.exe (llvm back end, O3, entry App::App.$Module.Main)

The last build falls back to larva's own -O3.

larva new writes the effective defaults into the new project's BUILD chunk (LarvaDefaults.ForNewProjects, overlaid with the machine's values), so a new project builds the same on a machine with other defaults. With an empty defaults file:

queen 1.0;

chunk PROJ:
   name: Demo,
   type: exe,
   version: v1.0

chunk BUILD:
   gc: default,
   backend: llvm,
   optimization: 3,
   borrowcheck: off,
   unsafe: allow

What reaches the frontend

The language pack never sees the .queen file. QueenProject.FrontendOptions() reduces it to the few things a frontend needs, sent in the compile request as options:

{ "borrows": false, "access": "file", "unsafe": false, "safe": ["…/Program.ant"] }

• borrows: whether &x and reference types are allowed (false when the borrow checker is off, because a borrow nothing checks would be unsound);
• access: access.default, when set;
• unsafe and safe: with unsafe: deny, the files that may not use unsafe. Only the files of projects that deny it are listed, so a dependency keeps what its own project allows.

Without a project, LarvaDefaults.FrontendOptions() builds the same object from the machine's defaults. --borrowcheck overrides borrows for one command.

The build pipeline

larva build with no inputs builds the project you are in (RunCommand.Find: the first .queen in the current folder or the nearest folder above). With a .queen among the inputs, SourceBuild.Build replaces it by the project's sources and collects everything else it implies:

Then the phases run in order. --timings prints each one's wall time:

 .queen ──► QueenProject.Load ──► sources, resources, libraries, options
                                         │
   System.Core ─────────────────┐        │
   GC-domain dependencies ──────┤        ▼
   assemblies (.dll/.lva) ──────┼──► frontend (language pack: "compile") ──► program .lva
   System.Bindings (if used) ───┘                                                │
                                                                                 ▼
                    HIR link (HirLinker: core + domains + assemblies + program)
                                         │
                                 lowering (HirLowerer → IR module)
                                         │
                                 borrow check (when on)
                                         │
                       ── --check stops here: "no errors (…)" ──
                                         │
                                 IR checks (Compiler.CompileObject)
                                         │
                                 IR link (Linker.Link, entry, domains)
                                         │
                     native (llvm | x64)  or  native (wasm | wasm-llvm)
                                         │
                                 copy FILES and lib: next to the output

--check runs up to the borrow check and prints the phases in its summary:

$ larva build --check --borrowcheck enforce
no errors (1 file; System.Core (compiled library) 63 ms, frontend 1185 ms, HIR link 59 ms, lowering 752 ms, borrow check 141 ms; total 2208 ms)

How BUILD reaches the phases

gc: custom is read (CustomGc, GcCollector) but lowers like gc: default. Not implemented yet: a project's own collector (System.IGarbageCollector and GC.Install do not exist in System.Core).

A small project, built twice

queen 1.0;

chunk PROJ:
   name: App,
   type: exe,
   version: v1.0

chunk BUILD:
   backend: x64,
   optimization: 1

chunk FILES:
   copy: assets/**,
   into: config

chunk RESOURCES:
   embed: data/**

import System;

into namespace App
{
	Main()
	{
		Console.WriteLine(Resources.ReadText("data/greeting.txt") ?? "missing");
	}
}

$ larva build --timings -o bin/App.exe
bin/App.exe (x64 back end, entry App::App.$Module.Main)
timings: System.Core (compiled library) 49 ms, frontend 1379 ms, HIR link 72 ms, lowering 785 ms, IR checks 321 ms, IR link 126 ms, native (x64) 1780 ms; total 4529 ms
$ larva build --timings -o bin/App.exe
bin/App.exe (x64 back end, entry App::App.$Module.Main)
timings: System.Core (compiled library) 39 ms, frontend 1100 ms, HIR link 58 ms, lowering 664 ms, IR checks 358 ms, IR link 125 ms, native (x64) 1000 ms; total 3358 ms
$ bin/App.exe
Hello from a resource

The build folder then holds bin/App.exe and bin/config/assets/settings.json (the FILES copy). The second build is faster only in the native phase, where the system linker and the runtime's object come from warm caches. Everything else is done again: larva build itself has no per-project incremental compilation (see below).

The same project, with System.Core compiled from its sources because --no-cache skips the cache (in a working copy of Larva, where there is no System.Core.dll):

$ larva build --timings --no-cache -o bin/App.exe
bin/App.exe (x64 back end, entry App::App.$Module.Main)
timings: System.Core (compiled now) 1584 ms, frontend 89 ms, HIR link 45 ms, lowering 552 ms, IR checks 228 ms, IR link 134 ms, native (x64) 886 ms; total 3528 ms

Here the program's frontend phase drops to 89 ms. The language pack had just compiled System.Core in the same process, so the most likely reading is that, in the other runs, most of the frontend time of a small program is the pack taking in System.Core's declarations rather than compiling the program itself.

Caches

Larva caches the things that are expensive and rarely change. All of them live under %LOCALAPPDATA%\Larva and are keyed by a hash of their inputs. The toolchain files are written to a .partial file and then moved into place, so two builds at once never see half a file.

An installed larva reads System.Core from System.Core.dll next to it (a PE image whose .lva section is the library; see The .lva format). It checks the first bytes, LVA and the format version, and refuses a library from another Larva. That is why an installation always reports System.Core (compiled library), and why --no-cache changes nothing there: it only skips the source cache of a working copy. --core <dir|file> and LARVA_CORE choose another System.Core, either a compiled library or a folder of sources.

Because the keys include the inputs, a change to a source or to the frontend gives a new file instead of invalidating the old one. Nothing removes old entries; deleting the folder is safe.

What is not cached

The design called for an incremental cache per assembly, keyed by hash, with .lvo objects kept as an internal IR cache. That is not how larva build works today: every build compiles all of the program's sources in one compile call, links, lowers and generates code again. The bin/ folder holds only outputs; larva does not write an obj/ folder (it skips obj/ when scanning only so that build output of other tools is never taken for sources). Planned: per-assembly incremental builds.

The cheap path is larva run, which skips the build entirely when nothing changed.

How larva run decides to rebuild

RunCommand.Run finds the project (the .queen given, the one in the folder given, or the nearest from the current folder up), loads it, and computes the output path the same way larva build does. It then builds when any of these holds:

• --rebuild was given;
• any build option was given (--backend, -O1, --target…): the existing output may have been built differently;
• the output does not exist;
• UpToDate finds an input newer than the output.

UpToDate compares write times. The inputs are:

The build's own report goes to standard error, so the program's standard output stays clean. The program runs in the current folder and larva exits with its exit code. A WebAssembly project runs with node.

$ larva run
Hello from a resource
$ larva run --rebuild -O0
…
Hello from a resource

The first run started the program built earlier, without building. The second built again (its report, on standard error, is the … line).

Dependencies with another GC

A program and a library may use different memory managers: the library's .queen says gc: refcount while the program uses the tracing collector. Compiling the library's sources into the program would give them the program's GC, so SourceBuild.Build treats such a dependency as a GC domain of its own. The design for this is in GC domains; this is the build side.

For every dependency whose Gc differs from the root project's, DomainLibrary:

1. compiles the dependency's sources (with its own dependencies' sources) against System.Core, with the dependency's own frontend options;
2. links that assembly with System.Core at the HIR level;
3. lowers it with Gc = dependency.Gc and ExportAll = true, so everything inside it is exported for the program to declare;
4. compiles and checks the IR object for that GC.

It returns the IR object and the .lva bytes. The program then:

• is compiled with the .lva as a reference, and with external set to the dependency names, so the frontend knows which types belong to another domain;
• is HIR-linked with [core, domain libraries…, assemblies…, program];
• is lowered with ExternalAssemblies set: static, non-generic functions of those types whose signatures carry only numbers, bools and pointers to them become imports (Linkage.Import) bound to the domain's object; the rest of such a type that the program needs is compiled in the program's own domain, and a use that would carry an object across is an error;
• is IR-linked with the domain objects and MixedDomains = true.

queen 1.0;

chunk PROJ:
   name: Counter,
   type: library,
   version: v1.0

chunk BUILD:
   gc: refcount

queen 1.0;

chunk PROJ:
   name: Main,
   type: exe,
   version: v1.0

chunk BUILD:
   backend: x64

chunk DEPS:
   proj: ../Counter

Counter.ant has Tally.Sum(i32 n) : i32, which builds a List<i32> inside the library and returns a number; the program prints Tally.Sum(10):

$ larva build --timings -o bin/Main.exe
bin/Main.exe (x64 back end, entry Main::Main.$Module.Main)
timings: System.Core (compiled library) 55 ms, Counter (RefCount) 2367 ms, frontend 31 ms, HIR link 3 ms, lowering 74 ms, IR checks 53 ms, IR link 191 ms, native (x64) 985 ms; total 3773 ms
$ bin/Main.exe
55

The Counter (RefCount) phase is the whole domain library. With gc: default in Counter.queen, the same build has no such phase: the library's sources go into the program's single frontend call.

Only functions over numbers, bools and pointers to them cross between domains. A call whose result is an object of the other domain is refused by the frontend with AN1079, which says that the function "belongs to another GC domain" and suggests packing strings and bytes with System.Exchange. Only first-level dependencies become domains: a dependency's own dependencies are compiled inside it.

The editor's view of .queen

The language server reads the project the same way (QueenProject.LoadText with the open buffer, so an unsaved .queen applies at once), with HiveResolver for packages. Completion and hover in .queen files come from QueenFiles, which reads only QueenSchema: the chunk the cursor is in is the last chunk NAME: line above it, chunk | completes chunk names, key: | completes the key's values (or its example), and hovering a chunk or a key shows its description and allowed values. The schema is the single list of keys; the reader, the editor and larva config all check against it.

Environment variables