Language packs

Larva contains no programming language. larva knows Larva HIR, its own tools and the .queen project format, and nothing about the syntax of any source file. Each language is a language pack: a folder that holds the language's frontend (source to HIR), its view (HIR written back as that language's source) and its editor services, plus a manifest, language.json.

Antril is one such pack. The same larva builds, explains and edits any language whose pack is installed, and a library written in one language is used from another as a compiled .dll (see The .lva format).

                       larva (one process)
 ┌──────────────────────────────────────────────────────────────┐
 │ build / lib / run / new / bind / types / show / search / lsp │
 │        │                                                     │
 │        │ JSON request  {"method": "compile", "params": {…}}  │
 │        ▼                                                     │
 │ LanguagePack.Call ──► larva_language_call(request, n, &m) ───┼──► pack library (Antril.dll)
 │        ▲                                                     │      frontend, view, editor
 │        │ JSON answer   {"result": …} or {"error": "…"}       │      services
 │        └──────────── larva_language_free(answer) ◄───────────┼──
 │                                                              │
 │ HIR linker, lowering, IR, back ends (the language is done)   │
 └──────────────────────────────────────────────────────────────┘

Why packs

The design document states the goal plainly: Larva is the platform and a language is only a frontend that turns text into typed HIR. Every tool (build, packages, the language server, bindings, the type browser) works on HIR assemblies, so a new language gets all of them by answering a handful of requests. Keeping the language out of larva also means:

• A language ships and versions on its own. The Antril pack lives in its own repository and is loaded at run time; larva does not reference it.
• larva can check what it loads. A pack is code that runs inside larva's process, so every file of it is hashed in its manifest and the manifest can be signed.
• A program has one source language, and other languages enter as libraries. LanguageRegistry.For refuses a build whose sources belong to two packs: "the sources are in A and B: a program is compiled from one language (the others' code as libraries, .dll)".

What a frontend implements

A pack is one library with two exported functions (the ABI) that answers requests. What each request asks of the language:

The language decides names, overloads, types, inference and its own sugar. It emits HIR that is already typed and resolved. How a virtual call dispatches, how a closure is built, how async becomes a state machine: Larva decides that during lowering, the same way for every language.

The view and the editor services are mostly shared code. A pack builds them on Larva.Hir.Services (a LanguageView subclass, a HirCatalog, a LanguageService and an IDocumentAnalysis of its own): see The catalog and language views. Antril's pack is AntrilLanguage (the request dispatcher) and LanguageExports (the two exported functions) in src/Antril/Pack/ of its repository.

The ABI

The manifest's abi names the version of the requests a pack answers. This larva makes the requests of ABI 1 (LanguagePack.Abi = 1); a pack with another abi is not loaded.

ABI 1 is two C functions, found by name in the pack's library:

/* One request: `request` is UTF-8 JSON of `length` bytes. The answer is UTF-8 JSON,
   allocated by the pack; its length goes to *resultLength. */
uint8_t *larva_language_call(const uint8_t *request, int32_t length, int32_t *resultLength);

/* Releases an answer larva has read. */
void larva_language_free(uint8_t *result);

The request is always an object with the method and its parameters:

{ "method": "compile", "params": { "name": "Program", "sources": [{ "path": "C:/app/main.ant" }], "references": [], "library": false } }

The answer is {"result": …} or {"error": "message"}. LanguagePack.Call serializes the request, calls larva_language_call, parses the answer, calls larva_language_free on the buffer (always, even when the answer is not JSON), and turns an error into a LanguagePackException whose message larva prints. The Antril pack answers one request at a time (it takes a lock around its dispatcher) and turns any failure into an error answer, so nothing unwinds across the boundary.

Assemblies in requests

Many requests carry HIR assemblies (System.Core, the libraries a project names, a program just compiled). An assembly travels in one of two forms:

LanguagePack.Reference(path) and LanguagePack.Reference(bytes) build these two forms.

Every request and its fields

The fields below are what larva sends and what it reads back. A pack may answer more (Antril adds types to compile's answer, which larva lib prints).

info

Sent once, when the pack is loaded (LanguagePack.Load). Takes no parameters.

compile

source

Writes an assembly as source. larva new uses it to write a template in the chosen language, and larva bind --source to write bindings as readable source.

The answer is {files: {"File.ext": "text"}}, each file with its own imports at the top, once.

catalog

Runs larva types, show or search through the language's view. The command logic itself is shared (CatalogCommands in Larva.Hir.Services), so the output is the same shape in every language.

The answer is {output, error, exit}: larva writes output to stdout and error to stderr and exits with exit.

lsp.*

larva lsp keeps the protocol, the open documents, the projects and the decompiled files, and asks the pack of each document for its features. The requests:

workspace.key identifies a project's set of libraries: the pack builds the workspace (System.Core, the project's assemblies, dependencies compiled for another GC) once per key and reuses it for every document of that project. Details are in The language server.

There is no templates request: a template is HIR, and the source request writes it (see Templates).

The manifest

language.json sits at the root of the pack folder. This is the manifest of the Antril pack after larva languages pack with a key:

{
  "id": "antril",
  "name": "Antril",
  "version": "0.8.0",
  "description": "Antril for Larva: its frontend (Antril source to Larva HIR), its view (HIR written as Antril) and its editor services",
  "author": "Antril Organization",
  "license": "Proprietary",
  "homepage": "https://antril.org",
  "extensions": [
    ".ant"
  ],
  "icon": "ant-file.ico",
  "abi": 1,
  "larva": "0.8.0",
  "entry": "Antril.dll",
  "files": {
    "Antril.dll": "cb1cf4db81a4326caf4c0f20c3b5a8ed275750fd9754c4be70942e9224944a74",
    "ant-file.ico": "19d0d02786974f63ebce851f4d5332d221d8644878b28a1b8c2f520a4f625331"
  },
  "hir": 5,
  "publicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAECKuwwfugOAJlRRLTo5jD0me19XVWzTeufDrsRFtPuFLAKYw1tA7Yq1I65THXQQcVxHU1aSXvZPHBV3ac25tpkw==",
  "fingerprint": "f4ca87d2738082c5836de88a57ed0bfae9c16b52a3635651e47030b837f9e7a1",
  "signature": "MEQCIAoozTyVoZfrjNfRsJecM8wu1q5as2AIXHSmMks\u002BHMJvAiBI7tgn5DVyajTOdLMznslRgiTtgR1U17CMd6vdOTeXwg=="
}

A pack's identity for caches is the hash its manifest lists for its library (LanguagePack.Identity). A System.Core compiled from sources is cached under a key made of the pack's id, that identity and the .lva format, so a new build of the frontend recompiles it.

Verification

LanguagePack.Verify runs before a pack is loaded, and larva languages runs it for every installed pack. It returns the first problem it finds:

1. abi is present and is 1.
2. larva (if given) is not newer than this larva.
3. hir (if given) is 5.
4. entry is present, files lists at least one file, and the entry is among them.
5. Every listed file is inside the folder, exists, and has the SHA-256 the manifest gives.
6. No .dll, .so, .dylib or .exe anywhere in the folder is missing from files. Code the manifest does not list is never there by accident, and it is never loaded.
7. If the manifest has a signature: fingerprint is the SHA-256 of publicKey, and the signature verifies against the manifest.
8. If the pack was installed from a registry: the registry's checks (see Registries).

A pack that fails is reported and not loaded. larva languages prints each pack and its status:

$ larva languages
Antril 0.8.0  (antril: .ant)
  Antril for Larva: its frontend (Antril source to Larva HIR), its view (HIR written as Antril) and its editor services
  author:    Antril Organization
  license:   Proprietary
  homepage:  https://antril.org
…
  signature: signed, key f4ca87d2738082c5
  status:    ok (files verified)

status: ok (files verified) means steps 1 to 8 passed: every file has the hash the manifest lists and, for a signed pack, the manifest is the one its author signed. The other statuses, each produced by changing the pack above:

  status:    NOT USABLE: ant-file.ico is not the file its manifest lists (changed since the pack was made)
  status:    NOT USABLE: extra.dll is not among its `files`
  status:    NOT USABLE: its signature does not match its manifest (changed since it was signed)
  status:    NOT USABLE: it needs larva 9.0.0 or later (this is 0.8.0)

A build that needs the pack stops with the same reason: "larva: the language pack Antril (…) cannot be used: ant-file.ico is not the file its manifest lists (changed since the pack was made)". A file whose extension no pack reads is refused before anything is loaded:

$ larva build --check Tiny.cpp
larva: Tiny.cpp: no installed language reads .cpp files (installed: Antril (.ant))

What is signed

The signature is ECDSA P-256 with SHA-256, in the DER (RFC 3279) encoding, over the canonical manifest (LanguagePack.Canonical):

• the manifest without its top-level signature and registry keys;
• every object's keys in ordinal order, at every depth;
• no spaces, UTF-8, strings escaped the way the default JSON writer escapes them.

Because files is inside the signed text, the signature covers every file of the pack through its hash. Changing a library, adding one, or editing any manifest field breaks either a hash or the signature. publicKey and fingerprint are signed too, so the key cannot be swapped without the signature failing.

Keys and larva languages pack

larva languages key makes a signing key, a PKCS#8 PEM file with an ECDSA P-256 private key. It never writes over an existing file:

$ larva languages key antril.pem
antril.pem: a signing key (fingerprint f4ca87d2738082c5836de88a57ed0bfae9c16b52a3635651e47030b837f9e7a1); keep it private
public key: MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAECKuwwfugOAJlRRLTo5jD0me19XVWzTeufDrsRFtPuFLAKYw1tA7Yq1I65THXQQcVxHU1aSXvZPHBV3ac25tpkw==
$ larva languages key antril.pem
larva languages: antril.pem exists (a key is not written over)

larva languages pack <folder> [--key <key.pem>] completes a pack's manifest in place (LanguagesCommand.PackFolder):

1. It removes publicKey, fingerprint, signature and registry.
2. It writes files: every file of the folder except language.json, in ordinal order, with its SHA-256.
3. It sets abi to 1 if missing.
4. It loads the frontend and asks info which format it writes, and puts that in hir. The pack must therefore be built for the machine that packs it.
5. With --key, it adds publicKey, fingerprint and signature.

It prints the folder followed by a summary such as Antril 0.8.0, 2 file(s), signed (key f4ca87d2738082c5) (unsigned without a key). The fingerprint is shortened to 16 hex digits wherever larva shows it.

Loading

LanguageRegistry finds packs in these folders, in this order; the first pack of an id wins:

Each subfolder that holds a language.json is a pack. Reading the manifests is cheap and loads no code; a pack's library is loaded the first time a request is made to it (LanguagePack.Load), after Verify passes.

Native packs and managed packs

Load looks at the entry library first:

• A native library (what an installation ships). larva loads it as a shared library and looks up larva_language_call and larva_language_free among its exports. The installed larva.exe is compiled ahead of time and so is the Antril pack: Antril.dll is a native library of about 5 MB that exports those two functions.
• A managed library (a pack built in a working copy). If the entry file carries managed metadata, larva loads it as a managed assembly and searches its public static methods for the two entry-point names, then calls them through their addresses, exactly as it would call the native exports. Only a working copy of larva, which can load managed code at run time, does this. An installed, ahead-of-time compiled larva refuses such a pack and says it loads native packs.

Either way the pack is then asked for info, and loading fails if the hir it reports is not 5.

The same pack source builds both ways: in Antril, LanguageExports.Call and LanguageExports.Free are marked with their export names, so publishing the library natively turns them into real exports, and a managed build keeps them as methods larva finds by those names.

Where the Antril pack comes from

The Antril repository builds its pack into artifacts/pack/Antril/ (Antril.dll, ant-file.ico, language.json). A working copy of Larva finds it through LARVA_LANGUAGES. An installation's packs are native and signed, placed in Languages/<Name>/ next to larva by the release scripts.

Registries

New in this version, and still changing: languages can come from a languages registry, the way packages come from Hive. These commands exist in the code now:

Each takes --provider <url>; without it, larva config languages.provider, else the built-in https://www.antril.org/languages-api/. A registry is a folder or an https address with index.json (every approved language), {id}/index.json (its versions and their packs) and {id}/{version}/{id}-{version}-{platform}.zip.

What changes for a pack installed this way (LanguagePack.FromRegistry, true for packs in the user's folder):

• The author must have signed it.
• A registry must have signed it too: the manifest's registry field holds {fingerprint, signature}, a signature over the manifest without registry (LanguagePack.RegistryCanonical), so the author's signature is itself signed. The registry key must be one larva trusts: a key built into larva, or one added with larva config languages.trust <public key>.
• The author's key is pinned the first time the language is installed, in trust.json of the user's language folder. A later version signed by another author key is refused.

An install downloads the zip for this machine's platform (LanguagePlatform.Current, for example win-x64), checks its SHA-256 against the registry's index, unpacks it into a staging folder (a path that would leave the pack refuses the whole pack), verifies everything above plus abi and larva, and only then moves it into place. larva languages shows a registry: line for such packs.

Templates are HIR

larva new makes a project from a template in the language you choose. A template is written once, in no particular language: its code is a HIR assembly, and the chosen language's pack writes it as source.

A template is a folder Templates/<Name>/. Its identifier is the folder name with - for each space (Console App is Console-App); lookup ignores case. It holds:

The Library template that comes with Larva:

{
  "name": "Library",
  "description": "A library other projects use: a class with a method",
  "author": "Larva",
  "version": "1.0.0",
  "type": "library",
  "namespace": "App",
  "files": { "App.Greeter": "Greeter" }
}

larva new <template> <Name> --lang <language> (NewCommand.Create):

1. Reads the template's HIR: template.lva, or its src/ compiled by their pack against System.Core.
2. Renames the template's namespace (App, and App.*) to the project's name, and the assembly too.
3. Places each type in the file files gives it (plus the language's extension); the rest go to the main file (main, else the project's name for a library or Program for an executable).
4. Asks the pack's source request to write the assembly, with nonPublic: true and header: false.
5. Writes the sources and <Name>.queen (with BUILD keys from this machine's defaults, see larva config). Nothing is written over without --force.

$ larva new Library Shapes --lang antril

The project gets Shapes.queen and Greeter.ant, written by the Antril view from the template's HIR:

into namespace Shapes
{
	/// What the library offers: a greeting.
	public class Greeter
	{
		public:
			/// A greeting for `name`.
			static Greet(str name) : str => "Hello, " + name + "!";
	}
}

The HIR in template.lva has no + on strings: it calls String.Concat<u8> twice. The Antril view writes the sugar back (see The catalog and language views).

larva new --pack <template folder> -o <folder> compiles a template's sources into <folder>/<Name>/template.lva next to a copy of template.json; an installation ships only that:

$ larva new --pack Templates/Library -o tpl
tpl\Library: Library (1 types)

Templates are found in LARVA_TEMPLATES (folders separated like LARVA_LANGUAGES), Templates/ next to larva, %LOCALAPPDATA%/Larva/Templates, and the Templates/ folder of a working copy of Larva above larva. larva new with no arguments lists the templates and the installed languages:

$ larva new
Templates (larva new <template> [Name] [--lang <language>]):
  Console-App  A program for the console: its Main prints a greeting
  Library      A library other projects use: a class with a method

Languages (--lang):
  antril      Antril 0.8.0 (.ant)

With only one language installed, --lang can be left out.

Environment variables