Hive internals

Hive is Larva's package manager. Packages with Hive explains how to use it; this page explains how it works. Language packs have a separate registry with its own rules, described in The languages registry.

Hive was once a tool of its own; it is now larva hive, and Hive.Core is a library inside larva. Because larva is compiled ahead of time, Hive.Core writes and reads JSON through a source-generated serializer context (HiveJson) instead of reflection.

The pieces at a glance

  library project (.queen)                          program project (.queen)
          │ larva hive pack                                  │ DEPS  package: Ana/Greetings@^1.0
          ▼                                                  │
  bin/Greetings-1.0.hive ── larva hive publish ──► registry  │ larva hive install
   (zip: hive.json,                                Ana/Greetings/index.json
    hive.sig, files)                               Ana/Greetings/1.0/Greetings-1.0.hive
                                                        │
                                  download, check hash, │ signature, every file
                                                        ▼
                                  cache: packages/Ana/Greetings/1.0/  ──►  hive.lock
                                                        │
                                  larva build / run / lsp (HiveResolver, no network)
                                                        ▼
                                  package.queen loaded as a dependency project

Installing is the only step that touches the network. Building only looks in the cache.

The .hive file

A package is a zip archive. HiveArchive.Create writes its entries in a fixed order:

Every entry is written with the strongest compression and the same time stamp, 2000-01-01 00:00 UTC, so the same inputs give the same archive apart from the signature (ECDSA signatures are randomized: packing twice gives an identical hive.json and a different hive.sig). The archive is first written to <name>.partial and then moved into place.

The package of a small library, Ana/Greetings 1.0, compiled and signed:

$ unzip -v bin/Greetings-1.0.hive
Archive:  bin/Greetings-1.0.hive
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
     562  Defl:X      321  43% 2000-01-01 00:00 bbe89ded  hive.json
      70  Defl:X       75  -7% 2000-01-01 00:00 2153b175  hive.sig
    4096  Defl:X     1972  52% 2000-01-01 00:00 4886d1a2  lib/Greetings.dll
     173  Defl:X      132  24% 2000-01-01 00:00 89a4682b  package.queen
--------          -------  ---                            -------
    4901             2500  49%                            4 files

Paths inside a package

HiveArchive.SafePath accepts a path only if it is relative, uses /, is at most 240 characters, and each part is non-empty, not . or .., does not end in a dot or a space, has no control characters or < > " | ? *, and is not a Windows device name (CON, PRN, AUX, NUL, COM1…, LPT1…). hive.json and hive.sig themselves cannot be package files. The same check runs when packing, when parsing a manifest, before downloading a file named in an index, and once more when unpacking (every target must stay under the package's folder).

Folders inside a package

HivePacker decides where each file goes:

A resource whose path would land in lib/, src/ or native/, or be package.queen, is refused. Two different files that would get the same path are refused.

package.queen

Every package carries a generated package.queen: the package described as a library project. This is the file a program compiles with once the package is installed, so an installed package is just another dependency project to Larva.Build (see Projects and builds). The one in Ana/Greetings:

queen 1.0;

chunk PROJ:
   name: Greetings,
   type: library,
   version: v1.0,
   author: Ana

chunk SOURCES:
   scan: false

chunk DEPS:
   assembly: "lib/Greetings.dll"

The packer adds what applies: larvaobj: lib when there are .lvo objects, an assembly: per Larva library, a lib: per native library, a package: per package the project depends on, a RESOURCES chunk, and for a source package include: src/** in SOURCES. scan: false keeps the package's folder from being searched for sources.

Limits

Counting real bytes while decompressing is what stops a zip bomb: an entry that claims to be small and expands to gigabytes is cut off at its declared size.

hive.json

HiveManifest is written as indented camelCase JSON with null fields left out. The manifest of the package above:

{
  "format": 1,
  "package": "Ana/Greetings",
  "version": "1.0",
  "license": "MIT",
  "kind": "compiled",
  "gc": "default",
  "dependencies": [],
  "files": [
    {
      "path": "lib/Greetings.dll",
      "size": 4096,
      "sha256": "983c3a1998709c4e3c0fb2173abda5e480ed38e22e51805e26713e8bcc900f8a"
    },
    {
      "path": "package.queen",
      "size": 173,
      "sha256": "945759d1090a816a3f985ffc1e135178ea22466c394dcb6ca6555d3a649545a3"
    }
  ],
  "key": "633d6fd406a21ba62eb5d91ee29d3202c2020ea8958c308bb242ea4445a62502"
}

Because the signature covers the manifest and the manifest lists the hash of every file, one signature vouches for the whole package. HiveManifest.Parse also rejects unsafe paths, sizes below zero, hashes that are not 64 characters, a version that is a range, and dependencies that do not parse.

The manifest's gc is informational today: the generated package.queen has no BUILD chunk, so a program that installs the package loads it with the program machine's defaults, and a package built for another GC is not linked as a separate GC domain.

Signing keys

HiveKeys uses ECDSA over the NIST P-256 curve with SHA-256.

Signing the bytes rather than a re-serialized object means a verifier never has to reproduce the writer's JSON formatting: it hashes what is in the archive.

$ larva hive key
…
public key:  MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWcf85mJcX1Ivape5QdiHiXQmf+fLSA0V50VtTET1Js/BEGxUA+UZamc1VGfyHSawktSeUaKZoRajUPTINWy12Q==
fingerprint: 633d6fd406a21ba62eb5d91ee29d3202c2020ea8958c308bb242ea4445a62502

The … line names the file written (hive.key by default, or -o <file>). An existing file is never overwritten. The project names its key with sign: in HIVE; larva hive pack reads it, computes its public key and fingerprint, puts the fingerprint in the manifest, then signs.

Opening a package

HiveArchive.Open(bytes, publicKey, requireSignature) checks everything before anything is written to disk:

1. The bytes are a zip archive, with at most MaxFiles + 2 entries.
2. hive.json exists and parses (see above).
3. hive.sig is read if present. Without it, the package is refused when a signature is required or a key is given.
4. With a key: the manifest's key must equal the key's fingerprint, and the signature must verify over the manifest bytes.
5. The declared total size is within MaxUnpacked.
6. Every entry other than the manifest and signature must be listed in the manifest, have exactly its size and SHA-256, and appear once. Empty folder entries are ignored.
7. Every listed file must be present.

Extract then writes the files into <folder>.partial, adds hive.json and hive.sig, deletes any previous <folder>, and renames .partial into place. An interrupted install leaves a .partial folder that the cache ignores and the next install replaces.

Installing runs nothing from the package: there are no install scripts, and the files are only read by later builds.

larva hive verify runs Open on a file. Without --key it reports who signed but cannot check it; with --key the signature must be that key's:

$ larva hive verify bin/Greetings-1.0.hive
Ana/Greetings 1.0 (compiled, gc default): 2 files, 4269 bytes
  signed by 633d6fd406a21ba6 (not checked: give --key)
  every file matches its hash
$ larva hive verify bin/Greetings-1.0.hive --key MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWcf85mJcX1Ivape5QdiHiXQmf+fLSA0V50VtTET1Js/BEGxUA+UZamc1VGfyHSawktSeUaKZoRajUPTINWy12Q==
Ana/Greetings 1.0 (compiled, gc default): 2 files, 4269 bytes
  signature good (633d6fd406a21ba6)
  every file matches its hash

The same package checked against another key, and two altered copies (one with a byte of package.queen changed, one with a file the manifest does not list):

$ larva hive verify bin/Greetings-1.0.hive --key MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEKo2b3Tsz7CQTCYXoU6YAY6pU3eSB0xSv+90nRqXnmy2OuUU+vT/MZZayBCo10JGKxaegfkdyIrIo1Im2yaO2pw==
larva hive: Ana/Greetings 1.0 says it was signed by key 633d6fd406a21ba6, not by 6386be6f2911c074
$ larva hive verify bin/tampered.hive
larva hive: `package.queen` in Ana/Greetings 1.0 does not match its hash: the package is damaged or was changed
$ larva hive verify bin/extra.hive
larva hive: the package carries `lib/extra.dll`, which its manifest does not list

Packing

larva hive pack [project] (HiveCommand.Pack, then HivePacker.Pack):

1. Loads the project. It must be type: library, have a version, and not be private: true.
2. Picks the package name: HIVE's package:, else PROJ's author (spaces become -) and name.
3. For a compiled package with sources, runs larva lib into a temporary hive-<guid> folder, against the project's assembly: libraries, giving Name.dll.
4. Collects the files (see the folders), generates package.queen, fills the manifest, signs it if sign: is set, and writes bin/<Name>-<version>.hive.

$ larva hive pack
…
  2 files, 4269 bytes before compression
…

The elided lines are larva lib's report for the temporary Greetings.dll, and packed <path>\bin\Greetings-1.0.hive (2920 bytes). An unsigned package gets a note that projects with verify: strict will refuse it.

A project with scan: false and no sources (a package around a binding built elsewhere) skips step 3; its assembly: libraries are the code. A package with nothing compiled and no sources is refused.

The library is compiled with the machine's frontend options (larva config), not the project's access.default or borrowcheck. Its resources are not compiled into Name.dll: they travel as files and the generated package.queen embeds them into the program that uses the package.

The cache

HiveCache.Default() is HIVE_CACHE, else %LOCALAPPDATA%\Larva\hive. A project's cache: in HIVE replaces it for that project.

<cache>/
  keys.json                          trusted keys: { "Ana/Greetings": ["633d6fd4…"] }
  packages/
    Ana/
      Greetings/
        1.0/
          hive.json                  the manifest, as it was in the archive
          hive.sig                   its signature (when signed)
          .sha256                    hash of the .hive file it came from
          package.queen
          lib/Greetings.dll
        1.1.partial/                 an interrupted install: ignored

A version counts as installed when its folder has both package.queen and hive.json, and its name parses as a version. Versions() lists them newest first; larva hive list prints every one.

Trusted keys

keys.json maps each package to the fingerprints allowed to sign it. HiveCache.Trust runs on every install of a signed version:

• the first time a package is installed, its signer is remembered;
• later, a version signed by a remembered key is accepted;
• a version signed by any other key is refused, with a message that names larva hive trust Author/Name <fingerprint>.

larva hive trust only adds a fingerprint to the list. This is "trust on first use": a registry that starts serving a package under a new key cannot make an existing installation accept it silently.

Choosing versions

A reference is Author/Name[@range]; VersionRange.Parse reads the range:

HiveInstaller.Choose finds one version per package by iterating to a fixed point:

chosen := {}
repeat (at most 100 rounds):
    requirements := the project's package: entries
                  + the dependencies of every version in chosen
    for each package with requirements:
        fitting := index versions accepted by every requirement,
                   not yanked (unless it is the version hive.lock pins)
        if fitting is empty: fail, listing who wants what and what is published
        next[package] := the pinned version if it fits, else the newest fitting
    if next == chosen: done
    chosen := next

The dependencies of a version are read from the registry's index, so choosing needs no package downloads. Each package's index comes from the first provider in the list that has it. The result is a fresh hive.lock holding only the packages still needed.

larva hive restore does not choose: for each locked entry it finds that version in the index, requires its SHA-256 to equal the lock's, and installs it.

Installing

HiveInstaller.Fetch installs one chosen version:

1. If the cache already has that version and its .sha256 equals the index's hash, nothing is done.
2. Download the file and compare its SHA-256 with the index's. A mismatch is refused.
3. If the version is signed, take its public key from the index's keys, and check that the key's fingerprint is the one claimed.
4. HiveArchive.Open with that key: signature and every file. A signature is required unless the project says verify: hashes.
5. The manifest's package and version must be the ones asked for, so a registry cannot answer one name with another package.
6. HiveCache.Trust with the signer's fingerprint.
7. Extract into packages/Author/Name/version, then write .sha256.

larva hive add first adds a package: line to the DEPS chunk of the .queen file (creating the chunk at the end when there is none) and then installs. Both add and install read the project without resolving packages, since those are what is being installed.

hive.lock

hive.lock is JSON beside the .queen file (HiveLock), its packages sorted by name:

Building with packages

larva build, larva run, larva show/types and the language server pass HiveResolver.Create() to QueenProject.Load. The resolver never downloads:

• The first project that asks (the one being built) decides the cache and the lock for the whole tree.
• For each package:, it lists the installed versions that the range accepts. It takes the version hive.lock pins when that one is installed, else the newest.
• It returns that version's package.queen, which QueenProject loads as a dependency project. A package that is not installed is error QN108, which suggests larva hive install (or larva hive restore when the lock pins it).

From there a package is an ordinary dependency: its assembly: libraries are referenced and linked, its lib: files are copied next to the program, its resources are embedded, and a source package's sources are compiled with the program.

Registries

Providers

A project's provider: entries (or hive.provider from larva config, or antril.org/hive when neither is set) are parsed by HiveProvider.Parse, relative to the project's folder:

Installing tries the providers in order for each package. Publishing goes to the first one, or to --provider.

The index

A registry is only files, so a folder, a static web server or the site's API can all be one. For a package Author/Name it has:

{root}/Author/Name/index.json
{root}/Author/Name/{version}/{Name}-{version}.hive

index.json (HiveIndex):

Versions are kept sorted, oldest first.

Publishing rules

HiveProvider.Add updates an index, and the rules are the same in a folder registry and on the server:

• A version that exists cannot be published again.
• The first version of a package brings its key into keys.
• Later versions must be signed by a key already in keys. A new key is accepted only for a package that has none.
• Once a package has a key, an unsigned version is refused.

So a registry cannot be used to switch a package's signer quietly, and installers that remember keys add a second line of defense.

larva hive publish packs, opens the result to read its manifest, and calls the provider's Publish with the file, its hash, the manifest's key and dependencies, and the public key derived from sign:. A folder provider writes the file and replaces index.json through a .partial file.

The HTTPS protocol

HiveHttpProvider speaks plain HTTP requests over HTTPS:

Any status other than success or 404 is an error, shown with its reason phrase. Requests time out after two minutes, and response bodies are limited while they are read.

The public registry at https://www.antril.org/hive-api implements the same protocol with Hive.Core itself. Its PUT answers:

Errors come as plain text, also placed in the reason phrase so that larva hive can show them. Package files are served with Cache-Control: public, max-age=31536000, immutable, since a published version never changes; indexes with no-cache.

Not implemented yet: yanking a version (the index has the yanked field and installers honor it, but no command or API sets it).

Settings that affect Hive