Bindings internals

Native bindings shows what larva bind does for you. This page shows how it does it: the project Larva.Bindings (four pieces: export tables, the .bind description, the C header reader, the generator), the command in Larva.Cli/BindCommand, and the profile runner in Larva.Cli/BindProfile with its library System.Bindings.

The idea

A native library's file says which symbols it exports, but not their signatures. Larva does not try to guess them. It keeps two inputs apart:

• the exports of the binary (Exports.Read), which say what can be called;
• a description of the API, either written by hand (.bind) or read from a C header (HeaderReader), which says how.

Both meet in BindingGenerator.Build, which writes an ordinary Larva HIR assembly: value types for structs and unions, enums, and static classes whose methods carry the Import flag. That assembly is saved as a Larva library (.dll with an .lva section, or a bare .lva) and is used like any other library. Nothing in the result is specific to bindings: the frontends, the linker and the back ends see [Import] methods, as if a programmer had written them.

The calls are direct calls with the C convention. There is no marshalling layer; the only conversion is the temporary zero-terminated copy of a string argument.

  kernel32.dll / libfoo.so         api.h                 api.bind
          │                          │                       │
   Exports.Read               HeaderReader.ReadFile          │
   (PE export directory,      (clang -E -dD, then a          │
    ELF .dynsym)               C subset parser)              │
          │                          │  description text     │
          │                          └──────────┬────────────┘
          │                                     ▼
          │                        BindingDescription.Parse
          │                         (functions, structs, unions,
          │                          enums, constants, placements)
          │                                     │
          └──────────── exports ───────────────▶│
                                                ▼
                                    BindingGenerator.Build
                                       (HirAssembly)
                                       │            │
                              LvaFormat.Save    pack "source" request
                                       │            │
                              Name.Bindings.dll   Name.ant (any language)

Export tables

Exports.Read(path) looks at the first bytes of the file. MZ means a PE image (ReadPe); 0x7F 'E' 'L' 'F' means an ELF file (ReadElf). Anything else is a BindingException. Only names come out: no ordinals, no forwarders resolved, no types.

PE (.dll)

A DLL without an export directory gives an empty list. Both 32-bit and 64-bit images are read, although only 64-bit code calls them.

ELF (.so)

ReadElf accepts only 64-bit (EI_CLASS = 2), little-endian (EI_DATA = 1) files. It walks the section headers (e_shoff at 0x28, e_shentsize at 0x3A, e_shnum at 0x3C) and keeps the sections of type 11, SHT_DYNSYM. Each symbol counts when it is a function (STT_FUNC = 2), defined (section index not 0) and global or weak (bind 1 or 2). Its name is read from the string table the section's sh_link names.

Not implemented yet: Mach-O (.dylib). Reading one is an error ("neither a PE (.dll) nor an ELF (.so) file"); a library name that does not exist as a file is bound without the export check.

What the exports are used for

$ larva bind --list kernel32.dll
AcquireSRWLockExclusive
AcquireSRWLockShared
ActivateActCtx
ActivateActCtxWorker
ActivatePackageVirtualizationContext
AddAtomA
AddAtomW
AddConsoleAliasA
…

--list prints them (1698 names for this kernel32.dll). In a bind, they do two different jobs depending on the input:

• From a .bind file, every function must be exported. BindingGenerator.Build(description, exports) fails with all the missing names at once, because a hand-written description that names a missing symbol is a mistake:

  $ larva bind kernel32.dll bad.bind
  larva: kernel32.dll does not export: NoSuchThing
  

• From a header, missing functions are dropped (BindingDescription.KeepExported) and counted, because a header declares more than one build of a library exports: other platforms, inline helpers, macros turned into prototypes.

A library name without a path is looked up in the system directory on Windows (Environment.SystemDirectory), and in /lib/x86_64-linux-gnu, /usr/lib/x86_64-linux-gnu, /lib64, /usr/lib and /lib elsewhere. When the file is not found, larva warns and binds without checking.

The .bind description

BindingDescription is both a text format and the model every path goes through. A header is turned into .bind text first and parsed back; a profile writes .bind text too. That keeps one parser and one set of checks, and it is why --describe can always write the description that was used.

The model

Types stay text in the model ("u16*", "func*<void*, void*> : i32"). They are resolved only when an assembly is built, by BindingDescription.ParseType, because some of them are references into another assembly: str and String<u16> are System.String<u8> and System.String<u16> of System.Core, written as an AssemblyRef plus a TypeRef in the new assembly (HirAssembly.Reference("System.Core", "System", "String", 1)).

Parsing

Parse makes two passes over the lines.

1. Names first. Every struct, union and enum name is registered in a throwaway "probe" assembly. A struct can then point to one declared below it, or to itself.
2. Declarations. Each line is read by its first word: library, assembly, namespace, class, file, enum, struct/union, const, and anything else is a function. Every type is checked against the probe assembly now, so errors carry a line number.

namespace, class and file are placement lines: they apply to the declarations that follow. The first namespace and class also become the description's defaults. ToText() writes the model back with a placement line wherever the placement changes, so a parsed and rewritten description reads the same.

The function syntax is split by SplitTopLevel, which ignores commas inside <…> and (…). The parameter name is the last word, so func*<void*, void*> : i32 compare reads as one parameter.

What a type can be

The rules are the binder's rules for [Import], checked early so the error names the line:

$ larva bind kernel32.dll s.bind
larva: line 2: a C function gives numbers, bools, pointers or func*, not a string (`str`)
$ larva bind kernel32.dll v.bind
larva: line 3: a struct crosses to C by pointer (`P*`), by value only as a field of another struct

A result cannot be a string, because a C function returns a pointer it owns; on this side that is i8* or u16*. A struct cannot travel by value because [Import] calls do not implement the C ABI's rules for aggregates (the frontend rejects them too). Not implemented yet: structs by value in signatures, variadic functions, bitfields.

Layout

BindingDescription.Layout(type, structs, enums) computes size and alignment the way a C compiler does:

The generator does not use these numbers to place fields: it leaves that to the lowering, which already lays out value types sequentially with natural alignment (Layouts.FieldOffsets in Larva.IR), the same rule as C. Layout dates from the time unions were storage words sized by the generator; today it is the reference for C's rule that the layouts the lowering produces are checked against.

Reading C headers

HeaderReader reads a subset of C and produces .bind text. It is not a C compiler: it relies on clang to expand the preprocessor first, then parses declarations with a hand-written tokenizer.

Preprocessing

HeaderReader.Preprocess(path, includes) runs:

clang -E -dD -x c [-I <dir>]... [-I <system dir>]... <header>

• -dD keeps every #define in the output, each after a line marker naming its file. That is how macros become constants.
• FindClang looks on PATH, then in C:\Program Files\LLVM\bin, /usr/bin, /usr/local/bin and /opt/homebrew/opt/llvm/bin.
• On Windows, SystemIncludes adds the newest Windows SDK's ucrt, um and shared folders and the newest MSVC include folder (the one with vcruntime.h). clang started from a plain shell, without the Visual Studio environment, does not find them on its own.
• Without clang (or with --no-pp), the header is read as it is: conditional compilation is not evaluated and directives other than #define and #pragma pack are dropped.

Whose #defines count

A preprocessed windows.h holds thousands of macros from the compiler and the SDK. DefineRoots limits constants to the user's own files: the header's folder and the -I folders. Each #define is kept only when the last line marker names a file under one of those roots; before any marker (an unprocessed header), every #define counts.

#define NAME value becomes a constant when the value is an integer: decimal or hex, with U/L suffixes, in parentheses, negated, or wrapped in a X_C(literal) macro. Macros with arguments and non-numeric values are ignored.

A preprocessed header's line markers name the header the way it was given to clang. With a relative path, the marker is relative too and does not match the absolute roots, so the header's own #defines are missed. Pass the header by its full path, or use --no-pp for a self-contained header. larva bind --profile always passes full paths and is not affected.

Declarations

Declarations splits the code into top-level declarations ending in ; (function bodies are skipped, so a definition counts as its prototype). Decorations are removed first: __declspec(...), __attribute__((...)), __pragma(...), storage words, calling conventions (__cdecl, __stdcall, WINAPI, APIENTRY, CALLBACK, NTAPI…) and qualifiers (const, volatile, restrict).

The reader makes two passes, types first and prototypes second, because a prototype may use a typedef from a header included later. Struct fields are resolved lazily at the end, for the same reason.

Aggregates that cannot be laid out stay opaque (listed in Opaque with the reason): bitfields, fields of unknown types, repeated fields, arrays of unknown size. Pointers to an opaque type are void*. Functions that cannot be called are skipped (listed in Skipped): variadic ones, structs by value in a signature, unknown types. A pointer to a function pointer is void*.

C types

HeaderReader(longIs64) chooses C's data model: LLP64 on Windows (long is 32 bits), LP64 elsewhere (long is 64 bits).

A char* or wchar_t* parameter is a string with ?, because C functions often accept NULL. As a result, it stays a pointer (i8*, u16*).

An example

#define CLOCK_VERSION 3
#define CLOCK_MASK 0xFF00u

typedef unsigned long DWORD;
typedef unsigned short WORD;

typedef struct _SYSTEMTIME {
    WORD wYear; WORD wMonth; WORD wDayOfWeek; WORD wDay;
    WORD wHour; WORD wMinute; WORD wSecond; WORD wMilliseconds;
} SYSTEMTIME, *LPSYSTEMTIME;

typedef union _LARGE_INTEGER {
    struct { DWORD LowPart; long HighPart; };
    long long QuadPart;
} LARGE_INTEGER;

#pragma pack(push, 1)
typedef struct Packed { unsigned char tag; unsigned int value; char name[16]; } Packed;
#pragma pack(pop)

typedef struct Bits { unsigned a : 3; unsigned b : 5; } Bits;

void __stdcall GetSystemTime(LPSYSTEMTIME time);
int __stdcall QueryPerformanceCounter(LARGE_INTEGER* counter);
DWORD __stdcall GetCurrentProcessId(void);
int __stdcall lstrlenW(const wchar_t* text);
int __stdcall lstrlenA(const char* text);
int wsprintfA(char* out, const char* format, ...);
void SetBits(Bits* bits);
void NotThere(int x);

$ larva bind kernel32.dll clock.h --no-pp --class Clock -o Clock.Bindings.dll --source Clock.ant --describe clock.bind
larva: skipped wsprintfA: variadic
larva: opaque Bits: bitfields
larva: 2 declared function(s) not exported by kernel32.dll, left out: SetBits, NotThere
Clock.Bindings.dll: Clock with 5 function(s) of kernel32.dll, 4 struct(s)
Clock.ant: Antril source

The description it read (clock.bind):

library kernel32.dll
class Clock
const CLOCK_VERSION = 3
const CLOCK_MASK = 65280
struct SYSTEMTIME { u16 wYear; u16 wMonth; u16 wDayOfWeek; u16 wDay; u16 wHour; u16 wMinute; u16 wSecond; u16 wMilliseconds }
struct LARGE_INTEGER_anon0 { u32 LowPart; i32 HighPart }
union LARGE_INTEGER { LARGE_INTEGER_anon0 anon0; i64 QuadPart }
struct Packed pack 1 { u8 tag; u32 value; i8[16] name }
GetSystemTime(SYSTEMTIME* time)
QueryPerformanceCounter(LARGE_INTEGER* counter) : i32
GetCurrentProcessId() : u32
lstrlenW(String<u16>? text) : i32
lstrlenA(str? text) : i32
SetBits(void* bits)
NotThere(i32 x)

--describe writes the text before the export filter, so SetBits (with its opaque Bits* as void*) and NotThere are still listed; the library does not contain them. LPSYSTEMTIME became SYSTEMTIME*, the anonymous struct inside the union got its own name, and #pragma pack(push, 1) reached Packed.

From description to HIR

BindingGenerator.Build(description, exports) creates a HirAssembly named after Assembly (by default the class name plus Bindings) in this order:

1. Enums. A TypeDef with Kind = Enum (or Bitflags for flags), Underlying set, and one FieldDef per member: Static | Const, value EnumConstant.
2. Structs and unions. Every TypeDef (Kind = Struct, public) is added before any field is resolved, so a field can point to a struct declared later. Then each field becomes a public FieldDef.
3. Static classes. One per (namespace, class) pair, in the order they first appear, with Flags = Static.
   • A constant is a Static | Readonly field of type i32, or i64 when the value does not fit, with Value = IntegerConstant. The lowering uses that value as the global's initial value.
   • A function is a MethodDef with Flags = Static | Import, ExternName = the C symbol and Library = the native library's file name.

Every name passes through BindingGenerator.Name: a C name that is a keyword of an installed language (the packs' keywords, collected by BindCommand.Reserve) gets a _ suffix. The C symbol in ExternName keeps its spelling, so internal becomes internal_ on this side and still calls internal.

Unions, packs and fixed arrays

C layouts that differ from Larva's default sequential layout are expressed with two attributes of System.Runtime.InteropServices, written into the assembly as references to System.Core:

LayoutAttribute builds the ValueLayoutAttribute constructor as a MemberRef (.new taking Layout) with an EnumConstant argument (Explicit = 1, Sequential = 0) and, for a pack, the named argument Pack. OffsetAttribute does the same for FieldOffsetAttribute(i32).

No back end knows about these attributes. The lowering (HirLowerer.ValueLayout) computes the offsets, turns the IR struct into storage words of the right alignment and size, records each offset in TypeInfo.FieldOffset, and rewrites every field access as *(T*)((u8*)&value + offset) (BodyLowerer.FieldPlace). See Lowering features.

The same code written by hand compiles to the same layout:

import System;
import System.Runtime.InteropServices;

into namespace Layouts
{
	/// What `larva bind` writes for `union LARGE_INTEGER { LARGE_INTEGER_anon0 anon0; i64 QuadPart }`.
	public value LARGE_INTEGER_anon0
	{
		public:
			u32 LowPart;
			i32 HighPart;
	}

	[ValueLayout(Layout.Explicit)]
	public value LARGE_INTEGER
	{
		public:
			[FieldOffset(0)] LARGE_INTEGER_anon0 anon0;
			[FieldOffset(0)] i64 QuadPart;
	}

	/// What it writes for `struct Packed pack 1 { u8 tag; u32 value; i8[16] name }`.
	[ValueLayout(Layout.Sequential, Pack = 1)]
	public value Packed
	{
		public:
			u8 tag;
			u32 value;
			fixed i8[16] name;
	}

	Main()
	{
		LARGE_INTEGER n;
		n.QuadPart = 0x100000002;
		Console.WriteLine(n.anon0.LowPart);
		Console.WriteLine(n.anon0.HighPart);
		Console.WriteLine(sizeof(LARGE_INTEGER));
		Console.WriteLine(sizeof(Packed));
	}
}

$ larva build -o layout.exe layout.ant
layout.exe (llvm back end, O3, entry layout::Layouts.$Module.Main)
$ layout.exe
2
1
8
21

Packed is 21 bytes: 1 + 4 + 16, with no padding.

Unions went through two designs. The first made a union a value of storage words (_0, _1…) and generated get/set properties whose bodies read and wrote through *(T*)&this. When [ValueLayout] and [FieldOffset] reached the language, unions became real fields at offset 0: simpler HIR, plain field access, and the decompiled source no longer needed bodies to keep the layout.

The library

HirLinker and the frontends read the generated assembly like any other. larva show prints it:

$ larva show --lang hir Bindings.Clock Clock.Bindings.dll
// Functions of kernel32.dll, called directly.
public class Bindings.Clock
{
    static field CLOCK_VERSION: i32
    static field CLOCK_MASK: i32
    static method GetSystemTime(time: Bindings.SYSTEMTIME*) -> void
    static method QueryPerformanceCounter(counter: Bindings.LARGE_INTEGER*) -> i32
    static method GetCurrentProcessId() -> u32
    static method lstrlenW(text: System.String<u16>?) -> i32
    static method lstrlenA(text: System.String<u8>?) -> i32
}

The file is 2048 bytes: a PE container whose only content is the .lva section (see The .lva format). A program passes it to larva build or names it with assembly: in a project:

import System;
import Bindings;

into namespace App
{
	Main()
	{
		LARGE_INTEGER counter;
		unsafe { Clock.QueryPerformanceCounter(&counter); }
		Console.WriteLine(counter.QuadPart > 0);
		Console.WriteLine(sizeof(LARGE_INTEGER));
		Console.WriteLine(sizeof(Packed));
		Console.WriteLine(Clock.CLOCK_VERSION);
	}
}

$ larva build -o app.exe main.ant Clock.Bindings.dll
app.exe (llvm back end, O3, entry app::App.$Module.Main)
$ app.exe
true
8
21
3

The source

With --source Name.ext, the source is not written by Larva.Bindings at all. BindCommand.Write sends the assembly to the language pack that owns the extension, as a source request (assembly, references, files, defaultFile), and writes what comes back. The pack decompiles the HIR with its view; Antril writes [ValueLayout], [FieldOffset], fixed arrays and readonly constants as shown in Native bindings. When the assembly references System.Core (strings, attributes), the core library is sent as a reference so the view can name String<u16> and the attributes.

The source is only for reading. A program uses the compiled library; the generated file is excluded from the build.

Output names

BindCommand.DefaultOutput names the library <Assembly>.dll, or <Assembly>.Bindings.dll when that equals the native library's file name. Refused rejects an -o that matches it:

$ larva bind kernel32.dll k.bind -o kernel32.dll
larva: kernel32.dll: the bindings would have the native library's name, and Windows would load them in its place; name them otherwise (such as kernel32.Bindings.dll)

A Larva library is a DLL with no code. If it had the native library's name, the Windows loader would find it first (next to the program, or in the current folder) and every import would fail. Refused also refuses to overwrite a DLL that is not a Larva library (QueenProject.IsAssembly).

Binding profiles

A profile is a class in the user's language that implements System.Bindings.IBindingProfile, usually by deriving from BindingProfile. larva compiles it and runs it, in its own process, while it binds. An earlier design used a declarative .bindings file in .queen syntax; it was replaced because every new rule needed a new key, while a class can decide anything with ordinary code.

System.Bindings

Library/System.Bindings is written in Antril, next to System.Core:

An installation ships it compiled as System.Bindings.dll next to larva (built by the installer scripts with larva lib). A working copy of the repository has no such file; BindProfile.BindingsLibrary then finds Library/System.Bindings above larva's folder, compiles it against System.Core and caches it as %LOCALAPPDATA%\Larva\cache\System.Bindings-<hash>.lva. The hash covers the language pack's identity, the .lva format version, System.Core's bytes and every source file.

The language server gives System.Bindings to every analysis, so a profile file has completion and errors like any other source.

How larva runs a profile

BindProfile.Run does four steps:

 Profile.ant ──compile (library)──▶ profile.lva ──catalog──▶ the one public IBindingProfile
                                                              (or --class)
 1. configure   glue program:  p.Configure(s); BindingRunner.WriteSettings(s, "settings.txt")
                       │
                       ▼  settings.txt (key/value lines), command line on top
 2. headers     HeaderModel.Read(headers, library, includes, long, enums, flags, scope, exports)
                       │
                       ▼  model.txt (tab-separated: E/M/T, S/F, C, U/P lines)
 3. hooks       glue program:  m = BindingModel.Read(s, "model.txt"); BindingRunner.Run(p, m, "binding.bind")
                       │
                       ▼  binding.bind
 4. generate    BindingDescription.Parse → clash check → BindingGenerator.Build → .dll + source files

Each step that runs profile code is a glue program that larva writes in the profile's language (Glue), compiles against System.Core, System.Bindings and the profile library, and links with HirLinker. For Antril it looks like this:

import System;
import System.IO;
import System.Bindings;
import ClockBinding;

into namespace LarvaBindRun
{
	Main() : i32
	{
		try
		{
			BindingSettings s = new BindingSettings();
			IBindingProfile p = new ClockProfile();
			p.Configure(s);
			BindingRunner.WriteSettings(s, "…/settings.txt");
		}
		catch (Exception e)
		{
			File.WriteAllText("…/configure.failed", e.ToString());
			return 1;
		}
		return 0;
	}
}

The glue catches what the profile throws and writes it to a file, because an exception thrown in native code cannot be caught from outside; larva reads the file and prints the profile's error.

Where the glue runs. On Windows x64 the linked program is lowered, compiled with Compiler.CompileObject, linked with the IR linker and compiled by X64Backend into memory; larva calls its entry through a function pointer (native.Program.Load()). No system linker runs and no file is written. Other platforms fall back to the VM interpreter (Interpreter with no step limit). The design doc records why: generating the model as source (388 KB of statements) ran out of memory in the IR verifier, and running the hooks on the VM took more than ten minutes for SDL3; in memory as x64 code it takes about two seconds. NativeRuntime.Reset() runs before and after, so each glue program starts with a fresh runtime.

The model file. ModelText writes what the headers declared as one line per item, fields separated by tabs, and BindingModel.Read parses it on the other side:

The headers and typedefs come from HeaderReader.Origins (the header each name was declared in, from the line markers) and TypedefUses (Function#0, Function#result, Struct.field). They are what lets a profile split a binding by header, and give a parameter written as SDL_InitFlags the enum made from SDL_INIT_*.

HeaderModel.Read prepares that model: several headers are read through a temporary umbrella header that #includes each; C enums take their members out of the constants (LooksLikeFlags: three or more distinct single bits make a bitflag enum; Underlying picks the smallest of i32, u32, i64, u64); functions the library does not export are dropped (or are an error with ExportCheck.Check); and unless AllHeaders is set, only declarations from the user's folders stay, plus the system structs they use (found by following field and parameter types).

After the hooks, larva checks for clashes (Clashes: two types with one name in a namespace, two members of one class) and stops with an error on the profile's file if any remain. BindingRunner already resolves most of them by keeping the C name of the later one and warning.

A run

import System;
import System.Bindings;

into namespace ClockBinding
{
	public class ClockProfile : BindingProfile
	{
		public:
			override Configure(BindingSettings s)
			{
				s.Libraries.Add("kernel32.dll");
				s.Headers.Add("clock.h");
				s.Namespace = "Win";
				s.Class = "Clock";
				s.Strip.Add("CLOCK_");
				s.ConstantCase = NameCase.Pascal;
				s.FunctionCase = NameCase.Pascal;
			}

			override Function(CFunction f) : bool => !f.Entry.StartsWith("lstrlenA");
	}
}

--timings prints each step as it ends (nested lines are the glue's own phases), and --dry-run prints the description instead of writing anything:

$ larva bind --profile ClockProfile.ant --dry-run --timings
    compiled: 1585 ms
    linked and lowered: 759 ms
    compiled to x64: 1427 ms
    run: 112 ms
  configure: 9 ms
larva: 3 declaration(s) of the headers that kernel32.dll does not export, left out
  headers read: 461 ms
    compiled: 29 ms
    linked and lowered: 35 ms
    compiled to x64: 928 ms
    run: 7 ms
  profile run: 0 ms
library kernel32.dll
assembly ClockBindings
namespace Win
class Clock
struct SYSTEMTIME { u16 wYear; u16 wMonth; u16 wDayOfWeek; u16 wDay; u16 wHour; u16 wMinute; u16 wSecond; u16 wMilliseconds }
struct LARGE_INTEGER_anon0 { u32 LowPart; i32 HighPart }
union LARGE_INTEGER { LARGE_INTEGER_anon0 anon0; i64 QuadPart }
struct Packed pack 1 { u8 tag; u32 value; i8[16] name }
const Version = 3
const Mask = 65280
GetSystemTime(SYSTEMTIME* time)
QueryPerformanceCounter(LARGE_INTEGER* counter) : i32
GetCurrentProcessId() : u32
lstrlenW as LstrlenW(String<u16>? text) : i32
// 4 function(s), 4 struct(s), 0 enum(s), 2 constant(s); nothing written (--dry-run)

The timings came from a working-copy larva, not the ahead-of-time compiled one, so the first compile includes warming up. CLOCK_VERSION lost its prefix and became Version; lstrlenW became LstrlenW with an as, since the entry keeps its C spelling; the profile's Function hook removed lstrlenA. The 3 declaration(s) are SetBits, NotThere and the skipped variadic wsprintfA, none of which kernel32 exports.

Without --dry-run:

$ larva bind --profile ClockProfile.ant -o Clock.Bindings.dll --source src
larva: 3 declaration(s) of the headers that kernel32.dll does not export, left out
Clock.Bindings.dll: 4 function(s), 4 struct(s), 0 enum(s), 2 constant(s)

followed by a line naming the source folder and its file count. With Split, the description's file lines carry stems (Video); Place adds the language's extension and BindingGenerator.Files maps each type's full name to its file, which the pack's source request uses to split the output.

On a bigger library: SDL3 with its profile gives 1247 functions, 131 structs and 102 enums in about 8 seconds in all on a working-copy larva, about 2 of them running the profile.

Limits