native compilation¶

status: implemented, and still growing. by compile builds real extension modules today; the milestone list in plan.md records what has shipped and what has not

by compile turns a set of .by or .py modules into a native CPython extension (.so / .pyd / .dylib) that imports and behaves exactly like the transpiled python it replaces:

by compile                 # compile the whole project to ./build/
by compile app.hot         # compile one module, leave the rest interpreted
python -c "import app"     # the extension is picked up ahead of the .py

by compile writes everything by build writes and the extensions as well, so build/ holds the whole project as importable python — every .by transpiled, every hand-written .py, the markers, the data files, the sourcemap — with a native extension beside each module that was compiled. two things depend on that. a compiled module reads its fixtures relative to itself the way its interpreted twin does, so the twin property below is a claim about two modules in the same project, not about one of them running somewhere the other's data never reached. and the tree's manifest only means anything while one command authors the whole tree: deleting what the last run wrote and this one did not is right for a mirror and wrong for a heap.

the observable behaviour of a compiled module and its interpreted twin must be identical. that is not an aspiration, it is the property the entire test strategy is built on (plan)

it holds for the programs by check accepts, and a compiled function is the leg that has to be made to hold it: it checks its arguments at its boundary, where an interpreted one checks nothing until the opt-in parameters soundness gate is turned on. that gate used to read a parameter's plan off its annotation, so a type stated with a default was invisible to it:

def f(safe='/'):
    return len(safe)

f(b'abc')   # compiled: TypeError.  twin, before: 3

a default is a written type. def f(safe='/') says safe is a str as plainly as an annotation would — if something else belonged there, something else would be written — and the type system already agrees: a default's requirement is WhenContradicted::Stands, holding even where the body contradicts it, while everything the body merely samples can be withdrawn. so the gate now plans against the parameter, and reaches what the source stated either way.

it deliberately stops there. the rest of an unannotated parameter's bound is recovered from how its body happens to use it, and a recovered requirement is not something to raise a TypeError over. x=None states nothing either — that says the argument may be left out, not that None is what belongs there

why a second backend and not a mypyc invocation¶

the obvious cheap move is to transpile .by to .py and hand the result to mypyc. it does not work, and the reason is the whole thesis of this design

transpilation is lossy by construction. it exists to produce portable python, so every basedpython-only fact is either erased or re-encoded in a form the runtime tolerates:

a compiler reading the transpiled output sees the second column. every fact in the third column — every fact worth compiling for — has already been thrown away. so the compiler must sit where the transpiler sits: directly on the .by AST and ty's inferred types

architecture at a glance¶

                  .by or .py source
                         │
              ┌──────────┴──────────┐
              │  parse (ruff_python_parser)
              │  check (ty_python_semantic)
              └──────────┬──────────┘
                         │  AST + SemanticModel
              ┌──────────┴───────────────┐
              │  (.by only)              │
     by_transforms                    by_irbuild
   (source → python)              (source → BIR)
              │                          │
              │                     by_opt passes
              │                          │
              │                    by_codegen_c
              │                          │
              │                   cc + ld (platform)
              ▼                          ▼
         build/*.py            build/*.cpython-*.so

ordinary python enters the same front end. by_irbuild lowers the .py AST directly, and nothing about that path routes through the transpiler — which is what lets a .py file compile at all, and lets it be its own interpreted fallback: it is already the thing that runs. the source language is one value, by_irbuild::Language, and it settles three questions at once — how the source parses, whether a loop's binding is fresh on each iteration (python shares one, basedpython does not), and where a declined function's definition comes from

the two backends are siblings, not stages. they share a front end and nothing else. this is the single most important structural decision in the design, and it has one uncomfortable consequence: every surface construct now has two lowerings that must agree. plan describes how that is held down

goals¶

• speed, in that order of priority: hot numeric and string code first, attribute and method dispatch second, everything else third
• total language coverage from day one, via interpreted fallbacks — a construct with no native lowering still runs, just not fast
• safety that does not depend on ty being right. a checker bug must produce a TypeError, never a segfault. the unboxing rule in ir exists for this and nothing else
• incremental compilation at function granularity, falling out of salsa rather than bolted on
• debuggability: .by line numbers in gdb, in perf, and in tracebacks
• no new toolchain for the user — the c compiler cpython was built with

non-goals¶

• compiling the standard library or third-party packages. calls out of the unit use the python calling convention and are guarded by the soundness checks that already exist
• a JIT. we compile ahead of time, and the ahead-of-time facts are the ones we have
• replacing cpython's object model, allocator, or GC
• beating a hand-written rust extension. beating cpython by 2–10× on typed code, and by considerably more where the type system lets us leave the object model behind entirely, is the target

the tier ladder¶

compilation is not all-or-nothing. each tier buys more speed by assuming more, and each assumption is one the user opts into explicitly:

tier 2 is the default for a project, tier 1 for a single module compiled out of a larger interpreted program. tier 3 requires a current api lockfile — the lockfile stops being only a review artifact and becomes the ABI contract. see optimizations

what makes this different from mypyc¶

mypyc is the reference implementation of this idea and this design borrows its structure without apology: typed IR, native classes, tagged integers, generated C, a hand-written C runtime. the differences are all downstream of one thing — mypyc reads PEP 484 types, and we read basedpython's

mypyc's own future work list asks for integer range analysis, and for a way to enforce that an attribute is always defined — it already infers that by dataflow where it can (mypyc/analysis/attrdefined.py), and what it is missing is the declaration. so the difference is not that mypyc cannot work these facts out; it is that it has nowhere to read them from:

the whole of optimizations is that table, expanded

one more row is in flight. it gets its own document, because it changes this design rather than extends it:

doc map¶

• technology — what we emit, what the runtime is written in, how it is packaged, and why not llvm / cranelift / rust / asm
• ir — BIR: runtime types, ops, calling conventions, the pass pipeline, and incrementality
• optimizations — every optimization the type system unlocks, ranked by payoff against cost
• planned features — the five modifiers still in flight, what each buys the compiler, and what codegen needs their design to preserve
• runtime — object model, refcounting, exceptions, interop, debugging
• benchmarks — what the suite measures, the method it enforces, and what it refuses to time
• plan — semantic deltas, testing, milestones, risks