sourcemaps — design plan¶
status: plan. only the slice by run needs is built today (a forward,
line-level table). this document is the design for the full bidirectional,
column-accurate sourcemap and the work to get there. nothing here is
implemented beyond what is noted under "what exists today"
why¶
a .by file is transpiled to python before anything runs or is analyzed
downstream. every tool that reports a position therefore risks pointing at
generated python instead of the .by the user wrote:
- runtime tracebacks — frames land in generated
.py, with line numbers shifted by the preamble and surface syntax replaced by its lowering - type-checker diagnostics on the
.py— if a downstream tool checks the generated output, spans don't map back - editor features — go-to-definition, hover, rename, inlay hints driven off
the generated python need to round-trip to
.bycoordinates - reverse transpile —
.py→.byhas the same problem mirrored
a correct sourcemap is the single primitive all of these share
what exists today¶
source_map::line_table(source, edits)— builds a forward, line-level table (output line → input line,Nonefor generated lines) for one edit-application pass- a class a type expression lowers to (a typed dict literal, an anonymous named
tuple, an inline protocol) is declared in the preamble, and like the rest of
the preamble is charged to no
.byline. it is made as the module starts, not where the type is written, and python keeps one line table per code object: the line a traceback names is the line coverage counts as run, a trace event reports and a breakpoint stops at transpile_typed_with_mapcomposes a whole-pipeline line table by lifting the phase-0 table over the count of generated preamble lines (valid because phase 1 applies no body edits and later phases only prepend). consumed byby runto rewrite tracebacks and byby transpileto map a transpiler-invalid-output span back to its.byline for a source-annotated diagnostic_by_sourcemap.pycarries three tables keyed by the generated.pypath:SOURCEMAP, the.bypath and its line table,DIGESTS, the sha-256 of both files that entry describes, andCOLUMNS, which columns of each generated line are the.by's text (see columns).by buildwrites it intobuild/beside the python it describes, andby runinto the temporary tree it executes — where it lives only as long as that run. see staleness below- a line table is packed into one string: the zero-based
.byline of each generated line, or-for a line the transpiler wrote on its own, joined by,."-,0,1,1,3"is a five line file whose first line is the transpiler's.by runimports the map in a tree it has just staged, so python compiles it every time, and a string is one constant where a list is a constant for every line of the project. a reader unpacks a module's table the first time it is asked about that module
limitations to remove¶
- copied text only — a column maps only where the generated text is copied
from the
.by(columns): a column inIntersection[int, str]has no column ofint & strbehind it. fine for traceback frames and a debugger's calls, not enough for hover/rename - forward only — no
.by→.pydirection (needed by editors that start from a.byposition) - prepend-only assumption — the composition trick assumes later phases never insert mid-body. true today; brittle as a contract
- untested arithmetic — offset math has no property tests
staleness¶
built, unlike the rest of this document
a map describes a pair of files, and the lines it reports are only true while
both are still the ones it was built from. an editor that saves the .by after
the transpile leaves the map describing a pair that no longer exists — and it
goes on resolving lines with total confidence. that is a wrong answer rather
than a missing one, which is the failure a sourcemap exists to prevent
so every entry carries a digest of both sides, and a consumer must recompute them from disk before it trusts a line:
DIGESTS = {
"/tmp/.tmpXXXX/demo.py": {
"by": "sha256:…", # the .by bytes the transpiler read
"py": "sha256:…", # the python bytes it wrote
},
}
three things that shape pins down:
- the bytes that were read and written — both digests are taken in the transpiling pass, over the source text it ran on and the python it produced. hashing the files again afterwards would only be a second chance for them to have changed
- raw bytes — so a consumer recomputes with
sha256(open(path, "rb").read())and needs to know nothing about encoding or line endings - the algorithm named in the value — it can be changed later without breaking readers, and a reader that meets one it does not know refuses the entry instead of comparing hex it could never have produced
DIGESTS is a separate top-level name rather than a third element of the
SOURCEMAP tuple: that tuple has consumers (the traceback shim, the pycharm
plugin), a positional change would break every one of them, and a name they
never read is invisible to them
the traceback shim is the first consumer, and it shows what refusing looks like:
when either digest disagrees it leaves the frame in the generated python and
writes a note saying which file no longer matches. a frame pointing at
build/main.py is a worse answer, but a frame quoting a .by that has been
rewritten since is a false one
by run also reads the map when it loads a staged module: python's own loader
would compile the .py, so the shim compiles it instead, under the .by path and
with each line the .by line it came from (a generated line is line 0). every
location the running program reads back — a traceback it formats, the place a
warning is attributed to, co_filename, inspect — is then the author's, and so is
the line of a warning python raises while compiling it, and of the SyntaxError it raises
instead when the warning filters make that warning an error (-W error). the .py is compiled from its
text, as python's loader would, and its code objects are then moved to the .by lines:
python 3.9 to 3.11 refuse a syntax tree nested deeper than their recursion limit, even
one whose source they compile. the digests are checked then, and a mismatch leaves
the module to python's loader. a .by saved over during the run is caught when a frame is reported: the frame
names the one generated line that .by line was written as, or no line when
there was more than one
the loader that does this is python's own source loader with that one method
replaced, not a wrapper around it. python removes its import machinery from a
traceback raised while a module is imported, but only when nothing stands between
that machinery and the module's code, and a wrapper's exec_module did: a module
that failed as it was imported showed <frozen importlib._bootstrap> three times
and the shim once above the line that failed, in every traceback no excepthook
rewrites. it now shows what python shows for a program it runs itself, the line
that imported and the line that failed
pytest compiles a test module and a conftest itself, to rewrite their
assertions, from an import hook it puts at the head of sys.meta_path. left to
itself, that hook compiles the staged .py at the lines of the generated python: a
failure would be reported naming it, and a debugger would never stop at a
breakpoint in the .by. by run puts a pytest plugin in the tree, _by_pytest, declared by a
pytest11 entry point in a _by_pytest-0.dist-info beside it, so pytest loads it
wherever the tree is on sys.path — after it installs its hook and before it
imports any test module or conftest. the plugin tells the shim how pytest's hook
compiles a module, and the shim puts its own finder back ahead of the hook. a
staged module the hook would load is then compiled as the hook compiles it, its
assertions rewritten, and moved to the .by lines like any other: its top level, a
class body, a comprehension and a lambda as much as a function, from the moment it
is compiled. a failure in a test or a fixture is reported at the .by line,
quoting the .by, a breakpoint on any line of a test module stops there, and an
error the module's top level raises, or an import in it fails with, is reported in
place of pytest's report of it as by run reports an error in a program, python's
traceback with each frame at its .by line. a hand-written .py the build carries
over is compiled under the project file's path the same way. the module's
__file__ stays the staged .py, as do pytest's node ids,
tests/test_calc.py::test_sum: pytest checks that a module it imported is the file
it collected, and a node id is what pytest is given back to run a test again, and
what its cache and its --deselect read, so both have to name a file in the tree
pytest runs in
a module some other import hook compiles itself keeps that hook's loader, which
python calls directly; only its __file__ is set, as the module is created. under
pytest, once pytest has imported such a module and before any of its tests runs,
the plugin moves the code of every function the module defines — the methods of
its classes, what a decorator wraps, a fixture's function, and the functions nested
in each — to the .by lines. its top level ran as it was imported, before the
plugin saw it, at the lines of the generated python
both tables are keyed by the generated path exactly as the map spells it. the
shim resolves symlinks before matching a frame's filename against it — a
temporary tree under /tmp on macOS is reached by a different path than the one
python reports — so it carries the original key alongside the resolved one. a
consumer that normalises keys has to do the same, or its digest lookup misses
and every entry reads as stale
columns¶
built, unlike the rest of this document
a debugger names a call on a line by its text, and the text of a call the
transpiler rewrote is the generated python's: classify(p) read through the
lazy-import holder is _by_lazy.classify(p), and a comprehension's iterable is
_soundness_iter(points, Point). so the map also says, for each generated line,
which of its byte columns are text the .by spells, as a third table keyed as
the other two are:
each line of the generated file, in order, is its runs joined by , and ended by
;, so a line with no runs is a bare ;. a run start end by_line by_column says
columns start to end of the generated line are the .by's text from
by_column of by_line, both zero-based. a run start end is a qualifier the
transpiler wrote in front of the author's name to say where the name lives,
_by_lazy., which reads the object the name did. every column in no run is the
transpiler's. columns are byte offsets, as python's own code positions are. the
entry is packed into one string for the reason a line table is, and unpacked when
its module is first located
every step of the transpile describes its output against its input, and the
descriptions are composed. a step that rewrites a line without saying where its
text came from — a polyfill's header, an import the runtime's helpers join —
leaves that line the transpiler's, which loses its columns and never claims text
the author did not write, and every claimed run is checked against the .by
before it is written
by run reads the table when it compiles a staged module as its .by. on python
3.11 and later each location of the code — what co_positions answers, a
traceback marks and a debugger names a step-into target by — is:
- the author's text: the
.bylines and columns of exactly the text it covers, when every column of the generated span is in a run, and the copied runs are one run of the.by._by_lazy.classify(p)isclassify(p) - the transpiler's: the
.byline with no columns,col_offsetandend_col_offsetbothNone, for a span with any column in no run._soundness_iter(points, Point)is the line of the comprehension it is written for, and no columns - line 0, as before, for code no
.byline is behind
python 3.10 and older have no columns, and every location spans its whole line
code by writes for itself¶
built, unlike the rest of this document
a debugger looks for _by_sourcemap.py where the program is, so the map also says
which of the tree's python is by's rather than the program's. it says it after
the two tables, as names a reader that predates them never reads:
LAUNCHER = "/tmp/.tmpXXXX/_by_runner.py"
BY_CODE = [
"/tmp/.tmpXXXX/_by_coverage_plugin.py",
"/tmp/.tmpXXXX/_by_project.py",
"/tmp/.tmpXXXX/_by_pytest.py",
"/tmp/.tmpXXXX/_by_repl.py",
"/tmp/.tmpXXXX/_by_runner.py",
"/tmp/.tmpXXXX/_by_runtime.py",
"/tmp/.tmpXXXX/_by_sourcemap.py",
"/tmp/.tmpXXXX/pkg/_by_runtime.py",
]
LAUNCHER is the script a program in the tree is started through: by run
starts the interpreter on its runner, _by_runner.py, which makes the program's
entry module __main__ and runs it. none of the runner's frames is the program's,
and a debugger that launched the file LAUNCHER names treats them as python's own
runpy frames under python -m. a by build or by compile tree, which no
program is started from, says LAUNCHER = None
BY_CODE is every file of by's own python in the tree: the runner and the
plugins it gives pytest and coverage.py, the module by repl runs, the module
naming the project, the map itself, and each copy of the runtime's helpers. none
of it is the program's code, though it sits in the tree beside the program's, so
a debugger that steps through only the program's own code leaves these files
out as it leaves out the standard library, by what the map says rather than by
their names. a by build or by compile tree lists the map and its copies of the
runtime
every path is spelled as the tables spell theirs, so a consumer that resolves symlinks resolves these too
a re-stage rewrites only the lines of the entry it re-staged, so what the map says about by's own code stays as the build wrote it
last, the map names the build of by that wrote it, as _by_build.json names it:
a build is named by a hash of the by executable that wrote it. the same binary
names the same build wherever it is installed, and any rebuild names a new one, so
two binaries built apart from the same sources usually name different builds. a
running by opens its executable when it starts and hashes that, so a language
server that outlives a rebuild of by still names the binary it is running
a re-stage moves entries only in a map that names the build the tree's
_by_build.json names, and only when that build is the one running: a tree
whose map and record name different builds is refused, naming both
goals¶
- byte-accurate — map arbitrary
.bybyte offsets ↔.pybyte offsets, not just lines - bidirectional —
by_to_pyandpy_to_by, both total (return the nearest enclosing mapped span when a position is inside generated code) - whole-pipeline — composes across every phase, including the lazy-import preamble and any mid-body hoist, with no "only prepend" assumption
- reverse-aware — the reverse pipeline produces a map of the same shape
- tested — composition and round-trip are property-tested
design¶
representation: a segment list¶
model a single transformation step as an ordered list of segments, each either copied or generated:
Segment {
in: Option<Range<TextSize>>, // source bytes, None = generated
out: Range<TextSize>, // output bytes
}
a step's map is the list of segments covering the whole output in order. copied
segments carry the byte range they came from; generated segments (preamble,
hoisted classes, replacement text with no 1:1 origin) carry None. this
subsumes both line and column mapping — line lookups are derived by counting
newlines within a segment
this is the same shape as standard sourcemap "mappings", specialized to a single file pair and kept as byte ranges rather than line/column until rendered
per-phase production¶
each phase already knows its edits; each produces a segment list as a byproduct:
- phase 0 (ast_driver) — the splice applies a sorted, non-overlapping edit
list via
replace_range. walking that list yields segments directly: copied runs between edits, generated runs for replacement text, plus generated segments for the prepended imports and anyctx.hoistedinsertions (whose insertion offset is known) - phase 1 (lowering) — currently a preamble prepend only; one generated segment for the preamble, identity for the body
- import-redirect / lazy — within-line text replacements (
typing→typing_extensions,lazyprefixes) become copied/generated segment pairs; prepended polyfill preambles are generated segments - anon-NT cleanup — prepend + occasional body edit; same treatment
composition¶
compose two steps A: mid→src and B: out→mid into out→src by walking B's
segments and, for each copied segment, intersecting its mid range against A's
segments to resolve back to src (splitting where A changes origin). generated
segments stay generated. this is associative, so the pipeline map is a fold over
the per-phase maps. line-level line_table becomes a derived view of the
composed byte map, letting by run keep working unchanged
lookup¶
py_to_by(offset)— binary search the output ranges; return the mapped.byoffset, or the start of the nearest enclosing copied segment when inside generated codeby_to_py(offset)— the same against an index built on theinranges; when a.byoffset expands to multiple output spans (rare), return the first
reverse transpile¶
the reverse pipeline builds a map of the same type; consumers use one API for both directions
consumers and rollout¶
phase the work behind real consumers so nothing speculative ships:
by runtracebacks (done, line-level) — already shipping; migrate it to read the byte map's derived line view when step 2 lands, then delete the standalone line table- byte map + composition + property tests — the core; no API beyond what a consumer needs
- LSP position mapping — go-to-def / hover / diagnostics on
.byvia the generated python; the first consumer that needs columns and both directions - reverse-direction map — when an editor feature consumes
.py→.by
testing¶
- round-trip —
py_to_by(by_to_py(p)) == pfor every copied position - composition —
compose(a, b)equals mapping throughathenb, checked on randomized edit lists - golden — a handful of representative
.byfiles (lazy imports, intersections, hoisted anon-NT classes) with asserted maps at known offsets - traceback integration — the existing
by rune2e test, extended to column assertions once columns land