Skip to content

regex group types

a regular expression written out as a literal says exactly how many capture groups it has, what they are named, and which of them must have participated in any successful match. the re stubs cannot express any of that — every group is typed the same way whatever the pattern — so basedpython reads the pattern and types the result from it:

import re

if m := re.match("()?()", text):
    reveal_type(m[1])        # str | None
    reveal_type(m[2])        # str
    reveal_type(m.groups())  # tuple[str | None, str]

group 2 is inside no optional construct, so it is set whenever the match succeeded; group 1 sits under a ?, so it may not be.

this is an enhancement with no new syntax — it applies to .py files as well as .by.

where the groups come from

a pattern is read wherever it reaches re as a literal, and the groups travel with the compiled pattern:

p = re.compile(r"(?P<key>\w+)=(?P<value>.*)")

if m := p.search(line):
    reveal_type(m.group("key"))    # str
    reveal_type(m.groupdict())     # a TypedDict with 'key' and 'value'

match, fullmatch, search and finditer all carry the groups onto the Match they produce, and sub / subn pass them to a callable replacement:

re.sub("()?()", lambda m: str(m.groups()), text)  # m.groups() is tuple[str | None, str]

the whole surface

expression with "(a)(b)?"
m[0], m.group(0) str
m[1], m.group(1) str
m[2], m.group(2) str \| None
m.group(1, 2) tuple[str, str \| None]
m.groups() tuple[str, str \| None]
m.groups(0) tuple[str, str \| Literal[0]]
m.groupdict() a TypedDict over the named groups
re.findall(...) list[tuple[str, str]]
re.split(...) list[str \| None]

a bytes pattern gives bytes throughout.

groupdict() keeps the stub's plain dict when the pattern names no groups: an empty TypedDict would say nothing extra about a dict that is always empty, while costing the caller everything a dict can be passed to.

findall is the one place a group that did not participate is reported as the empty string rather than None, so its elements are never optional.

when a group is guaranteed

a group is set on every successful match unless something on the way to it could have been skipped:

"(a)"           # set
"(a)+"          # set — the repeat matches at least once
"(a)?"          # optional
"(a)*"          # optional
"(a){0,3}"      # optional
"(a)|(b)"       # both optional — only one branch ran
"(?:(a))"       # set — a non-capturing group changes nothing
"(?=(a))"       # set — a positive lookahead does capture
"(?!(a))"       # optional — if the match succeeded, this did not
"(?(1)(a)|(b))" # both optional — the condition is not evaluated statically

verbose mode changes what a pattern even parses to, so the flags argument is read too — re.X, re.VERBOSE, a | chain containing either, or a pattern that turns it on itself with (?x). where the flags cannot be read statically, nothing is refined rather than guessed.

patterns we cannot see

a pattern built at runtime keeps the stub's own answer, which in basedpython still says out loud that a group may not have participated — upstream typeshed types those positions as Any, which silently hides the None:

def f(pattern: str, text: str) -> None:
    if m := re.match(pattern, text):
        reveal_type(m.group(1))  # str | None

diagnostics

the same reading of the pattern catches two mistakes that otherwise only fail at runtime, both reported as invalid-regex:

re.compile("(")  # error: missing ), unterminated subpattern at position 0

if m := re.match("(a)(b)", text):
    m.group(3)   # error: No such group: 3

the group check also covers m[3], m.start(3), m.end(3) and m.span(3).

an invalid pattern written where it is passed is reported at the character python names the position of, not the whole literal: the ( left open, the ) that closes nothing. a character written as an escape, \x28, is reported as the whole escape, a pattern written as adjacent literals at the literal that has the character, and an error about a construct as a whole, {2,1} or a group's redefined name, at all of it. a pattern passed by name is reported at the name.

escapes are read by re's own rules for the python version checked against: an ASCII letter or digit after a backslash that is no escape, \q, is a bad escape rather than the letter, a bytes pattern has no \u, \U or \N, \z is an anchor only from 3.14, and a backreference has to name a group already closed. each is reported at the whole escape.

the name in a \N{...} is looked up as unicodedata.lookup looks it up in the python version checked against — in any case, among the characters' names and their aliases — so a name that python's unicode database does not have yet, or a named sequence, which stands for several characters, is reported too:

re.compile(r"\N{EM  DASH}")  # error: undefined character name 'EM  DASH' at position 0

a character is named from the python that ships the unicode version assigning it, which is read off each python's own unicodedata. two kinds of name are left to python: one derived from the code point, CJK UNIFIED IDEOGRAPH-4E00 or TANGUT IDEOGRAPH-17000, which pythons derive differently, and any name checked against a python newer than 3.15, whose unicode is newer than the table the names are read from

a construct the pattern reader does not model produces neither a refinement nor a diagnostic: a pattern reported invalid when re.compile would have accepted it is much worse than one left alone.