Why Rust?

Showing how adopting Rust may specifically help CPython, David noted how the number of issues labeled with “type-crash” has been steadily rising over time. “We’ve been making some big technical bets”, he said, referencing the new parser, the JIT, and free-threading. These large, complex features may be one of the reasons more crash reports are being opened on GitHub, and Rust could be a potential solution here. David explained that Jeff Vander Stoep described Rust in Android as “move fast and fix things”, and that “fewer revisions for patches of the same size” was the experience Android has had since adopting Rust.

Rust and Python are already working together in Python’s ecosystem of packages thanks to PyO3 and Maturin. David also emphasized that many technology companies were “choosing Rust as a bet” instead of only as the new shiny tool.

But adopting Rust into CPython would not all be smooth sailing; there were still concerns that David and the team are aware of.

One of the biggest concerns from a year ago was Rust’s lack of platform support compared to CPython, which at the time of writing officially supports 20 different architectures and platforms at either Tier 1, 2, or 3. David shared that Rust’s support of different platforms has “widened since last year” and was becoming “less and less of a concern”. The proposal would be to ask Python distributors to attempt using the optional Rust support in Python 3.16 (October 2027) and report platform-specific issues upstream in time to be resolved around the Python 3.17 timeline (October 2028).

David acknowledged that adopting Rust into a project is a social challenge as much as a technical challenge. Rust knowledge is “not universal amongst core developers”, which would be partially mitigated by “targeting small portions” of CPython and an incremental approach, as “1 million lines of C code can’t be ported all at once”.

David was clear that using Rust in itself does not necessarily mean that ported code would be free of bugs or security issues. Although Rust does mitigate classes of issues that commonly create bugs, the code can still have correctness issues. The team proposes mitigating this by adopting property-based testing, fuzzing, and using “prudent engineering practices” during the porting process.

The proposed first Rust module: zlib

Below are the proposed timelines for the Rust for CPython project making a “significant improvement” to CPython, with a PEP defining the success criteria for Rust expected in late 2026. Under this timeline, the first Rust code to ship in Python would be in Python 3.16, where it would be completely optional, with the existing C code kept as a fallback. The earliest that Rust would become required to build CPython is Python 3.18 in 2029, at least three years away.

The timeline includes a build system and CI, Rust API proof-of-concept happening in the Summer 2026, a PEP defining success criteria in late 2026, an optional Rust backend for the zlib module and private Rust API in Python 3.16 (October 2027), resolving platform issues and Rust in more places (io, json, xml, memoryview, parser) for Python 3.17 (October 2028). Finally, in some distant Python version (October 2029+) the Rust build would be made required and a public Rust API would be published.

The Rust for CPython team has selected the zlib module as the first module to be given an optional Rust implementation because they “wanted to achieve a significant improvement” with a “small scope”. The proposed Rust implementation will use zlib-rs, which is “heavily tested and used by the Firefox, uv, and Cargo” projects and “faster than zlib and zlib-ng on many platforms”. The module was also selected because it’d require using an external Cargo package, meaning this aspect of the build process would need to be designed and exercised.

This small change would have an impact: the zlib compression algorithm is “widely used by Python packaging”, meaning that (almost) “every pip install in Python 3.16 will be sped up” if the proposal is accepted.

Rust API Sketch

David provided an example of some Rust code calling a hypothetical Rust API for Python. The API would use a Rust attribute (#[pyfunction]) and be similar to Argument Clinic, a development tool for automatically generating blocks of code for handling C function arguments from Pythonic function syntax. Rust functions would always be passed the thread and interpreter state (Python<'_>), use smart pointers around objects (Py<...>), and lean into Rust error handling with the Result enum, returning either a result or an error that was raised.

#[pyfunction(signature = (
    data,
    /,
    wbits=MAX_WBITS,
    bufsize=DEF_BUF_SIZE,
))]
fn decompress(
    py: Python<'_>,
    data: Py<PyObject>,
    wbits: c_int,
    bufsize: isize
) -> PyResult<Py<PyBytes>, PyErrRaised> {

    // buf will be cleaned on scope exit
    let buf = PyObject::get_buffer(py, &data)?;
    let decoded = /* ... */;

    Ok(PyBytes::new(py, &decoded))
}

The example above shows a buffer being allocated and automatically cleaned up on scope exit, rather than being cleaned up manually as would be required when writing the same function in C.

Proposed success criteria

David moved on to success criteria: what would the Rust for CPython project need to show to move into new phases of the roadmap and eventually become a required part of building CPython? “For previous big changes like the JIT and free-threading, we’ve explicitly defined success criteria that would need to be met in order for the added complexity to be accepted”, David explained, “we expect we’d need to do the same here. What should those criteria be?”

David had these suggestions for core developers:

• Critical: A majority of active core developers are open to using the Rust API to implement functionality.
• No meaningful slowdown to CPython performance benchmarks.
• All tiered platforms must be supported by Rust. The experience of distributors building CPython should generally indicate that adding Rust support is manageable.

David noted that the first point reads “open”, not “familiar”, and shared a plan to survey Python core developers about how they’ve used Rust while contributing to CPython when deciding whether to move Rust out of experimental stages.

Discussion

On the topic of designing the new Rust API so that it’s “familiar” to users of the C API, Thomas Wouters advised against “making compromises for the dinosaurs”, including himself in the subset, instead asking whether the Rust API should be designed from first principles. David responded that there are “places to lean into Rust”, such as dropping resources on scope exit, but there are also idiomatic Rust designs which “won’t be the best fit”. David noted that it would be reasonable for core developers to be looking at both the C and Rust API at the same time while working. “We should be mindful of our audience, which is also ourselves”.

Larry Hastings asked why the Rust for CPython project wasn’t a “rewrite”, suggesting the team “display your success as a fork”. David acknowledged that “RustPython already exists” and that the Rust for CPython team had already spoken with the contributors of the project. “RustPython isn’t as performant as CPython, but could be used to inform what APIs we design”.

Larry also shared that he “wasn’t super excited to learn Rust to work on CPython”. David assured him that there are many areas of CPython that would not be considered for writing in Rust: “CPython should not be written in Rust for the sake of Rust”. “CPython will be a dual-language project for a meaningful amount of time”. However, David cautioned that “it would be disingenuous to say that Rust would be optional forever”, as one of the aforementioned roadmap items for the Rust for CPython project is to become a required part of the build process and provide a public Rust API.

Photo by Hugo van Kemenade (CC BY-NC-SA 4.0)

Pablo Galindo Salgado was more concerned about the future, which was “reaching for our dependencies from Cargo”, noting that this would be a “huge problem” and a potential “showstopper” for the project. “We vendor our dependencies, and we have a very selective set”, he said, noting that each time a vulnerability is published for one of those projects, the release managers need to make new releases, which can be “tiresome”. “Right now we’re only focusing on the APIs and the basics”, he added, highlighting that the challenge of taking on many Rust dependencies hasn’t been addressed yet.

David answered that the team should “select as few external dependencies as possible”; “zlib-rs is only one dependency”, and the Rust sources would be vendored so that “building CPython would not require Cargo”. David added that “Cargo has a relatively clean system for vendoring dependencies” and that the “Rust for CPython proof-of-concept uses this system”. The vendored sources “wouldn’t live in the CPython tree”. Łukasz Langa agreed that “it’s better for dependencies to live separately”, referencing the cpython-source-deps repository, with Thomas reminding everyone that this would be a new usage of that repository; today it is only used for binary installers of CPython.

<-Back to all posts