Add support for the Pyodide package index#1953
Draft
abstractedfox wants to merge 2 commits into
Draft
Conversation
|
agents
@cloudflare/ai-chat
@cloudflare/codemode
create-think
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
Add index.py as default entry point Start of basic package support This is the beginning of work on this feature, it's a little comment-heavy to that end and has some patterns that may not be permanent (these have generally been noted as such). As of this commit, there are tests that start to poke at this, namely the one that brings in fastapi, which currently fails due to an extension it doesn't like. Python package proof of concept Proof of concept for package support in dynamic Python workers. As of this commit, it can retrieve an sdist with a flat layout from pypi and install it into the virtual filesystem. It doesn't do dependency resolution, so the current tests fail when attempting to import FastAPI's deps. Clarifying comment Use wheels properly Used sdists before, now it uses wheels. Pre cleanup commit Wheel changeover cleanup Use simpler packages for minimal package support test This test only needs to confirm that retrieval of a hard list of pure python packages, without dependency resolution, works. Move python logic out of 'bundle' path This seems to be specific to JS packages Slight cleanup, remove test More cleanup Re-add parameter that was mistakenly deleted Revert "remove dependency that was causing issues" This dependendency caused issues when working in a container, but its removal was purely for my sake This reverts commit 4f8990d. Order deps alphabetically Remove unused variable Make createWorker Python return more uniform with JS Make createWorker registry pattern more consistent with JS Also makes PyprojectToml interface less stringent Small fixes These got caught by the checks done by `pnpm run check` Style fixes Remove parsing for version string Comment dep version parameter Compensate for distribution packages' names not matching their imports Remove default values from test These are already set implicitly by createWorker Move Python package metadata collection into its own function Move Py deps installation into its own function Account for type checker Nested dependency installation without version resolution Undo unintended compat date change Bump up compatibility date (test) Prevent adding python and JS deps to the same worker Implicitly add workers-runtime-sdk package Remove check for when there are no dependencies, since there will always be at least one Add python_workers flag if user-supplied flags exclude it Typo Clarifying comment Include installWarnings with returned object Type checker fix 1st pass moving to 'simple API' on PyPI A fair amount of this is AI code, a beautification pass would be beneficial Merge from python-package-resolution Change tests to assert existence of compat flags Remove redundant compat flag default This is implicit on Python dynamic workers now Mark comment as TODO Finish propagating `result` object and explicitly disable pyproject.toml and package.json in one worker Retrieve pyodide lockfile Add pyodide cdn as a fallback index Would be good to alter the flow next so pyodide can be deliberately chosen as the primary index, and so tests can be written for this (forcing the issue shows that it does work, but it's not conducive to testing) Remove remnant from brute force testing Split pypi and pyodide logic Proper typing Allow the caller to prefer the pyodide index Fix nested deps with Pyodide index + make this cleaner Small fixes Clear warning There were warnings in the console about a binary/octet-stream response being parsed as text. Since the response actually is text, this avoids that warning. Working test for pyodide index Remove redundant code Fix broken code Missed this when deleting the original Set pyodide index as default Change return type Reconstitute work on this repo Add index.py as default entry point Start of basic package support This is the beginning of work on this feature, it's a little comment-heavy to that end and has some patterns that may not be permanent (these have generally been noted as such). As of this commit, there are tests that start to poke at this, namely the one that brings in fastapi, which currently fails due to an extension it doesn't like. Python package proof of concept Proof of concept for package support in dynamic Python workers. As of this commit, it can retrieve an sdist with a flat layout from pypi and install it into the virtual filesystem. It doesn't do dependency resolution, so the current tests fail when attempting to import FastAPI's deps. Clarifying comment Use wheels properly Used sdists before, now it uses wheels. Pre cleanup commit Wheel changeover cleanup Use simpler packages for minimal package support test This test only needs to confirm that retrieval of a hard list of pure python packages, without dependency resolution, works. Move python logic out of 'bundle' path This seems to be specific to JS packages Slight cleanup, remove test More cleanup Re-add parameter that was mistakenly deleted Order deps alphabetically Remove unused variable Make createWorker Python return more uniform with JS Make createWorker registry pattern more consistent with JS Also makes PyprojectToml interface less stringent Small fixes These got caught by the checks done by `pnpm run check` Style fixes Remove parsing for version string Comment dep version parameter Compensate for distribution packages' names not matching their imports Remove default values from test These are already set implicitly by createWorker Move Python package metadata collection into its own function Move Py deps installation into its own function Account for type checker Undo unintended compat date change Bump up compatibility date (test) Prevent adding python and JS deps to the same worker Implicitly add workers-runtime-sdk package Remove check for when there are no dependencies, since there will always be at least one Add python_workers flag if user-supplied flags exclude it Typo Clarifying comment Include installWarnings with returned object Change tests to assert existence of compat flags Remove redundant compat flag default This is implicit on Python dynamic workers now Mark comment as TODO Change installDependenciesPython to replicate the JS impl's usage of the result object Finish propagating `result` object and explicitly disable pyproject.toml and package.json in one worker Add more TODOs
abstractedfox
force-pushed
the
python-package/pyodide-index
branch
from
July 24, 2026 05:59
30a9599 to
819c561
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The short description is that this adds the Pyodide package index as a possible index to retrieve packages from. The slightly longer description is that it allows either PyPI or the Pyodide index to be preferred as the main index, with the other serving as a fallback if that one fails for any reason. I moved a lot of things around to try to make that clean since they work fairly differently, so let me know if anyone sees anything that looks weird.
Addresses a comment made by @ryanking13 on #1881 and builds on #1950, putting this up as a draft until that gets merged since the number of commits will look excessive until then