extensions:pip.bzl

This is the successor to pip_parse for including third party PyPI dependencies into your bazel module using bzlmod.

See also

The environment markers specification for the explanation of the terms used in this extension.

Added in version 1.6.0.

Attributes:

config_settings – (list[label])
The list of labels to config_setting targets that need to be matched for the platform to be selected.

mandatory
arch_name – (str) (default “”)
The CPU architecture name to be used. You can use any cpu name from the @platforms//cpu: package.

Note

Either this or env platform_machine key should be specified.

optional
auth_patterns – (dict[str, str]) (default {})
An optional dict mapping host names to custom authorization patterns.

If a URL’s host name is present in this dict the value will be used as a pattern when generating the authorization header for the http request. This enables the use of custom authorization schemes used in a lot of common cloud storage providers.

The pattern currently supports 2 tokens: <login> and <password>, which are replaced with their equivalent value in the netrc file for the same host name. After formatting, the result is set as the value for the Authorization field of the HTTP request.

Example attribute and netrc for a http download to an oauth2 enabled API using a bearer token:
```
auth_patterns = {
    "storage.cloudprovider.com": "Bearer <password>"
}
```
netrc:
```
machine storage.cloudprovider.com
        password RANDOM-TOKEN
```
The final HTTP request would have the following header:
```
Authorization: Bearer RANDOM-TOKEN
```
optional
env – (dict[str, str]) (default {})
The values to use for environment markers when evaluating an expression.

The keys and values should be compatible with the PyPA dependency specifiers specification.

Missing values will be set to the specification’s defaults or computed using available toolchain information.

Supported keys:
- implementation_name, defaults to cpython.
- os_name, defaults to a value inferred from the os_name.
- platform_machine, defaults to a value inferred from the arch_name.
- platform_release, defaults to an empty value.
- platform_system, defaults to a value inferred from the os_name.
- platform_version, defaults to 0.
- sys_platform, defaults to a value inferred from the os_name.
Note

This is only used if the RULES_PYTHON_ENABLE_PIPSTAR is enabled.

optional
marker – (str) (default “”)
An environment marker expression that is used to enable/disable platforms for specific python versions, operating systems or CPU architectures.

If specified, the expression is evaluated during the bzlmod extension evaluation phase and if it evaluates to True, then the platform will be used to construct the hub repositories, otherwise, it will be skipped.

This is especially useful for setting up freethreaded platform variants only for particular Python versions for which the interpreter builds are available. However, this could be also used for other things, such as setting up platforms for different libc variants.

optional
netrc – (str) (default “”)
Location of the .netrc file to use for authentication

optional
os_name – (str) (default “”)
The OS name to be used. You can use any OS name from the @platforms//os: package.

Note

Either this or the appropriate env keys should be specified.

optional
platform – (str) (default “”)
A platform identifier which will be used as the unique identifier within the extension evaluation. If you are defining custom platforms in your project and don’t want things to clash, use extension isolation feature.

optional
whl_abi_tags – (list[str]) (default [])
A list of ABIs to select wheels for. The values can be either strings or include template parameters like {major} and {minor} which will be replaced with python version parts. e.g. cp{major}{minor} will result in cp313 given the full python version is 3.13.5. Will always include "none" even if it is not specified.

Note

We select a single wheel and the last match will take precedence.

See also

See official docs for more information.

optional
whl_platform_tags – (list[str]) (default [])
A list of platform_tag matchers so that we can select the best wheel based on the user preference. Will always include "any" even if it is not specified.

The items in this list can contain a single * character that is equivalent to matching the lowest available version component in the platform_tag. If the wheel platform tag does not have a version component, e.g. linux_x86_64 or win_amd64, then * will act as a regular character.
Note

Normally, the * in the matcher means that we will target the lowest platform version that we can and will give preference to whls built targeting the older versions of the platform. If you specify the version, then we will use the MVS (Minimal Version Selection) algorithm to select the compatible wheel. As such, you need to keep in mind how to configure the target platforms to select a particular wheel of your preference.

We select a single wheel and the last match will take precedence, if the platform_tag that we match has a version component (e.g. android_x_arch, then the version x will be used in the MVS matching algorithm).

Common patterns:
- To select any versioned wheel for an <os>, <arch>, use <os>_*_<arch>, e.g. manylinux_2_17_x86_64.
- To exclude versions up to X.Y - submit a PR supporting this feature.
- To exclude versions above X.Y, provide the full platform tag specifier, e.g. musllinux_1_2_x86_64, which will ensure that no wheels with musllinux_1_3_x86_64 or higher are selected.
See also

See official docs for more information.

Changed in version 1.6.3: The matching of versioned platforms have been switched to MVS (Minimal Version Selection) algorithm for easier evaluation logic and fewer surprises. The legacy platform tags are supported from this version without extra handling from the user.

optional

pip.override(file, patches, patch_strip=0)

Apply any overrides (e.g. patches) to a given Python distribution defined by other tags in this extension.

Attributes:

file – (str)
The Python distribution file name which needs to be patched. This will be applied to all repositories that setup this distribution via the pip.parse tag class.

mandatory
patches – (list[label])
A list of patches to apply to the repository after ‘whl_library’ is extracted and BUILD.bazel file is generated.

mandatory
patch_strip – (int) (default 0)
The number of leading path segments to be stripped from the file name in the patches.

optional

pip.parse(hub_name, python_version, add_libdir_to_library_search_path=False, auth_patterns={}, download_only=False, enable_implicit_namespace_pkgs=False, environment={}, envsubst=[], experimental_extra_index_urls=[], experimental_index_url='', experimental_index_url_overrides={}, experimental_requirement_cycles={}, experimental_target_platforms=[], extra_hub_aliases={}, extra_pip_args=[], isolated=True, netrc='', parallel_download=True, pip_data_exclude=[], python_interpreter='', python_interpreter_target=None, quiet=True, requirements_by_platform={}, requirements_darwin=None, requirements_linux=None, requirements_lock=None, requirements_windows=None, simpleapi_skip=[], target_platforms=[], timeout=600, use_hub_alias_dependencies=False, whl_modifications={})

This tag class is used to create a pip hub and all of the spokes that are part of that hub. This tag class reuses most of the attributes found in pip_parse. The exception is it does not use the arg ‘repo_prefix’. We set the repository prefix for the user and the alias arg is always True in bzlmod.

Attributes:

hub_name – (str)
The name of the repo pip dependencies will be accessible from.

This name must be unique between modules; unless your module is guaranteed to always be the root module, it’s highly recommended to include your module name in the hub name. Repo mapping, use_repo(..., pip="my_modules_pip_deps"), can be used for shorter local names within your module.

Within a module, the same hub_name can be specified to group different Python versions of pip dependencies under one repository name. This allows using a Python version-agnostic name when referring to pip dependencies; the correct version will be automatically selected.

Typically, a module will only have a single hub of pip dependencies, but this is not required. Each hub is a separate resolution of pip dependencies. This means if different programs need different versions of some library, separate hubs can be created, and each program can use its respective hub’s targets. Targets from different hubs should not be used together.

mandatory
python_version – (str)
The Python version the dependencies are targetting, in Major.Minor format (e.g., “3.11”) or patch level granularity (e.g. “3.11.1”).

If an interpreter isn’t explicitly provided (using python_interpreter or python_interpreter_target), then the version specified here must have a corresponding python.toolchain() configured.

mandatory
add_libdir_to_library_search_path – (bool) (default False)
If true, add the lib dir of the bundled interpreter to the library search path via LDFLAGS.

Added in version 1.3.0.

optional
auth_patterns – (dict[str, str]) (default {})
An optional dict mapping host names to custom authorization patterns.

If a URL’s host name is present in this dict the value will be used as a pattern when generating the authorization header for the http request. This enables the use of custom authorization schemes used in a lot of common cloud storage providers.

The pattern currently supports 2 tokens: <login> and <password>, which are replaced with their equivalent value in the netrc file for the same host name. After formatting, the result is set as the value for the Authorization field of the HTTP request.

Example attribute and netrc for a http download to an oauth2 enabled API using a bearer token:
```
auth_patterns = {
    "storage.cloudprovider.com": "Bearer <password>"
}
```
netrc:
```
machine storage.cloudprovider.com
        password RANDOM-TOKEN
```
The final HTTP request would have the following header:
```
Authorization: Bearer RANDOM-TOKEN
```
optional
download_only – (bool) (default False)
Whether to use “pip download” instead of “pip wheel”. Disables building wheels from source, but allows use of –platform, –python-version, –implementation, and –abi in –extra_pip_args to download wheels for a different platform from the host platform.

optional
enable_implicit_namespace_pkgs – (bool) (default False)
If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary and py_test targets must specify either legacy_create_init=False or the global Bazel option --incompatible_default_to_explicit_init_py to prevent __init__.py being automatically generated in every directory.

This option is required to support some packages which cannot handle the conversion to pkg-util style.

optional
environment – (dict[str, str]) (default {})
Environment variables to set in the pip subprocess. Can be used to set common variables such as http_proxy, https_proxy and no_proxy Note that pip is run with “–isolated” on the CLI so PIP_<VAR>_<NAME> style env vars are ignored, but env vars that control requests and urllib3 can be passed. If you need PIP_<VAR>_<NAME>, take a look at extra_pip_args and envsubst.

optional
envsubst – (list[str]) (default [])
A list of environment variables to substitute (e.g. ["PIP_INDEX_URL", "PIP_RETRIES"]). The corresponding variables are expanded in extra_pip_args using the syntax $VARNAME or ${VARNAME} (expanding to empty string if unset) or ${VARNAME:-default} (expanding to default if the variable is unset or empty in the environment). Note: On Bazel 6 and Bazel 7.0 changes to the variables named here do not cause packages to be re-fetched. Don’t fetch different things based on the value of these variables.

optional
experimental_extra_index_urls – (list[str]) (default [])
The extra index URLs to use for downloading wheels using bazel downloader. Each value is going to be subject to envsubst substitutions if necessary.

The indexes must support Simple API as described here: https://packaging.python.org/en/latest/specifications/simple-repository-api/

This is equivalent to --extra-index-urls pip option.

Changed in version 1.1.0: Starting with this version we will iterate over each index specified until we find metadata for all references distributions.

optional
experimental_index_url – (str) (default “”)
The index URL to use for downloading wheels using bazel downloader. This value is going to be subject to envsubst substitutions if necessary.

The indexes must support Simple API as described here: https://packaging.python.org/en/latest/specifications/simple-repository-api/

In the future this could be defaulted to https://pypi.org when this feature becomes stable.

This is equivalent to --index-url pip option.

Changed in version 0.37.0: If download_only is set, then sdist archives will be discarded and pip.parse will operate in wheel-only mode.

Changed in version 1.4.0: Index metadata will be used to deduct sha256 values for packages even if the sha256 values are not present in the requirements.txt lock file.

optional
experimental_index_url_overrides – (dict[str, str]) (default {})
The index URL overrides for each package to use for downloading wheels using bazel downloader. This value is going to be subject to envsubst substitutions if necessary.

The key is the package name (will be normalized before usage) and the value is the index URL.

This design pattern has been chosen in order to be fully deterministic about which packages come from which source. We want to avoid issues similar to what happened in https://pytorch.org/blog/compromised-nightly-dependency/.

The indexes must support Simple API as described here: https://packaging.python.org/en/latest/specifications/simple-repository-api/

optional

experimental_requirement_cycles – (dict[str, list[str]]) (default {})

A mapping of dependency cycle names to a list of requirements which form that cycle.

Requirements which form cycles will be installed together and taken as dependencies together in order to ensure that the cycle is always satisified.

Example: sphinx depends on sphinxcontrib-serializinghtml When listing both as requirements, ala

py_binary(
  name = "doctool",
  ...
  deps = [
    "@pypi//sphinx:pkg",
    "@pypi//sphinxcontrib_serializinghtml",
   ]
)

Will produce a Bazel error such as

ERROR: .../external/pypi_sphinxcontrib_serializinghtml/BUILD.bazel:44:6: in alias rule @pypi_sphinxcontrib_serializinghtml//:pkg: cycle in dependency graph:
    //:doctool (...)
    @pypi//sphinxcontrib_serializinghtml:pkg (...)
.-> @pypi_sphinxcontrib_serializinghtml//:pkg (...)
|   @pypi_sphinxcontrib_serializinghtml//:_pkg (...)
|   @pypi_sphinx//:pkg (...)
|   @pypi_sphinx//:_pkg (...)
`-- @pypi_sphinxcontrib_serializinghtml//:pkg (...)

Which we can resolve by configuring these two requirements to be installed together as a cycle

pip_parse(
  ...
  experimental_requirement_cycles = {
    "sphinx": [
      "sphinx",
      "sphinxcontrib-serializinghtml",
    ]
  },
)

Warning: If a dependency participates in multiple cycles, all of those cycles must be collapsed down to one. For instance a <-> b and a <-> c cannot be listed as two separate cycles.

optional

experimental_target_platforms – (list[str]) (default [])
NOTE: This will be removed in the next major version, so please consider migrating to bzlmod and rely on pip.parse.requirements_by_platform for this feature.

A list of platforms that we will generate the conditional dependency graph for cross platform wheels by parsing the wheel metadata. This will generate the correct dependencies for packages like sphinx or pylint, which include colorama when installed and used on Windows platforms.

An empty list means falling back to the legacy behaviour where the host platform is the target platform.

WARNING: It may not work as expected in cases where the python interpreter implementation that is being used at runtime is different between different platforms. This has been tested for CPython only.

For specific target platforms use values of the form <os>_<arch> where <os> is one of linux, osx, windows and arch is one of x86_64, x86_32, aarch64, s390x and ppc64le.

You can also target a specific Python version by using cp3<minor_version>_<os>_<arch>. If multiple python versions are specified as target platforms, then select statements of the lib and whl targets will include usage of version aware toolchain config settings like @rules_python//python/config_settings:is_python_3.y.

Special values: host (for generating deps for the host platform only) and <prefix>_* values. For example, cp39_*, linux_*, cp39_linux_*.

NOTE: this is not for cross-compiling Python wheels but rather for parsing the whl METADATA correctly.

optional
extra_hub_aliases – (dict[str, list[str]]) (default {})
Extra aliases to make for specific wheels in the hub repo. This is useful when paired with the whl_modifications.

Added in version 0.38.0: For pip.parse with bzlmod

Added in version 1.0.0: For pip_parse with workspace.

optional
extra_pip_args – (list[str]) (default [])
Extra arguments to pass on to pip. Must not contain spaces.

Supports environment variables using the syntax $VARNAME or ${VARNAME} (expanding to empty string if unset) or ${VARNAME:-default} (expanding to default if the variable is unset or empty in the environment), if "VARNAME" is listed in the envsubst attribute. See also envsubst.

optional
isolated – (bool) (default True)
Whether or not to pass the –isolated flag to the underlying pip command. Alternatively, the RULES_PYTHON_PIP_ISOLATED environment variable can be used to control this flag.

optional
netrc – (str) (default “”)
Location of the .netrc file to use for authentication

optional
parallel_download – (bool) (default True)
The flag allows to make use of parallel downloading feature in bazel 7.1 and above when the bazel downloader is used. This is by default enabled as it improves the performance by a lot, but in case the queries to the simple API are very expensive or when debugging authentication issues one may want to disable this feature.

NOTE, This will download (potentially duplicate) data for multiple packages if there is more than one index available, but in general this should be negligible because the simple API calls are very cheap and the user should not notice any extra overhead.

If we are in synchronous mode, then we will use the first result that we find in case extra indexes are specified.

optional
pip_data_exclude – (list[str]) (default [])
Additional data exclusion parameters to add to the pip packages BUILD file.

optional
python_interpreter – (str) (default “”)
The python interpreter to use. This can either be an absolute path or the name of a binary found on the host’s PATH environment variable. If no value is set python3 is defaulted for Unix systems and python.exe for Windows.

optional
python_interpreter_target – (label) (default None)
If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. An example value: “@python3_x86_64-unknown-linux-gnu//:python”.

optional
quiet – (bool) (default True)
If True, suppress printing stdout and stderr output to the terminal.

If you would like to get more diagnostic output, set RULES_PYTHON_REPO_DEBUG=1 or RULES_PYTHON_REPO_DEBUG_VERBOSITY=INFO|DEBUG|TRACE

optional
requirements_by_platform – (dict[label, str]) (default {})
The requirements files and the comma delimited list of target platforms as values.

The keys are the requirement files and the values are comma-separated platform identifiers. For now we only support <os>_<cpu> values that are present in @platforms//os and @platforms//cpu packages respectively.

optional
requirements_darwin – (label) (default None)
Override the requirements_lock attribute when the host platform is Mac OS

optional
requirements_linux – (label) (default None)
Override the requirements_lock attribute when the host platform is Linux

optional
requirements_lock – (label) (default None)
A fully resolved ‘requirements.txt’ pip requirement file containing the transitive set of your dependencies. If this file is passed instead of ‘requirements’ no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by ‘build/run/test’. Note that if your lockfile is platform-dependent, you can use the requirements_[platform] attributes.

Note, that in general requirements files are compiled for a specific platform, but sometimes they can work for multiple platforms. rules_python right now supports requirements files that are created for a particular platform without platform markers.

optional
requirements_windows – (label) (default None)
Override the requirements_lock attribute when the host platform is Windows

optional
simpleapi_skip – (list[str]) (default [])
The list of packages to skip fetching metadata for from SimpleAPI index. You should normally not need this attribute, but in case you do, please report this as a bug to rules_python and use this attribute until the bug is fixed.

EXPERIMENTAL: this may be removed without notice.

Added in version 1.4.0.

optional
target_platforms – (list[str]) (default [])
The list of platforms for which we would evaluate the requirements files. If you need to be able to only evaluate for a particular platform (e.g. “linux_x86_64”), then put it in here.

If you want freethreaded variant, then you can use _freethreaded suffix as rules_python is defining target platforms for these variants in its MODULE.bazel file. The identifiers for this function in general are the same as used in the pip.default.platform attribute.

If you only care for the host platform and do not have a usecase to cross-build, then you can put in a string "{os}_{arch}" as the value here. You could also use "{os}_{arch}_freethreaded" as well.

Warning

Experimental API. This API is still under development and may change or be removed without notice.

Added in version 1.8.0.

optional
timeout – (int) (default 600)
Timeout (in seconds) on the rule’s execution duration.

optional
use_hub_alias_dependencies – (bool) (default False)
Controls if the hub alias dependencies are used. If set to true, then the group_library will be included in the hub repo.

True will become default in a subsequent release.

optional
whl_modifications – (dict[label, str]) (default {})
A dict of labels to wheel names that is typically generated by the whl_modifications. The labels are JSON config files describing the modifications.

optional

pip.whl_mods(hub_name, whl_name, additive_build_content='', additive_build_content_file=None, copy_executables={}, copy_files={}, data=[], data_exclude_glob=[], srcs_exclude_glob=[])

This tag class is used to create JSON file that are used when calling wheel_builder.py. These JSON files contain instructions on how to modify a wheel’s project. Each of the attributes create different modifications based on the type of attribute. Previously to bzlmod these JSON files where referred to as annotations, and were renamed to whl_modifications in this extension.

Attributes:

hub_name – (str)
Name of the whl modification, hub we use this name to set the modifications for pip.parse. If you have different pip hubs you can use a different name, otherwise it is best practice to just use one.

You cannot have the same hub_name in different modules. You can reuse the same name in the same module for different wheels that you put in the same hub, but you cannot have a child module that uses the same hub_name.

mandatory
whl_name – (str)
The whl name that the modifications are used for.

mandatory
additive_build_content – (str) (default “”)
(str, optional): Raw text to add to the generated BUILD file of a package.

optional
additive_build_content_file – (label) (default None)
(label, optional): path to a BUILD file to add to the generated BUILD file of a package. You cannot use both additive_build_content and additive_build_content_file arguments at the same time.

optional
copy_executables – (dict[str, str]) (default {})
(dict, optional): A mapping of src and out files for [@bazel_skylib//rules:copy_file.bzl][cf]. Targets generated here will also be flagged as executable.

optional
copy_files – (dict[str, str]) (default {})
(dict, optional): A mapping of src and out files for [@bazel_skylib//rules:copy_file.bzl][cf]

optional
data – (list[str]) (default [])
(list, optional): A list of labels to add as data dependencies to the generated py_library target.

optional
data_exclude_glob – (list[str]) (default [])
(list, optional): A list of exclude glob patterns to add as data to the generated py_library target.

optional
srcs_exclude_glob – (list[str]) (default [])
(list, optional): A list of labels to add as srcs to the generated py_library target.

optional