//python:pip.bzl

Rules for pip integration.

This contains a set of rules that are used to support inclusion of third-party dependencies via fully locked requirements.txt files. Some of the exported symbols should not be used and they are either undocumented here or marked as for internal use only.

If you are using a bazel version 7 or above with bzlmod, you should only care about the compile_pip_requirements macro exposed in this file. The rest of the symbols are for legacy WORKSPACE setups.

compile_pip_requirements(name, srcs=None, src=None, extra_args=[], extra_deps=[], generate_hashes=True, py_binary='<function py_binary from //python:py_binary.bzl>', py_test='<function py_test from //python:py_test.bzl>', requirements_in=None, requirements_txt=None, requirements_darwin=None, requirements_linux=None, requirements_windows=None, visibility=['//visibility:private'], tags=None, constraints=[], **kwargs)

Generates targets for managing pip dependencies with pip-compile (piptools).

By default this rules generates a filegroup named “[name]” which can be included in the data of some other compile_pip_requirements rule that references these requirements (e.g. with -r ../other/requirements.txt). It also generates two targets for running pip-compile:

validate with bazel test [name].test
update with bazel run [name].update

If you are using a version control system, the requirements.txt generated by this rule should be checked into it to ensure that all developers/users have the same dependency versions.

Args:

name – base name for generated targets, typically “requirements”.
srcs – (default None)
a list of files containing inputs to dependency resolution. If not specified, defaults to ["pyproject.toml"]. Supported formats are:
- a requirements text file, usually named requirements.in
- A .toml file, where the project.dependencies list is used as per PEP621.
src – (default None)
file containing inputs to dependency resolution. If not specified, defaults to pyproject.toml. Supported formats are:
- a requirements text file, usually named requirements.in
- A .toml file, where the project.dependencies list is used as per PEP621.
extra_args – (default [])
passed to pip-compile (aka piptools). See the pip-compile docs for args and meaning (passing -h and/or --version can help inform what args are available)
extra_deps – (default [])
extra dependencies passed to pip-compile.
generate_hashes – (default True)
whether to put hashes in the requirements_txt file.
py_binary – (default ‘<function py_binary from //python:py_binary.bzl>’)
the py_binary rule to be used.
py_test – (default ‘<function py_test from //python:py_test.bzl>’)
the py_test rule to be used.
requirements_in – (default None)
file expressing desired dependencies. Deprecated, use src or srcs instead.
requirements_txt – (default None)
result of “compiling” the requirements.in file.
requirements_darwin – (default None)
File of darwin specific resolve output to check validate if requirement.in has changes.
requirements_linux – (default None)
File of linux specific resolve output to check validate if requirement.in has changes.
requirements_windows – (default None)
File of windows specific resolve output to check validate if requirement.in has changes.
visibility – (default [“//visibility:private”])
passed to both the _test and .update rules.
tags – (default None)
tagging attribute common to all build rules, passed to both the _test and .update rules.
constraints – (default [])
a list of files containing constraints to pass to pip-compile with --constraint.
kwargs – other bazel attributes passed to the “_test” rule.

multi_pip_parse(name, default_version, python_versions, python_interpreter_target, requirements_lock, minor_mapping, **kwargs)

NOT INTENDED FOR DIRECT USE!

This is intended to be used by the multi_pip_parse implementation in the template of the multi_toolchain_aliases repository rule.

Args:

name – the name of the multi_pip_parse repository.
default_version – (str)
the default Python version.
python_versions – (list[str])
all Python toolchain versions currently registered.
python_interpreter_target – (dict[str, Label])
a dictionary which keys are Python versions and values are resolved host interpreters.
requirements_lock – (dict[str, Label])
a dictionary which keys are Python versions and values are locked requirements files.
minor_mapping – (dict[str, str])
mapping between X.Y to X.Y.Z format.
kwargs – extra arguments passed to all wrapped pip_parse.

Returns:

The internal implementation of multi_pip_parse repository rule.

package_annotation(additive_build_content=None, copy_files={}, copy_executables={}, data=[], data_exclude_glob=[], srcs_exclude_glob=[])

Annotations to apply to the BUILD file content from package generated from a pip_repository rule.

Args:

additive_build_content – (default None)
Raw text to add to the generated BUILD file of a package.
copy_files – (default {})
A mapping of src and out files for @bazel_skylib//rules:copy_file.bzl
copy_executables – (default {})
A mapping of src and out files for @bazel_skylib//rules:copy_file.bzl. Targets generated here will also be flagged as executable.
data – (default [])
A list of labels to add as data dependencies to the generated py_library target.
data_exclude_glob – (default [])
A list of exclude glob patterns to add as data to the generated py_library target.
srcs_exclude_glob – (default [])
A list of labels to add as srcs to the generated py_library target.

Returns:

str: A json encoded string of the provided content.

repo rule pip_parse(name, repo_mapping, add_libdir_to_library_search_path=False, annotations={}, download_only=False, enable_implicit_namespace_pkgs=False, environment={}, envsubst=[], experimental_requirement_cycles={}, experimental_target_platforms=[], extra_hub_aliases={}, extra_pip_args=[], isolated=True, pip_data_exclude=[], python_interpreter='', python_interpreter_target=None, quiet=True, requirements_by_platform={}, requirements_darwin=None, requirements_linux=None, requirements_lock=None, requirements_windows=None, timeout=600, use_hub_alias_dependencies=False)

Accepts a locked/compiled requirements file and installs the dependencies listed within.

Those dependencies become available in a generated requirements.bzl file. You can instead check this requirements.bzl file into your repo, see the “vendoring” section below.

For advanced use-cases, such as handling multi-platform dependencies, see the How-to: Multi-Platform PyPI Dependencies guide.

In your WORKSPACE file:

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "pypi",
    requirements_lock = ":requirements.txt",
)

load("@pypi//:requirements.bzl", "install_deps")

install_deps()

You can then reference installed dependencies from a BUILD file with the alias targets generated in the same repo, for example, for PyYAML we would have the following:

@pypi//pyyaml and @pypi//pyyaml:pkg both point to the py_library created after extracting the PyYAML package.
@pypi//pyyaml:data points to the extra data included in the package.
@pypi//pyyaml:dist_info points to the dist-info files in the package.
@pypi//pyyaml:whl points to the wheel file that was extracted.

py_library(
    name = "bar",
    ...
    deps = [
       "//my/other:dep",
       "@pypi//numpy",
       "@pypi//requests",
    ],
)

or

load("@pypi//:requirements.bzl", "requirement")

py_library(
    name = "bar",
    ...
    deps = [
       "//my/other:dep",
       requirement("numpy"),
       requirement("requests"),
    ],
)

In addition to the requirement macro, which is used to access the generated py_library target generated from a package’s wheel, The generated requirements.bzl file contains functionality for exposing entry points as py_binary targets as well.

load("@pypi//:requirements.bzl", "entry_point")

alias(
    name = "pip-compile",
    actual = entry_point(
        pkg = "pip-tools",
        script = "pip-compile",
    ),
)

Note that for packages whose name and script are the same, only the name of the package is needed when calling the entry_point macro.

load("@pip//:requirements.bzl", "entry_point")

alias(
    name = "flake8",
    actual = entry_point("flake8"),
)

Vendoring the requirements.bzl file

In some cases you may not want to generate the requirements.bzl file as a repository rule while Bazel is fetching dependencies. For example, if you produce a reusable Bazel module such as a ruleset, you may want to include the requirements.bzl file rather than make your users install the WORKSPACE setup to generate it. See https://github.com/bazel-contrib/rules_python/issues/608

This is the same workflow as Gazelle, which creates go_repository rules with update-repos

To do this, use the “write to source file” pattern documented in https://blog.aspect.dev/bazel-can-write-to-the-source-folder to put a copy of the generated requirements.bzl into your project. Then load the requirements.bzl file directly rather than from the generated repository. See the example in rules_python/examples/pip_parse_vendored.

Attributes:

name – (Name)
A unique name for this repository.

mandatory
repo_mapping – (dict[str, str])
In WORKSPACE context only: a dictionary from local repository name to global repository name. This allows controls over workspace dependency resolution for dependencies of this repository.

For example, an entry "@foo": "@bar" declares that, for any time this repository depends on @foo (such as a dependency on @foo//some:target, it should actually resolve that dependency within globally-declared @bar (@bar//some:target).

This attribute is not supported in MODULE.bazel context (when invoking a repository rule inside a module extension’s implementation function).

optional
add_libdir_to_library_search_path – (bool) (default False)
If true, add the lib dir of the bundled interpreter to the library search path via LDFLAGS.

Added in version 1.3.0.

optional
annotations – (dict[str, str]) (default {})
Optional annotations to apply to packages. Keys should be package names, with capitalization matching the input requirements file, and values should be generated using the package_name macro. For example usage, see this WORKSPACE file.

optional
download_only – (bool) (default False)
Whether to use “pip download” instead of “pip wheel”. Disables building wheels from source, but allows use of –platform, –python-version, –implementation, and –abi in –extra_pip_args to download wheels for a different platform from the host platform.

optional
enable_implicit_namespace_pkgs – (bool) (default False)
If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary and py_test targets must specify either legacy_create_init=False or the global Bazel option --incompatible_default_to_explicit_init_py to prevent __init__.py being automatically generated in every directory.

This option is required to support some packages which cannot handle the conversion to pkg-util style.

optional
environment – (dict[str, str]) (default {})
Environment variables to set in the pip subprocess. Can be used to set common variables such as http_proxy, https_proxy and no_proxy Note that pip is run with “–isolated” on the CLI so PIP_<VAR>_<NAME> style env vars are ignored, but env vars that control requests and urllib3 can be passed. If you need PIP_<VAR>_<NAME>, take a look at extra_pip_args and envsubst.

optional
envsubst – (list[str]) (default [])
A list of environment variables to substitute (e.g. ["PIP_INDEX_URL", "PIP_RETRIES"]). The corresponding variables are expanded in extra_pip_args using the syntax $VARNAME or ${VARNAME} (expanding to empty string if unset) or ${VARNAME:-default} (expanding to default if the variable is unset or empty in the environment). Note: On Bazel 6 and Bazel 7.0 changes to the variables named here do not cause packages to be re-fetched. Don’t fetch different things based on the value of these variables.

optional

experimental_requirement_cycles – (dict[str, list[str]]) (default {})

A mapping of dependency cycle names to a list of requirements which form that cycle.

Requirements which form cycles will be installed together and taken as dependencies together in order to ensure that the cycle is always satisified.

Example: sphinx depends on sphinxcontrib-serializinghtml When listing both as requirements, ala

py_binary(
  name = "doctool",
  ...
  deps = [
    "@pypi//sphinx:pkg",
    "@pypi//sphinxcontrib_serializinghtml",
   ]
)

Will produce a Bazel error such as

ERROR: .../external/pypi_sphinxcontrib_serializinghtml/BUILD.bazel:44:6: in alias rule @pypi_sphinxcontrib_serializinghtml//:pkg: cycle in dependency graph:
    //:doctool (...)
    @pypi//sphinxcontrib_serializinghtml:pkg (...)
.-> @pypi_sphinxcontrib_serializinghtml//:pkg (...)
|   @pypi_sphinxcontrib_serializinghtml//:_pkg (...)
|   @pypi_sphinx//:pkg (...)
|   @pypi_sphinx//:_pkg (...)
`-- @pypi_sphinxcontrib_serializinghtml//:pkg (...)

Which we can resolve by configuring these two requirements to be installed together as a cycle

pip_parse(
  ...
  experimental_requirement_cycles = {
    "sphinx": [
      "sphinx",
      "sphinxcontrib-serializinghtml",
    ]
  },
)

Warning: If a dependency participates in multiple cycles, all of those cycles must be collapsed down to one. For instance a <-> b and a <-> c cannot be listed as two separate cycles.

optional

experimental_target_platforms – (list[str]) (default [])
NOTE: This will be removed in the next major version, so please consider migrating to bzlmod and rely on pip.parse.requirements_by_platform for this feature.

A list of platforms that we will generate the conditional dependency graph for cross platform wheels by parsing the wheel metadata. This will generate the correct dependencies for packages like sphinx or pylint, which include colorama when installed and used on Windows platforms.

An empty list means falling back to the legacy behaviour where the host platform is the target platform.

WARNING: It may not work as expected in cases where the python interpreter implementation that is being used at runtime is different between different platforms. This has been tested for CPython only.

For specific target platforms use values of the form <os>_<arch> where <os> is one of linux, osx, windows and arch is one of x86_64, x86_32, aarch64, s390x and ppc64le.

You can also target a specific Python version by using cp3<minor_version>_<os>_<arch>. If multiple python versions are specified as target platforms, then select statements of the lib and whl targets will include usage of version aware toolchain config settings like @rules_python//python/config_settings:is_python_3.y.

Special values: host (for generating deps for the host platform only) and <prefix>_* values. For example, cp39_*, linux_*, cp39_linux_*.

NOTE: this is not for cross-compiling Python wheels but rather for parsing the whl METADATA correctly.

optional
extra_hub_aliases – (dict[str, list[str]]) (default {})
Extra aliases to make for specific wheels in the hub repo. This is useful when paired with the whl_modifications.

Added in version 0.38.0: For pip.parse with bzlmod

Added in version 1.0.0: For pip_parse with workspace.

optional
extra_pip_args – (list[str]) (default [])
Extra arguments to pass on to pip. Must not contain spaces.

Supports environment variables using the syntax $VARNAME or ${VARNAME} (expanding to empty string if unset) or ${VARNAME:-default} (expanding to default if the variable is unset or empty in the environment), if "VARNAME" is listed in the envsubst attribute. See also envsubst.

optional
isolated – (bool) (default True)
Whether or not to pass the –isolated flag to the underlying pip command. Alternatively, the RULES_PYTHON_PIP_ISOLATED environment variable can be used to control this flag.

optional
pip_data_exclude – (list[str]) (default [])
Additional data exclusion parameters to add to the pip packages BUILD file.

optional
python_interpreter – (str) (default “”)
The python interpreter to use. This can either be an absolute path or the name of a binary found on the host’s PATH environment variable. If no value is set python3 is defaulted for Unix systems and python.exe for Windows.

optional
python_interpreter_target – (label) (default None)
If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. An example value: “@python3_x86_64-unknown-linux-gnu//:python”.

optional
quiet – (bool) (default True)
If True, suppress printing stdout and stderr output to the terminal.

If you would like to get more diagnostic output, set RULES_PYTHON_REPO_DEBUG=1 or RULES_PYTHON_REPO_DEBUG_VERBOSITY=INFO|DEBUG|TRACE

optional
requirements_by_platform – (dict[label, str]) (default {})
The requirements files and the comma delimited list of target platforms as values.

The keys are the requirement files and the values are comma-separated platform identifiers. For now we only support <os>_<cpu> values that are present in @platforms//os and @platforms//cpu packages respectively.

optional
requirements_darwin – (label) (default None)
Override the requirements_lock attribute when the host platform is Mac OS

optional
requirements_linux – (label) (default None)
Override the requirements_lock attribute when the host platform is Linux

optional
requirements_lock – (label) (default None)
A fully resolved ‘requirements.txt’ pip requirement file containing the transitive set of your dependencies. If this file is passed instead of ‘requirements’ no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by ‘build/run/test’. Note that if your lockfile is platform-dependent, you can use the requirements_[platform] attributes.

Note, that in general requirements files are compiled for a specific platform, but sometimes they can work for multiple platforms. rules_python right now supports requirements files that are created for a particular platform without platform markers.

optional
requirements_windows – (label) (default None)
Override the requirements_lock attribute when the host platform is Windows

optional
timeout – (int) (default 600)
Timeout (in seconds) on the rule’s execution duration.

optional
use_hub_alias_dependencies – (bool) (default False)
Controls if the hub alias dependencies are used. If set to true, then the group_library will be included in the hub repo.

True will become default in a subsequent release.

optional

Envvars:

RULES_PYTHON_PIP_ISOLATED, RULES_PYTHON_REPO_DEBUG

pip_utils.normalize_name(name)

normalize a PyPI package name and return a valid bazel label.

Args:

name – str, the PyPI package name.

Returns:

a normalized name as a string.

rule whl_filegroup(name, whl, pattern='', runfiles=False)

Extract files matching a regular expression from a wheel file.

An empty pattern will match all files.

Example usage:

load("@rules_cc//cc:cc_library.bzl", "cc_library")
load("@rules_python//python:pip.bzl", "whl_filegroup")

whl_filegroup(
    name = "numpy_includes",
    pattern = "numpy/core/include/numpy",
    whl = "@pypi//numpy:whl",
)

cc_library(
    name = "numpy_headers",
    hdrs = [":numpy_includes"],
    includes = ["numpy_includes/numpy/core/include"],
    deps = ["@rules_python//python/cc:current_py_cc_headers"],
)