Management dependencies in python

Putting ideas into words is certainly no guarantee that they'll be right. Far from it. But though it's not a sufficient condition, it is a necessary one. -Paul Graham

If you are a developer in python, what would be the answer to the next question: How to manage dependencies in python?. Probably the answer include pip, virtualenv and requirements.txt. These tools are the stack that python developers have been used for a long of time in different projects. However, due to the concept 'modern python', new interesting features can be used as asyn/await, f string, pattern matching, type hints, walrus operator etc. Having said that, modern python come with a new specification to management dependencies called pyproject.toml

What is pyproject.toml?

The pyproject.toml is a file written in human-readable syntax, it is located in the root directory of the project and contains all the metadata, dependencies libraries and configuration of additional tools like: black, mypy and pytest. The next configuration is a basic example.

[project]
name = "spam"
version = "2020.0.0"
description = "Lovely Spam! Wonderful Spam!"
readme = "README.rst"
requires-python = ">=3.8"
license = {file = "LICENSE.txt"}
keywords = ["egg", "bacon", "sausage", "tomatoes", "Lobster Thermidor"]
authors = [
  {email = "hi@pradyunsg.me"},
  {name = "Tzu-Ping Chung"}
]
maintainers = [
  {name = "Brett Cannon", email = "brett@python.org"}
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python"
]

dependencies = [
  "httpx",
  "gidgethub[httpx]>4.0.0",
  "django>2.1; os_name != 'nt'",
  "django>2.0; os_name == 'nt'"
]

[project.optional-dependencies]
test = [
  "pytest < 5.0.0",
  "pytest-cov[all]"
]

[project.urls]
homepage = "https://example.com"
documentation = "https://readthedocs.org"
repository = "https://github.com"
changelog = "https://github.com/me/spam/blob/master/CHANGELOG.md"

[project.scripts]
spam-cli = "spam:main_cli"

[project.gui-scripts]
spam-gui = "spam:main_gui"

[project.entry-points."spam.magical"]
tomatoes = "spam:main_tomatoes"

Example took from PEP 621

Tools

Different tools use the pyproject.toml specification to automate the management dependencies process. There are a lot of tools shown in Figure 1, but the most popular are:

Poetry

Poetry is the most popular library for packing and managing dependencies. In additional, it has a CLI which developers can use easily interact with the pyproject.toml file, the next code is an example of pyproject.toml

[tool.poetry]
name = "poetry-intro"
version = "0.1.0"
description = ""
authors = ["jairo <me@jairoandres.com>"]

[tool.poetry.dependencies]
python = "^3.7"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Poetry includes an incredible CLI to deal with versioning problems which is common in projects that have different environments (DEV, PROD and QA). Developers who use poetry can install dependencies by environment and create group of dependencies, for example define dependencies only for documentation.

Figure 2: CLI poetry. Took from poetry docs.

poetry has been growing in popularity, that is the reason why famous libraries like pendulum uses poetry

Hatch

Hatch was recently added to PYPA repositories, that is because hatch is probably the tool that implement PEP standards in the most meticulous way, the next code is an example from pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "test"
description = ''
readme = "README.md"
requires-python = ">=3.7"
license = "MIT"
keywords = []
authors = [
  { name = "Jairo Castaneda", email = "me@jairoandres.com" },
]
classifiers = [
  "Development Status :: 4 - Beta",
  "Programming Language :: Python",
  "Programming Language :: Python :: 3.7",
  "Programming Language :: Python :: 3.8",
  "Programming Language :: Python :: 3.9",
  "Programming Language :: Python :: 3.10",
  "Programming Language :: Python :: 3.11",
  "Programming Language :: Python :: Implementation :: CPython",
  "Programming Language :: Python :: Implementation :: PyPy",
]
dependencies = [
]
dynamic = ["version"]

[project.urls]
Documentation = "https://github.com/unknown/test#readme"
Issues = "https://github.com/unknown/test/issues"
Source = "https://github.com/unknown/test"

[tool.hatch.version]
path = "test/__about__.py"

[tool.hatch.envs.default]
dependencies = [
  "pytest",
  "pytest-cov",
]
[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=test --cov=tests"
no-cov = "cov --no-cov"

[[tool.hatch.envs.test.matrix]]
python = ["37", "38", "39", "310", "311"]

[tool.coverage.run]
branch = true
parallel = true
omit = [
  "test/__about__.py",
]

[tool.coverage.report]
exclude_lines = [
  "no cov",
  "if __name__ == .__main__.:",
  "if TYPE_CHECKING:",
]

In comparison with other tools, The CLI hatch is faster. This has a huge amount of configurations references to personalized pyproject.toml.

Another good feature of Hatch, this allows execution scripts which is similar to NMP package.json scripts

[tool.hatch.envs.default.scripts]
cov = "pytest --cov-report=term-missing --cov-config=pyproject.toml --cov=test --cov=tests"
no-cov = "cov --no-cov"

The last example allows executing scripts of test coverage and defining them by environment. Managing dependencies in file pyproject.toml is less automated, but that is great at the same time because it avoids adding unnecessary data. Recently the popular framework FastAPI has changed changed flit to hatch

PDM

PDM, as described, is a modern Python package and dependency manager supporting the latest PEP standards. One thing to remark is that PDM doesn't use virtualenv, That is because of PEP 582 the next code is an example from pyproject.toml

[project]
name = "pdm-project"
version = "0.1.0"
description = ""
authors = [
    {name = "Jairo Castaneda", email = "me@jairo.com"},
]
dependencies = [
    "fastapi>=0.85.0",
]
requires-python = ">=3.10"
readme = "README.md"
license = {text = "MIT"}
[project.optional-dependencies]

[build-system]
requires = ["pdm-pep517>=1.0.0"]
build-backend = "pdm.pep517.api"

[tool]
[tool.pdm]
[tool.pdm.dev-dependencies]
dev = [
    "pytest>=7.1.3",
]

PDM has an incredible console

**Figure 4:** Folder pypackages which contain libraries installed by PDM

Other tools

Flit is other popular tool, but this tool is only for package management. In general, it is used to create and publish packages but need other tool to complement its management dependencies.

Bento is other tool, like Flit, bento is used for publish and packaging. Bento has a special syntax in a file called beto.info

What tool I should be used?

I have forgotten how many times I read the tittle “modern python package and dependency manager”. Python has a lot of tools for resolver problem with packaging management. But none of these tools are compatible which each other, for instance: the format in lock file (which is very important because of hashes, pins, and tree of dependencies) is totally different in each tool.

In my opinion, pyproject.toml is the future to management dependencies in python, i.e., NodeJS uses a file as an entry point to managing dependencies. Choosing the better tool should be a task which takes time to analyze what requirements the project has, and you should have three points in mind.

Review repository, starts, issues bugs compatibility with your CI/CD tools even some PaaS services don't support pyproject.toml
Review standards, what standard follows? What is the status of the PEP?, is it finalized? Is it draft?
Review who is behind the project?

Mentions

Thanks to Jeisson Rangel for reading drafts and give me feedback of this