Environment Tools: PDM, Poetry and Rye

We bring this series on environment tools to a close by looking at two older tools - PDM and Poetry - and one new entrant, Rye.

In case you haven't read any of the previous articles in the series, you can read them at the link below. These previous articles discuss the various problems faced while managing a python environment, along with some tools to solve the problems

series-environment-tools - Playful Python

Playful PythonSiddharta Govindaraj

While the tools that we saw before in the series solve one or two particular areas or pain points, the three tools that we are going go look at today do everything. Since we have covered most the common features in the previous articles, we'll just summarise what are offered by these three tools. Lets dive in! 🏊🏾

PDM

PDM stands for Package and Dependency Manager. The big selling point for PDM is that it allows us to create projects without having to create a virtualenv at all.

Introduction - PDM

logo

How does PDM provide isolated environments if there is no virtualenv?

Well, it installs everything that is required into a local folder called __pypackages__. If you are familiar with the javascript and node ecosystem, then this would sound familiar because the same approach is used there. This approach has the benefit that there is no virtualenv activation / deactivation that is required - just cd into the relevant project dir and everything works as it should.

In fact, this approach was proposed in PEP 582. However that PEP was rejected by the Python Steering Council in March this year. So, this method is not a standards compliant way to do it, but if you don't mind that, then PDM still supports this method and you can use it.

Another cool thing is that PDM can create a central cache of all the packages, so when multiple projects use the same packages, they can be referred to from the central cache instead of installing the same package in every project workspace. This is how it is done in the javascript and java ecosystems and can same quite a bit of disk space if many packages are common. But again, this isn't a standard way to do it.

If you do want to use a standard method - which is virtual environments - PDM does have support to create and manage venvs in a similar way to other tools.

So, you go to the project directory and do pdm init to create a template pyproject.toml file along with the virtual environment. One nice thing is that the venv is located in a .venv folder right within the project folder itself, so you don't need to go searching all over the computer to find it..

pdm add <dependency> will add the dependency to the project, as well as updating the pyproject.toml and the pdm.lock file. Yes, PDM supports lock files, and  pdm sync will sync the environment with the lock file. Further, the lock file contains hashes, so not only will the environment be reproducible, it will be secure as well.

Some other nice features are the ability to also install local packages - these could be a package in python wheel format that you have manually downloaded and want to install, or it can be a local directory as well. PDM also supports installing tarballs from a URL.

Like Hatch, PDM also has support for building packages and uploading the wheels to a repository.

Overall, the design of PDM is very much inspired from the Javascript ecosystem, and if you like how things work there, then PDM is the tool for you.

Poetry

Poetry was one of the oldest "all-in-one" tools in the python ecosystem. It does it all, from dependency management, lock files and secure builds to multiple environments, building wheels and uploading packages.

Poetry - Python dependency management and packaging made easy

Python dependency management and packaging made easy

Python dependency management and packaging made easy

You can pipx install poetry to get it. Poetry then provides the usual commands: poetry new <project name> to scaffold a new project, poetry add and poetry remove to add and remove dependencies from the pyproject.toml file.

When you initialise a new project, poetry will create a virtual environment automatically.

Like Hatch, Poetry supports the creation of multiple environments via the poetry env command. For example, for the same codebase you can create one environment for Python 3.7, another one for Python 3.11. You can then switch between environments and test that your app works with both versions of Python.

You don't need to manually activate and deactivate the virtual environment. Poetry provides poetry run ... to run a command in the context of the virtual environment.

Poetry supports lock files, these are stored in a file poetry.lock. The command poetry install will take the lock file and sync the virtual environment. Lock files contain hashes, so they are secure by default.

One drawback of Poetry is that it uses its own syntax within pyproject.toml to declare dependencies. It does not use the standard method. This is because the development of Poetry predates the standardisation of dependency and dependency group specification.

A neat thing about Poetry is that it supports plugins. Using plugins, it is possible to extend the behaviour of poetry in new ways.

For example, the poetry-multiproject-plugin allows us to use Poetry in monorepos (where multiple different projects are there in different subdirectories of a single source repository). Another plugin, poetry-dotenv-plugin will automatically read and set environment variables before running code during poetry run command.

Rye

Rye is a very new tool that just released a few weeks ago. It was built by Armin Ronacher, who is also the creator of the popular Flask web framework. We made a quick mention about it in last month's newsletter.

Introduction - Rye

An Experimental Package Management Solution for Python

logo

The big selling point of Rye is that it not only manages the virtual environment, like the other tools, but it manages Python versions as well. You can use rye to install and upgrade python versions that are installed on the system. In that sense, it includes the functionality that we previously saw in pyenv.

The thing with pyenv is that it has this slightly strange workflow, because you have to first download and install Python. Then you use that python version to install pyenv. Finally, you use pyenv to manage the versions of python on the system.

Rye by contrast does not require python on the system. This is because rye is written in Rust. So all you have to do is install the rye binary from the website (you can't pipx install it) and you are good to go. From there you can install and uninstall toolchains (rye's term for Python installations) with the rye toolchain fetch ... and rye toolchain remove ... commands. Rye not only supports CPython, but also PyPy and even local python builds that you have present on your system.

Once you have Python on your system, then it behaves in a similar manner to the other tools we have seen so far. You can use rye init to bootstrap a project template along with a pytroject.toml. After that rye sync will sync the project directory with the virtual environment (the venv will be created the first time you sync). Like PDM, rye keeps the virtualenv in a .venv folder within the project directory tree. Similar to other tools, rye also includes rye add to add dependencies and rye run ... to run scripts in the context of the virtual env. Rye supports locking of dependencies, using the file requirements.lock to store the locked versions so that all builds are reproducible.

And like the other tools on this page, rye also supports building a python wheel and uploading it to the package repository.

Summary

That brings us to a close after having looked at all the following tools in this series:

• pipx
• pyenv
• pip-tools
• Pipenv
• Hatch
• PDM
• Poetry
• Rye

Some of these tools try to solve one specific problem of managing the python development environment, while others are "all-in-one" tools that do everything that you might want. Hopefully this deep dive into all these tools has given you an insight into which tool is the right one for your project - or maybe you will just stick with good old pip and venv 😉