By Jordan Mora, Software Engineer at BairesDev.
Have you ever gotten into a new Python project or revised an old one, and while you set up your environment you started getting those annoying “cannot install dependency X because package version has conflicting dependencies ” issues with the project’s dependencies? Then you checked the requirements file and found that the offending dependency doesn’t have a pinned version, or worse, it’s not even there. Well, you are not alone; it happened to me once or twice.
When this happens, the best solution is always to:
- Check the version with another coworker
- Add it to the requirements file
- Re-run the installation process
- Hope this doesn’t happen to the next dependency
- Repeat
You can see it’s very reactive, because it is not preventing it from happening again, and sooner or later, it will. This thought lingered in my head. Is there a solution to this common problem? Something like npm for front-end projects?
The old way
The usual way of installing dependencies in a Python project is using pip. With pip, you add a dependency with the pip install X and then write it with the version pinned down (e.g., requests==2.26.0) in the requirements.txt file, so the next time, a newcomer just needs to run pip install -r requirements.txt to install them.
The problem with this approach is that when dependencies also have sub-dependencies, you might not always get the same version, as they are not specified in the requirements.txt file. Another issue with this process is that you might forget to add the dependency to the requirements file. This is where Python Poetry comes in.
What makes Python Poetry appealing
Poetry makes use of a TOML file instead of a text file for tracking dependencies, and a lock file for tracking dependencies and sub-dependencies. Each time the project dependencies are installed, you will always get the same versions as in the lock file. Poetry goes the extra mile and also saves the hash of a package, as some maintainers might update the package without changing the version.
Poetry comes with a convenient CLI for installing dependencies, so pip is no longer needed. When installing the dependencies, they are automatically added to the TOML file, saving you the dreadful task of remembering the packages you add to the project.
Python Poetry enforces the use of virtual environments. If you are not already using one, it will create a new virtual environment to isolate dependencies between different projects. If you are using Docker, you can deactivate this feature, since there is no point in using virtual environments inside containers.
Unlike the old way, Poetry gives you the ability to update all your packages, and depending on the dependency specification you are using, it can be a patch or a minor update (thorough research is recommended for major updates).
Dependency specification refers to the version constraint you apply to your dependencies. You can use caret, tilde, or wildcard constraints, or even specify a range of acceptable versions. You can also use git, path, and URL, and source dependencies.
Another nice feature of Poetry is that when you remove a dependency you no longer need, it will also remove its sub-dependencies, which is not the case with pip. This will prevent having orphan dependencies in your environment.
Finally, Poetry gives you the ability to separate main dependencies, the ones you need to make your project work, from development dependencies, which are used mainly for debugging purposes. This results in a production environment free of unnecessary dependencies.
I must note that Poetry is also able to package and publish libraries, but I will just focus on managing dependencies.
Composing your first poem
There’s no better way to understand something than seeing for yourself, so we’ll follow a little tutorial on how to migrate an existing project to Python Poetry. I will assume you have already installed Python version 3.7+ and git. If you are feeling adventurous, you can use a project you own or are working with, but I will provide a mockup Django project for this tutorial. The version of Python Poetry at the time of this writing is 1.3.1.
First things first, you can install Python Poetry by running the command:
curl -sSL https://install.python-poetry.org | python3-
You can check Poetry has been successfully installed by running poetry –version. You can also run poetry list to display the list of commands available.
Next, go to this demo GitHub repository and clone it to your local machine; it’s just an empty Django project with a requirements.txt file.
Now that we have the scenario ready, let’s proceed with the migration.
Initializing Poetry
The first step is to initialize Python Poetry in your project, so go ahead and run poetry init inside the repository. An interactive prompt will pop up to set up the TOML file. You can use the default suggestions, but make sure your Python version matches the Compatible Python version field. When you get to the “Would you like to define your main dependencies interactively?” section, just send “no.” Then it will ask if you want to set your development dependencies interactively; again, write “no.” Finally, it will ask you to confirm your settings; send “yes.”
Migrating your dependencies
Since you already have a requirements.txt file, the easiest way to migrate your dependencies is by running the command:
cat requirements.txt | xargs poetry add
Poetry will add each dependency to the TOML file and try to resolve any conflicts, and then it will write the lock file. In case you are not using a virtual environment, the dependencies will be installed in a newly created virtual environment.
Now that poetry has all your dependencies in its list, you can check it by running poetry show.
Since the requirements.txt file is no longer needed, you can get rid of it.
Cleaning up sub-dependencies
Since you probably have sub-dependencies listed as main dependencies, we can get rid of those. When you want to update high-level dependencies, there won’t be any constraints.
Run the command:
poetry show --tree
And you will see which ones are sub-dependencies.
Making dependencies updatable
The final step is to make your dependencies updatable. Personally, I only use patch updates as they are usually fixes and security patches. I do minor and major updates only after reading the changelogs and being sure it is something we want to add to the project.
You can use the dependency specification (mentioned earlier in the article) that suits you best. You can open up the TOML file again and edit the version of each dependency by adding the version constraint (tilde, caret, wildcard, etc.) to the version number. In my case, I used tilde constraints.
Now, Django and djangorestframework dependencies can receive (patch) updates. The next time an update is released for any of these dependencies, you can update them with the command:
poetry update
This will attempt to update all dependencies. You can also specify the dependencies you want to update. In case updates are found, Poetry will resolve dependency conflicts and run the update. After that, it will update the lock file.
In this case, if you run the update, and, assuming you made the same changes as I did, you will see djangorestframework will update from version 3.10.0 to 3.10.3, which is the latest patch at the time of this writing.
You can check the final product here in the migrated branch of the repository.
Next steps
From now on, you can use Poetry for adding and removing dependencies to and from your project, just using the CLI. In case you make an update outside of your constraint, you will need to run the add command with the new version.
For newcomers to the project, getting the exact same package versions as the last person that updated the lock file is as easy as running poetry install.
Conclusion
Similar to npm, Python Poetry is a powerful tool for dependency management that solves many of the issues present in using requirements.txt file along with pip. It also packs more functionality and the CLI is very easy to use. It can be applied to new and existing Python projects, making it easy to add, remove, and update dependencies as the project evolves.
I hope this tool can be used in many projects from now on. It has proven to be one of the best tools I have ever used for helping me and my team maintain a project up-to-date without becoming a cumbersome task.
If you are interested in learning more about Python Poetry, check out the documentation by following this link.
If you enjoyed this article, check out one of our other Python articles.
