I'm a big fan of the Poetry dependency management platform for Python, mainly as I've found it almost always to be a dream to work with. As many of its competitors, it wraps pip, venv and other tools with a unified front-end, and gets rid of the need to roll custom setup.py files (ew, right?). Often when we're working with a really simple application, just a requirements.txt file will do fine, and nothing more complex is required. When we need to do complex stuff is where these tools really shine.
I recently found that I was unable to install appimage-builder due to it having a dependency on LIEF, which in the ppublished version was (and still is to the date I'm publishing this), segfaulting when parsing aarch64 binaries. After testing the nightly LIEF build, I found that it no longer segfaulted, so I needed to figure out how to use that release instead of the version available on PyPI.
The source repository for the nightly version is stated as https://lief.s3-website.fr-par.scw.cloud/latest/lief. The
first thing to do here is to add that into our pyproject.toml as a custom repository. We can do this manually, or by
using the poetry CLI as follows:
poetry source add --priority=explicit lief-nightly https://lief.s3-website.fr-par.scw.cloud/latest/lief
This will create the following stanza in our pyproject.toml:
[[tool.poetry.source]]
name = "lief-nightly"
url = "https://lief.s3-website.fr-par.scw.cloud/latest"
priority = "explicit"
Poetry repositories can be primary,
supplemental or
explicit. A repository created without a
priority is considered to be primary, otherwise we need to specify it in the priority TOML key. primary can also
manually be specified as the priority if preferred.
Primary repositories are meant to be the default places where pip will look for dependencies. Because we have no control over which packages get published to this repository, this is clearly not the option we're looking for, as this could cause dependencies otherwise available in PyPI to be overridden.
Supplemental repositories are only searched if no package was found in the primary repositories. This is better, but again opens a potential risk of installing packages from places we may not be expecting. So I went with the third option:
explicit priority means that we define within the dependency's specification which repository it must come from. That
way, we can be even more certain that we are getting only the exact package(s) we want from the repository. This is by
far the most useful option for third-party repositories that we don't control ourselves.
Because we've chosen explicit priority, we need to specify the repository's name in the dependency. The simplest
option here is just to use the Poetry CLI, because it'll ensure that the lockfile is updated.
poetry add --source lief-nightly lief==0.16.0.dev0
This will create the following line in our pyproject.toml's main dependencies section':
lief = { version = "0.16.0.dev0", source = "lief-nightly" }
I always like to add a comment to overridden dependencies to explain why they're necessary, so I changed that to the following:
# We use LIEF nightly because the latest (0.15.1) segfaults on parsing aarch64 binaries.
# To check again when the next version is released.
lief = { version = "0.16.0.dev0", source = "lief-nightly" }
We could just use "*" as the version, and let Poetry do its thing every time we update, but because I want to regularly check the PyPI version to see if the issue has been fixed yet, I wanted it to notify me when the next dev version has been released.
That's all we need to do. You can now check your lockfile to ensure that the dependency chain has been updated correctly, and if you're using appimage-builder on aarch64, your builds will now work!
Until the next post!
Alex