💾 Archived View for dece.space › dev › faq › packaging-python-tools.gmi captured on 2021-11-30 at 20:18:30. Gemini links have been rewritten to link to archived content
➡️ Next capture (2023-01-29)
-=-=-=-=-=-=-
Last update: 2021-10-20.
So you wrote a nice Python program and would like to share it, but setuptools documentation is giving you headaches? Let me try to help. It's normal to feel a bit lost because the Python packaging world has been a bit of a mess for a long time and is currently undergoing a lot of changes to modernise it and provide a more stable interface to developers.
This document assumes you are familiar with the Python language. It is for SIMPLE programs and libraries: for complex projects using multiple languages or other complicated setup process it will obviously not be sufficient. That's all!
We will review some concepts, see how you can package a program or library using the common setuptools package, and how modern packaging requires changing a few things.
Here are brief descriptions of the high-level elements of Python packaging. I do not go into details of what build back-ends are or what is the exact timeline that lead to this or that, it's just to clear things up a bit.
PyPI is a public repository where people upload Python programs and libraries. Anyone can post their own programs there for everyone else, and by default the pip tool installs packages from PyPI.
Note that PyPI is not strictly required for package utilities as it is possible to install a Python package from other sources (filesystem, Git repo, etc).
distutils is a Python package to build and install other packages. You rarely have to deal with it directly and mostly use setuptools, which is another, more user-friendly Python package to build, install and distribute other packages. You generally use it to create OR install a package, so you need it installed with your Python distribution. On Debian, it's the python3-setuptools package. Even though it is not part of the standard library, setuptools has been the de facto utility to produce and distribute Python packages for years now.
Let's review files commonly found in packaging discussions.
`setup.py` is a legacy way people used to provide to distribute packages. It's a script using either distutils, setuptools or another build backend and running a few commands to install your package in a hopefully standard way. Due to being a Python script, it's cumbersome to detect what are the build time dependencies, what back-end it uses, using which options so it is becoming less and less popular.
Its static brother `setup.cfg` is a config file describing the package declaratively so it's easier for build systems to process it. Despite the generic name, this file is meant to be used with setuptools.
Finally, `pyproject.toml` is a config file, made standard by PEP 518, defining how a project should be built: which build back-end to use, what are the dependencies to *build* a project (not even to run it, just to produce a working package), etc.
And last but not least, wheels (with the whl extension) are packages that can be shared and installed. When you download a package from PyPI, you usually receive the source and/or a wheel and install it. This is a generally easy and portable solution so that's probably what you want to produce.
OK all of that is great, but how do I package my Python program?! For most needs, you can still use setuptools with a simple setup.cfg script to achieve the following:
In this section, I will not cover the pyproject.toml file and assume you just want to use setuptools to get the job done, with a simple example. To be clear, this is not the “standard” or “clean” way to do it (there is not one simple answer) but rather a straightforward way to use setuptools without thinking too much about corner cases. In the next section we will cover what's the modern and proper way to specify your build system using pyproject.toml.
Let's say I wrote a fancy Gemini browser named Baboop and I'm eager to share it. The project is structured as follows:
├─ baboop/ │  ├─ bookmarks.py │  ├─ browser.py │  ├─ history.py │  ├─ __init__.py │  └─ utils │  ├─ config.py │  ├─ __init__.py │  └─ net.py └─ README.md
To package it with setuptools, we will need additional files:
In Python you do not have to specify an entry point to your program, but when you wish to control execution you often use this weird snippet:
def main():
# do stuff
if __name__ == "__main__":
main()
This is good practice because the `main` function will be executed only if the script is executed, but not if it's imported by another module, and we will use exactly this for our tool. Put your entry point code (the main function and the if block) in a __main__.py module, in the root package baboop, and you are done!
At the root of the project, let's create a simple setup.cfg file. It looks a bit like INI files, and in our case will contain three sections:
[metadata] name = baboop version = 0.0.1 description = Terminal browser for Gemini long_description = file: README.md long_description_content_type = text/markdown license = GPLv3 author = dece author-email = dece@example.com home-page = https://example.com/dece/baboop classifiers = Environment :: Console Programming Language :: Python :: 3 [options] packages = baboop, baboop.utils python_requires = >= 3.7 setup_requires = setuptools >= 38.3.0 install_requires = requests~=2.25 [options.entry_points] console_scripts = baboop = baboop.__main__:main
Let's go with the metadata section first. Nothing too special, `name` is your package name, `long_description` here uses the special format `file: ` to tell the build back-end that the long description of the project is in fact the readme file, and `classifiers` are optional tags used on PyPI.
The options section is more interesting:
And finally comes our entry points section where we specify through the `console_scripts` list the name of the executable we want to ship with our package: a script named “baboop”, that will run the `main` function of the `baboop.__main__` module. When installed, our package should provide the “baboop” script in the user's path.
To continue, ensure your environment have both setuptools and wheel installed, using `pip3 list`. As we're going the quick way, we know we want to use setuptools as build back-end and that a wheel is convenient both for us and for our users, whether we want to use PyPI or not.
We have two quick ways of invoking setuptools: create a dummy script or just use a self-sufficient Python command.
The dummy setup.py script that follows can be called with the command `python3 setup.py bdist_wheel` which will create a wheel, which is the simplest way to use setuptools as our build back-end:
from setuptools import setup setup()
To keep the repository clean and avoiding creating a scarily deprecated setup.py we can use this command instead, which does exactly the same thing:
$ python3 -c 'from setuptools import setup; setup()' bdist_wheel
If everything goes right, a bunch of new folders should have appeared, and your newly created wheel will be in the dist folder. That's it, you have your package!
You probably want to install it, maybe in a virtualenv, to check if the installation process works smoothly. Installing a wheel is as simple as this:
$ pip install --user dist/baboop-0.0.1-py3-none-any.whl # If using a virtualenv, drop the --user.
And your package is installed. Check that baboop (or the less silly name of your own tool) is now in your path, and you can now share it with friends and give them this command, or upload the wheel to PyPI.
The previous section covers a simple way to use setuptools but in the future developers want to move to an installation process description independent of the build back-end. This means tools such as pip will have a standard way to build your package when someone wants to install it, and not simply run some setup.py with fingers crossed, hoping that the target system has setuptools or any other build dependency installed.
In this section, we will reuse our previous example but port it to the modern workflow. Nothing to be afraid of!
It is important to note that setuptools itself is not deprecated or anything, just that having a setup.py in your source tree is not recommended. Indeed, if that script has `import setuptools`, there is no simple way for tools like pip to know that setuptools is required before running that script, so the idea is to express in a static file that building your project requires setuptools, and this is what the pyproject.toml file is for.
In our example, this can be specified like so:
[build-system] requires = ["setuptools", "wheel"] build-backend = "setuptools.build_meta"
That's it! This file is enough for PEP 517 compliant build front-ends to know how to build your package, which means that pip now knows how to install it without the need for the dummy setup.py (you can delete it if you created it) or the arcane Python command. The build front-end will see that both setuptools and wheel are required, and do what's needed to use them (ask the user to install them, do it automatically in a separate env, etc).
You can now build your project's wheel without explicitely calling setuptools, using a PEP 517 compliant build front-end. A simple one is PyPA's “build”, which you can find on PyPI, used like this:
$ python -m build -w
PyPA's “build” documentation
This is out of the scope of this document, but you can find good instructions in the relevant section of this link to setup your PyPI account and run upload commands against a test instance of PyPI before doing it on the real repository.
Packaging Python programs: Uploading the distribution archives
If for some reason you do not want to use PyPI to store your packages and reuse them, and there are a lot of valid reasons, you may wonder how to work out dependencies between your projects because asking users to manually install dependencies through “git+https” URLs can be a bit deterrent.
Imagine we have a library testlib_high that does fancy high level stuff using another library testlib_low. In testlib_high we want to express the requirements for testlib_low:
install_requires = testlib_low
Package and install… Woops!
Processing ./dist/testlib_high-0.0.1-py3-none-any.whl ERROR: Could not find a version that satisfies the requirement testlib-low (from testlib-high) ERROR: No matching distribution found for testlib-low
pip can't find your package because it is not on PyPI. What can we do? Until recently it was possible to define “dependency links” which are URLs to use when looking up dependencies, but it is now deprecated and will soon be removed. Fortunately with recent versions of pip we can directly put our “git+https” as the place to fetch a specific package:
install_requires = testlib_low @ git+ssh://git@example.com/dece/TestlibLow.git#egg=testlib_low
This should both work with git+https and git+ssh URLs. The egg fragment part of the URL is necessary, it's usually safe to use the same name as the package.
Note that this method requires you to express, in the testlib_low repository, your build system or at least have a dummy setup.py script (as in the previous section) because pip will need one of them to know how to process this dependency URL.
Building and installing the wheel should now automatically fetch and install your dependency from the repository! 🙌