💾 Archived View for tilde.pink › ~kaction › log › 2022-11-25.1.gmi captured on 2023-04-26 at 13:52:22. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Woes of python module system

We all love that for every ugly problem in the world there is Python module that mostly solves it, but lets talk about details of module system. What exactly happens when Python executes "import foo.bar.baz"? As you can guess, answer is much more complicated than "find foo/bar/baz.py under some directory in sys.path and load it, if it wasn't loaded already".

Execution of "import foo.bar.baz" involves finding and loading of at least following files in sys.path:

foo/__init__.py
foo/bar/__init__.py
foo/bar/baz.py OR foo/bar/baz/__init__.py

If there is also "foo.bar.banana" in different directory in sys.path, life becomes even more complicated. And I am not even mentioning .pth files.

https://docs.python.org/3/tutorial/modules.html

https://docs.python.org/3/reference/import.html#path__

I think this complicated process was introduced for "from foo.bar import *" kind of imports, but it ended up as blatant violation of "explicit is better than implicit" principle. Also, loading of extra files makes is harder to split modules while preserving backward compatibility.

Say you had module "foo" that contained two functions like following:

# foo.py
def frobnicate(x):
    pass

class Banana:
    pass

and now you want to introduce "foo.frob" and "foo.banana" that would export "frobnicate" and "Banana" independently, yet keep "import foo" to provide both. You can't just put two imports into "foo/__init__.py"

# foo/__init__.py -- does NOT work

from foo.frob import frobnicate
from foo.banana import Banana

since "import foo.banana" would load "foo/__init__.py", defeating the whole purpose of splitting. There is workaround, but it is not pretty. I will write about it in other post in some time in future.

Unfortunately, it is couple decades as too late to fix Python module system, so we can only avoid exacerbating the problem. Please keep your "__init__.py" files empty.