Koan 19: The Unhelpful Eclipse
On the fragility of unintended behavior and the wisdom of explicit paths
When crafting software, we often mistake a side effect for a feature. We assume that because a behavior works once, it will always work in all situations. In the transition from Python 3.11 to 3.12 the importlib.resources underwent such a refinement. A shortcut that once worked was revealed to be an illusion.
Let us peer at the sky slowly.
Part 1: The Nature of the Traversable
In Koan 14 we explored how non-python files can be included within a python package. importlib.resources is a Python standard library which lets you bundle up configuration files, database schemas, binaries and any other non-source files you may need to run your code.
When you run importlib.resources.files(“my_package.test_data”), you might expect to get back a Path, but instead you get a Traversible object.
A Traversable is an abstract class which provides “a subset of pathlib.Path methods suitable for traversing directories and opening files.”.
importlib.resources.files returns a Traversable instead of a regular pathlib.Path because package data files can exist in various formats on disk:
Within a tar file (in the case of wheels)
In a plain directory (in the case of editable installs)
Spread across multiple packages (in the case of namespace packages, which we learnt about in Koan 7).
It is a promise that something behaves like a file system path even if it is trapped inside a zip file or a remote package.
Part 2: The Differences Emerge
In our offending example, importlib.resources.files returns a MultiplexedPath. And this usually gives us most of the features we need to work with a path:
But sometimes we have code trapped in a library or an upstream function which expects a Path and uses methods unsupported by a Traversable:
And that might tempt us into using a shortcut to convert the MultiplexedPath1 into a Path.
Part 3: The Accidental Shortcut
In Python <=3.11, MultiplexedPath.joinpath("") would return the first physical path it found. It felt like a convenient way to “extract” a real path.
By passing an empty string to joinpath you were asking the system to join “nothing” to the collection. The system responded by handing back the first item it held as a PosixPath. However, this turned out to be a quirk of the implementation, rather than a contract.
In Python 3.12 the implementation of joinpath was made more generic and robust. It began to strictly follow the logic of its ancestors. When you ask to join a path to a MultiplexedPath it now iterates through its internal requirements.
When you pass an empty string or a dot the iterator now reaches the end of its instructions and finds nothing to give. This tripped up a lot of developers, and users of libraries.
Part 4: Facing the Eclipse
If your intent is to turn a Traversable object into a real physical file path, you must use a tool designed for that purpose. We should not use joinpath to change the nature of an object. Instead, use as_file to convert a MultiplexedPath to a PosixPath safely:
The Emperor lost his kingdom because he had mistaken a pattern for a law. He claimed power over the night because he had never seen the night without the light. In the same way, a programmer who relies on the side-effects of a method mistakes a consistent coincidence for a formal promise.
The full list of supported methods can be found here - https://importlib-resources.readthedocs.io/en/latest/api.html#importlib_resources.readers.MultiplexedPath