Python Namespace Package and How to Use it?

In this tutorial, we will explore Python namespace packages and their significance in Python programming. Namespace packages are an advanced Python feature related to the __init__.py file commonly used in packages. If a package doesn't have an __init__.py file, it becomes a namespace package. We will delve deeper into this topic to understand its importance and functionality.

In general, there is little difference in how we use namespace packages versus regular packages in our project. We may still need to include an __init__.py file in a package without noticing any problems. Although namespace packages may be slightly slower to import compared to standard packages, it's unlikely that you'll encounter any significant issues.

Python namespace packages primarily benefit individuals involved in managing or designing interconnected packages. However, we can work on a project to make namespace packages more user-friendly for the general user.

It is an advanced topic, so one should be familiar with Python's basic concept and import system and also have some exposure to the packaging. Let's understand what a Python namespace package is and what it is for.

What is Python Namespace Package?

Before understanding the namespace package, let's recap the concept of namespace in Python.

"In Python, namespaces are containers with names defined in a particular context, such as a module, class, or function. They serve as a mechanism for organizing and distinguishing names in a program, preventing naming collisions and allowing for the reuse of names in different contexts. Namespaces can be accessed using the dot notation, where the namespace name is followed by the object's name within the namespace."

For example - When we import the requests, we can access the requests namespace and select from a host of different objects.

We can also refer to a Python dictionary as a namespace. We can take two variables that started out as completely distinct, independent variables with a Python dictionary and include them within the same dictionary namespace. Let's see the following example.

Example -

Explanation -

The above code snippet refers to the home_page and import_tutorial values from the sample_python namespace. Namespace packages work similarly, except that they group entire packages rather than individual values or other Python objects.

Namespace packages enable two packages on PyPI that share the same namespace. It could be more practical to have more than one namespace package. The true benefit of namespace packages becomes evident when you have at least two packages.

Businesses commonly utilize namespace packages when managing extensive collections of packages they wish to maintain within a company namespace. For instance, Microsoft Azure packages can be accessed under the azure namespace once installed.

In the next section, we will create company namespace for an example.

How to Create Namespace Package?

Suppose you are working in the Apex Corporation, and your team wants a namespace package apex_corp consisting of all its libraries. So Irrespective of the package you use, you will import it from apex_corp provided the Apex Corporation developed it.

In the absence of namespace packages, you would have two alternatives:

• Develop a monorepo consisting of a single package named apex_corp with numerous modules for all the diverse libraries and utilities you require.
• Develop multiple packages, but each package must be prefixed with apex_corp. For instance, you might have a package named apex_corp_dateutil.

The problem with developing a monorepo is that it forces everyone to download all of the code, even if they only utilize a small fraction of it. It also complicates version control and other packaging workflows, particularly if different teams are responsible for sub-packages.

The problem with creating multiple packages with a shared prefix is that it can become verbose, disorganized, and unsightly. Moreover, the CEO of the Apex Corporation has expressed dissatisfaction with this solution and prefers the monorepo alternative to package prefixing. Additionally, using shared prefixes is merely a convention that does not establish a genuine common namespace.

This scenario presents an excellent opportunity to use namespace packages. With namespace packages, we can have numerous distinct packages with their unique packaging workflows, but they can all exist under the same snake_corp namespace.

Following is the illustration of how we can achieve the namespace package creation. We will create three packages -

• apex-corp-dateutil - It consists of some date-utilities specific to the Apex Corporation.
• apex-corp-magic-numbers - It consists of some number-generation utilities specific to the Apex Corporation.
• apex-service - An employee project that wants to use both the apex-corp-dateutil and apex-corp-magic-numbers packages.

In the next section, we will set up the packages and install them in our virtual environment.

Setting Up Some Namespace Packages

The namespace package of the Apex corporation by creating the following folder and file structure -

The file structure comprises of three packages, and both utility packages, namely apex-corp-dateutil and apex-corp-magic-numbers, begin by defining an implicit namespace package named apex_corp. It is considered an implicit namespace package due to the absence of an init.py file.

As for the import name, we'll import the apex-corp-dateutil package as apex_corp.dateutil and apex-corp-magic-numbers as apex_corp.magic.

To contain the project, we will employ a virtual environment. We will install the two utility packages within the virtual environment, hence the presence of a pyproject.toml file for each package. This emulates the scenario where the packages are accessible on a corporate package server.

Within the dateutil.py module, we only need to incorporate a business-critical function that will return the number of days until the foundation Day of Apex.

Example -

The related pyproject.toml file contains the necessary information that setuptools requires to install the package correctly.

Example -

The above file makes sure that pip knows how to treat this package.

The tool.setuptools.packages.find section primarily governs package discovery, and namespace packages are usually enabled by default. However, we can be more specific regarding the inclusion of your namespace package by specifying options such as where, include, and namespaces.

Within the magic.py module, we can also include a number generator that's needed for certain Apex Corporation bureaucracy. Note that this number generator is purely a placeholder in this example.

The function in the magic.py module returns a random number, which doesn't have any practical use in this example. However, in a real-world scenario, this function could generate special tokens that may be required for certain tasks, such as accessing a corporate API service or generating a unique identifier.

The corresponding pyproject.toml file is almost identical to the previous one, with modifications to the package name, version, and description.

Now, we are ready to install the pip in a virtual environment. In the upcoming section, we'll learn the process of developing a new package while utilizing the namespace packages we've just created.

Installing and Using Namespace Package

Now, navigate to the outer apex-corp folder, where we can see the folders of the two utility packages. We will create the virtual environment there, and install the packages.

In this example, the -e flag has been included during package installation which allows for an editable installation. This feature grants the ability to make changes to most of the source code of the installed package without requiring a reinstallation. However, it is important to note that this flag is usually only necessary if you are developing the package and actively changing the source code.

We have correctly installed the required packages and activated the virtual environment, running this module should result in two numbers being printed on the console without any exceptions. The numbers printed may vary from the ones shown in this module as they are dependent on the input data used by the module.

Initially, implicit namespace packages may seem unfamiliar. Having a designated file to indicate that a package is meant to be treated as a namespace package would be more straightforward. It is comparatively easier to create a namespace package than a regular one. These concerns are valid and will be addressed in the next section.

Why Do Namespace Packages Exist?

Python namespace packages can be useful in two scenarios:

• When we have a large package, not all users require access to every component it offers. In such a case, namespace packages allow us to maintain a common namespace while enabling selective import of only the required components.
• When we have multiple distinct yet interrelated packages, and we want them to share the same namespace. Here, namespace packages enable the grouping of these packages under a single namespace, facilitating ease of use and organization.

Ideally, we would want to install packages in various locations and still have them function as cohesive packages in our code. Additionally, we prefer that the packages are relatively autonomous, so having numerous packages contributing to a namespace doesn't require installing all of them to begin utilizing the namespace.

The traditional way to establish a common namespace in Python involved using the pkgutil module. Although it is now considered obsolete, it is still possible to use this method in modern Python. Utilizing pkgutil to create a namespace package involves adding a code snippet to the __init__.py files of all the namespace packages, as follows -

Example -

Implicit namespace packages have received some criticism. It is currently easier to create a namespace package than a regular one. It can confuse beginners who believe that namespace packages are the default option and may feel the need to opt-out by creating an __init__.py file for their package.

PEP 420 outlines a scenario where the Python standard library could benefit from namespace packages. Specifically, the encodings module in the standard library has the potential to be converted into a namespace package. Doing so would enable various operating systems, which may require obscure encodings, to add their encodings to the encodings namespace effortlessly.

Example of Namespace Packages

Although not very common to encounter, namespace packages do exist in the public domain. Here are a few examples of public projects that employ namespace packages:

• OpenTelemetry
• discord.py
• Azure
• Google

Initially, we'll explore how installing a namespace package operates after uploading it to PyPI. Subsequently, we'll examine its source code to understand its inner workings. Lastly, we'll delve into how namespace packages can be extended to establish a prototype plugin system.

Conclusion

In this tutorial, we discussed namespace packages' functionality and when and how to utilize them. We have also explored how to create a namespace package for an organization. Even if you have yet to gain experience with namespace packages, this tutorial has given you an understanding of what they are and when they can be useful. Therefore, the next time you include a blank init.py file, you will better understand its purpose.