Resolving Circular Imports in Python
PYTHON PROGRAMMING
Circular imports occur quite often during the development of Python applications and packages. The error arises when two modules attempt to Import each other simultaneously – this creates a loop where neither module can fully load. As a result, this leads to an ImportError that states, more or less, that you can't import an object from a partially initialized module.
Oftentimes, resolving this error is straightforward. Sometimes, however, the code needs to be redesigned and refactored – occasionally quite extensively. This article aims to support you in this by explaining the methods for resolving the error.
Solutions
We'll explore methods to resolve circular imports without refactoring the code to change its behavior. Sometimes, it'll suffice to simply redefine the responsibilities of your classes so they don't depend directly on each other. However, this approach often won't be enough. Therefore, we'll focus on strategies for addressing circular imports without changing the behavior of the objects defined in the Modules causing them.
Join the modules
Sometimes, the best solution to circular imports is to refactor the code by combining the two modules that cause the problem. This is the most basic, the simplest, and often the best solution. Do the two modules import each other? This clearly indicates that they are closely related, so perhaps you could consolidate them into a single module.
This can be the best solution not only because it resolves the circular imports error, but also because it's often good to consolidate related functionalities within a single module. This approach ensures that circular imports won't occur.
Let's consider the following overly simplistic example:
Python"># module1.py
from module2 import XUser
class YUser:
def use_x(self):
return XUser()
# module2.py
from module1 import YUser
class XUser:
def use_y(self):
return User()Open the Python interpreter and import module1. You'll see this:
>>> import module1
Traceback (most recent call last):
...
ImportError: cannot import name 'YUser'
from partially initialized module 'module1'
(most likely due to a circular import) (...)You'll get a similar error when attempting to import module2.
In this case, joining the two modules is simple:
# module1_2.py
class YUser:
def use_x(self):
return XUser()
class XUser:
def use_y(self):
return YUser()Such an approach will often do the job. However, in many situations it won't, as you may want or need to keep the two functionalities in different modules, for whatever reasons. If this is the case, solving circular imports can be tricky. Below you'll find possible solutions, but remember to choose one that works best for you in a particular situation. Here are several strategies you can consider to resolve or avoid circular imports.
Deferred and semi-deferred imports
When I cannot combine the problematic modules (as described above), deferred imports are my preferred solution. This approach is straightforward and simple; it involves moving the problematic import statements within the modules.
Let's see how deferred imports work in our example. Essentially, you need to move the problematic import statement into a function or method that uses the module. This way, the import occurs at runtime when the function is called, rather than when the module is first loaded.
# module1.py
class YUser:
def use_y(self):
from module2 import XUser
return XUser()
# module2.py
class XUser:
def use_x(self):
from module1 import YUser
return YUser()You will now be able to import both modules.
Note that this solution means we're attempting to import module1 every time we call the module2.XUser.use_x() method, and module2 every time we call the module1.YUser.use_y() method. Does this mean that deferred imports affect performance?
They can, but usually only slightly. When Python encounters an import statement, it doesn't immediately import it over and over again. Instead, it uses a module-caching system, which prevents repeated imports (i.e., re-execution) of module code after it has already been imported. Once a module is imported, it doesn't need to be re-imported. Therefore, even if the import line is repeated or a module is imported within a loop, the module isn't re-imported. If, for any reason, you need to reload the module – that is, re-import it (e.g., because its code haschanged during execution) – you can use the importlib.reload() function from the importlib module
What does this imply in terms of performance? Deferred imports can minimally affect performance – not due to the reloading of the module containing deferred import(s), but due to the overhead of a repeated import statement. In the context of an individual case, this overhead is very small because it involves a dictionary lookup in sys.modules, which is a fast operation. Hence, this overhead is typically negligible unless in highly performance-sensitive contexts, such as when the import statement is called repeatedly, for example, inside a tight loop.
In such a situation, you can try to improve performance by using semi-deferred imports. This means using a deferred import in one module only – the one which is less frequently imported. In our example, we don't know which of the modules would be imported more frequently. Let's use a deferred import for module1:
# module1.py
class YUser:
def use_y(self):
from module2 import XUser
return XUser()
# module2.py
from module1 import YUser
class XUser:
def use_x(self):
return YUser()This version will work just fine, and so would a version with deferred import in module2. In this code, we have no loops in which the import would be performed, so in practice, performance won't benefit too much from using semi-deferred imports. However, if we called the module2.XUser.use_x() method many times, for instance in a loop, the effect on performance would likely be visible.
Deferred imports have a serious drawback, however: they go against the PEP 8 style guide, according to which import statements should be placed at the top of a Python file:
Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants.
The use of "always" **** implies no flexibility for developers. Thus, if you choose to use deferred imports, be aware that this approach breaks the conventional rule for Python imports, which could potentially confuse other developers who are accustomed to seeing imports at the beginning of files. Most likely, linters will deduct points for placing imports like this.
Import the modules at the end of the module
This is a far less common technique. It's very simple, though: place the import which leads to circular imports at the end of the module. This can sometimes resolve circular imports because by the time the import is executed, all other module-level code has already been processed.
# module1.py
class YUser:
def use_y(self):
return XUser()
from module2 import XUser
# In module2
class XUser:
def use_x(self):
return YUser()
from module1 import YUserThis method ensures that all module-level definitions (functions, classes, variables) are processed before any import statements are executed. This can prevent issues arising from trying to use incompletely initialized modules.
It's a pretty straightforward method, but to many, it won't look natural. It isn't free of disadvantages, too. First of all, it works only when the execution of the import statements doesn't interfere with any module-level code that needs to run first.
As was the case with deferred imports, placing imports at a module's end is against the official Python style guide. It can also be confusing to other developers, affecting code readability and maintainability.
Dynamic Imports
This is the most complex and powerful method among those mentioned in this article, and the most general one. It's used in complex scenarios, especially in large applications, consisting of modules of complex dependencies.
Dynamic imports are performed using the importlib package (e.g., the importlib.import_module() function) and allow for importing modules based on runtime conditions and user input. They are particularly useful in such scenarios as plugin systems or applications where modules are loaded on-demand, based on configuration or user choices. Such complex situations might not be addressed by simply deferring imports. Dynamic imports provide the flexibility to load completely unknown modules at runtime, which is not the case with deferred or semi-deferred imports.
Due to their complexity and a wide array of applications, dynamic imports deserve a dedicated article. Thus, we won't discuss them in detail here. It's enough for you to remember about them when you're struggling with circular imports. Who knows, maybe at the end of the day dynamic imports will occur to be the only working solution to circular imports? Don't forget however, that in most typical situations in which you need to resolve a circular imports problem, dynamic importing would be an overkill.
Conclusion
Circular imports in Python can pose significant challenges, particularly in large, complex applications. As we've discussed, you can choose one of several methods to resolve these issues, each with its pros and cons:
- deferred or semi-deferred imports
- placing imports at the end of a module
- employing dynamic imports
The key is to choose the method that best fits the specific needs and context of your code and application.
It's crucial to balance code readability, maintainability, and performance when selecting a method for a particular situation. Unfortunately, some methods go against the official Python style guide, PEP 8. Despite this, they can still be justified in certain contexts. However, always consider the long-term impact of your decision on the codebase. Whenever you decide to employ such methods, be sure to explain your choice in the documentation. Otherwise, new developers or maintainers may spend significant time trying to understand a choice that appears to break the rules. If you employ any unconventional techniques, always do so transparently.
In conclusion, effectively handling circular imports requires an understanding of Python's import mechanism and your code's architecture. To ensure the high quality of your project and code, avoid circular imports through careful and thoughtful design of the application and its code. Optimal code design should always be the preferred solution.
Sometimes, however, you may have to employ one of the methods we described. Theoretically, it's best to avoid using "tricks" such as deferred imports or placing an import statement at the bottom of the module. However, in some scenarios, these tricks may offer the simplest – sometimes even the best – the best solution. If that's the case, don't remain silent about what you did; explain the method and your reasons behind using it. This approach is fair to other developers, as well as to the maintainers and users of the code.
