Magic Methods in Python
Magic methods are special methods that are called implicitly by the Python interpreter to execute corresponding operations. They help customize common operators or constructs. They make code more intuitive and Pythonic.
For example, to add two Python objects we might write
a + b. The actual execution would call the magic method
a.__add__() implicitly to perform this addition. If this magic method doesn't exist, the Python interpreter would call
b.__radd__(). Application code where addition is invoked is itself simple and readable.
There are several conceptual groups of magic methods such as object lifecycle, object representation, object comparison, object attributes, class creation, callability, containers, numerics, context management, and more. This article describes many of these and shares how to use them.
What are the characteristics of magic methods?
Several built-in classes define various magic methods. Output of
dir()can be filtered to see only dunder names but not all are magic methods. Further,
callable()can be used to introspect these names, but not all callables are magic methods. Ultimately, Python documentation clarifies which attributes are magic methods. For example, as illustrated in the figure,
__doc__isn't one and
__class__despite being callable isn't one.
The code above shows implicit calls to
__repr__()magic methods by
repr()respectively. These return suitable string representations of the
Point2dinstances. Magic methods thus serve these purposes:
- Enable classes to define their own behaviour corresponding to Python operators or constructs.
- Enable developers to write Pythonic code. A developer could have written
to_string()method but magic methods provide a uniform convention. This reduces technical debt.
- Enable code reuse and design patterns. For example, custom objects can be compared and sorted based on magic methods without extra effort.
Could you give an overview of the main magic methods?
- Object Lifecycle: These relate to creation, initialization and destruction of objects. Eg.
- String Representation: These represent objects as strings. Eg.
- Comparisons: To compare objects via operators
>=, etc. Eg.
- Math Operations: Arithmetic operators
*; bit operators
<<; and math functions. Eg.
- Type Conversions: Convert from one type to another. Eg.
- Type Check: Check class hierarchy. Eg.
- Context Manager: To set up and clean up objects within a context, such as open/close of files. Eg.
- Attribute Access: Control access or inspect instance attributes. Eg.
- Operations on Sequences: Find length, get/set/delete items or iterate through items of a sequence. Eg.
- Asynchronous: Pertaining to asynchronous operations. Eg.
- Miscellaneous: Not covered above. Eg.
- Object Lifecycle: These relate to creation, initialization and destruction of objects. Eg.
Are magic methods Python's approach to operator overloading?
However, there are many magic methods that don't correspond to an operator. For example,
__repr__()are magic methods corresponding to functions rather than operators.
Further, for example, the
__eq__(), which corresponds to the
==operator. When we implement this magic method we're technically overriding a base class method in OOP parlance. However, we're actually overloading the
This is why it's better to think of magic methods as intuitive interfaces.
What magic methods differ between Python 2 and Python 3?
For object comparisons, Python 2's magic method
__cmp__()is replaced with rich comparison methods in Python 3:
__ge__(). Because it can be extra work to implement all these magic methods for a class, Python 3 provides the decorator
@functools.total_ordering. We can provide implementations for
__eq__()and one of the ordering operators. The decorator will supply the rest.
Python 2's magic method
__nonzero__()is replaced with
__div__()is replaced with
__unicode__()is removed since strings in Python 3 are Unicode by default.
__coerce__is no longer needed since other magic methods do this in Python 3. For example,
2 + 3.5would do type coercion in Python 2 but in Python 3,
__radd__()on the float object is called to add the integer.
Which magic methods are available for object lifecycle constructs?
__del__()are magic methods that allow customization of object creation, initialization and destruction respectively.
__new__()is mainly intended to allow subclasses of immutable types to customize instance creation and to customize class creation in custom metaclasses.
__init__()is mainly intended for object customization before returning to the caller; that is,
__new__()deals with creation and is a static class method while
__init__()deals with initialization and is an instance method.
__del__()is for runtime clean up just before an instance is destroyed, which only happens if the corresponding reference count becomes zero.
There are magic methods to customize class creation itself. Classes themselves are objects: a class is created from its metaclass. Classes themselves use memory, can be assigned, copied and passed as parameters, and even dynamically created. Magic method
__init_subclass__()influences the behaviour of future subclasses as this method is called whenever the class is subclassed. Related magic methods include
Which magic methods are available for object representation?
Objects of type
strare inherently strings but any Python object can be given a string representation. Built-in functions
print()serve this purpose. However, their default behaviours can be customized by defining suitable magic methods.
repr()calls magic method
__repr__()to obtain the official string representation. Built-in functions
print()call magic method
__str__()to obtain an informal and printable string. Since
__repr__()is typically used for debugging, it should return an information-rich and unambiguous string. If possible, it should be a valid Python expression string for data serialization so that the instance can be recreated from the string.
format()and string method
str.format()call magic method
__format__()to evaluate formatted string literals, also known as format-strings. If this magic method is not defined,
__str__()is called instead.
Which magic methods are available for object comparison?
Comparison magic methods allow custom classes to implement specific comparison behaviours. Python defines six object comparison magic methods:
__ge__(), corresponding to
The magic methods should return the singleton
NotImplementedif the comparison is not implemented for the requested operands. They should return a boolean value. Otherwise, Python will call
bool()on the returned object.
__ne__()is undefined, its default behaviour is to call
__eq__()and invert the result. No such relation exists among other comparison operators. For example, Python doesn't enforce that
x < yis same as
y > x. However, implementations should maintain semantic consistency.
How can I customize object conversions?
Any Python object can be used in boolean context, such as
if obj: pass. This operation is trivial for a
booltype object. For other types, we need a way to evaluate the object in a boolean context. Magic method
Evaluating an object in a boolean context or directly calling the built-in function
__bool__(), which must return either
False. Any other return value will cause Python to raise an exception. If
__len__(). The result is considered
Trueif nonzero and
Falseotherwise. If a class doesn't define either of these, all its instances are considered
Likewise, built-in functions
float()trigger other magic methods. A call of
__trunc__()in that order;
__index__()indicates that the object is of integer type. It must return an integer. Python slicing and functions
oct()rely on this.
Which magic methods emulate numeric objects?
Numbers can be added, multiplied or bit-shifted. What if we wish to emulate these behaviours on non-numeric objects? For example, we could add two RGB colour objects or left shift beads on an abacus object. Magic methods associated with numeric operators enable this.
- Left Operand: Given
a + b, the call
- Right Operand: Given
a + b, the call
a.__add__(b)is attempted first. If this doesn't exist or returns
b.__radd__(a)is called. For example,
3 + 5.5will call
3.__add__(5.5). Since an integer doesn't know how to add a float, this returns
5.5.__radd__(3)is called and this succeeds with the addition.
- In-Place Operation: Given
a += b, the call
a.__iadd__(b)is made. This method should modify the object in-place. If this method doesn't exist,
b.__radd__(a)are tried in that order.
Related to these are unary operators
__neg__(); built-in function
__abs__(); similar built-in functions (some in
- Left Operand: Given
Which magic method makes class instances callable?
Given a class
Aand its initializer
__init__(self, i), we can instantiate an object by doing
a = A(2). Now
ais an instance. But it can't be called like a method or function. However, if the class
Aincludes the magic method
__call__(), we can write
a(). Thus, the instance becomes callable (with or without arguments) when the magic method
Callable instances have practical use cases. With
__init__()an object can be initialized in a fundamental way. Suppose it has only one specific behaviour. Hence it defines no methods other than
__call__(). Different types of instances are then created and called. An example is initializing a
Hasherobject with an algorithm:
md5 = Hasher(hashlib.md5)and
sha1 = Hasher(hashlib.sha1). Actual hashing happens later by calling these instances:
Which magic methods are available for customization of attribute access?
__delattr__()allow us to customize attribute access for class instances. This includes use of, assignment to or deletion of
xis an attribute of instance
dir()calls magic method
__dir__()to customize the list of valid instance attributes. Moreover,
__dir__()can be used to customize access to module attributes.
A feature of the language called descriptor introduces further magic methods:
__delete__(). These are more powerful and present a more modular way to control attribute access.
How can I make a custom object hashable?
Certain data structures in Python such as sets, frozensets and dictionaries are implemented using hash tables for efficient lookups and insertions. This requires computing a hash value on the object by calling the built-in function
hash(). Internally, this calls the magic method
By defining this magic method on a class, we can compute the hash value for objects of that class. It must return an integer. Objects that compare equal via the method
__eq__()must have the same hash value. One suggested way to compute a proper hash is to pack comparison-sensitive components of an object into a tuple and hash that tuple. For example,
return hash((self.x, self.y))as the body of
__hash__()method on a class
Point2d. Note that
__hash__()in general is tightly coupled with
__eq__()and has atypical behaviour related to
Python 2.1 is released. Magic methods
__ge__() are introduced to allow overriding the corresponding comparison operators.
Python 2.2 is released. This introduces descriptors into the language via PEP 252. Descriptors themselves have the following attributes:
__delete__(). Descriptors are now used to support static methods and class methods. New features slots and properties are also new kinds of descriptors.
Python 2.5 is released. This introduces the
with statement and its associated magic methods
__exit__(). A common use case of this is to open a file in
__enter__() when the
with context is entered and close the file in
__exit__() as soon as execution comes out of the
with context. Another new method is
__index__() that solves a problem faced by NumPy users. It's now possible to use NumPy types as slice indexes.
Python 3.0 is released. Magic method
__cmp__() is removed. Instead use
__lt__() for sorting,
__hash__() and other rich comparisons. Operator
!= returns the opposite of
__delslice__() are removed. The slice
a[i:j] is implemented as
a.__getitem__(slice(i, j)). Methods
__delitem__() are introduced. Converters
hex() now use
__nonzero__() is now
Python 3.7 is released. Module-level methods introduced are
__dir__(). These can be used to implement module attribute deprecation and lazy loading. New methods
__mro_entries__() bring 7x improvement to performance in relation to types and the
typing module. New module
dataclasses has decorators and functions to automatically add some magic methods to user-defined classes.
- Fedora Python SIG. 2018. "Comparing and Sorting." In: The Conservative Python 3 Porting Guide, Fedora Python SIG, January 1. Accessed 2022-01-08.
- Hettinger, Raymond. 2019. "What’s New in Python 3.8." Release Notes, The Python Software Foundation, October 14. Accessed 2022-01-08.
- Hunner, Trey. 2022. "Python Doesn't Have Type Coercion." Topic series: Overlooked Fundamentals, Python Morsels. Accessed 2022-01-08.
- Kettler, Rafe. 2011. "A Guide to Python's Magic Methods." Version 1.17. Updated 2016-01-19. Accessed 2022-01-08.
- Kuchling, A.M. 2002. "What’s New in Python 2.2." Release Notes, v2.2.2, The Python Software Foundation, October 14. Updated 2021-12-21. Accessed 2021-12-21.
- Kuchling, A.M. 2006. "What’s New in Python 2.5." Release Notes, The Python Software Foundation, September 19. Accessed 2022-01-08.
- Kuchling, A.M. 2008. "What’s New in Python 2.6." Release Notes, The Python Software Foundation, October 1. Accessed 2022-01-08.
- Louis, Pila. 2020. "Constructors in Python (__init vs __new__)." Dev.to, September 16. Accessed 2022-01-08.
- Millikin, John. 2010. "What's a correct and good way to implement __hash__()?" StackOverflow, May 25. Updated 2019-07-07. Accessed 2022-01-13.
- Pranskevichus, Elvis. 2018. "What’s New in Python 3.7." Release Notes, The Python Software Foundation, June 27. Accessed 2022-01-08.
- Python Docs. 2022a. "Data Model." Section 3, The Python Language Reference, v3.10.1, January 7. Accessed 2022-01-08.
- Python Docs. 2022b. "Lexical analysis." Section 2, The Python Language Reference, v3.10.1, January 7. Accessed 2022-01-08.
- Python Docs. 2022c. "Glossary." Documentation, v3.10.1, January 7. Accessed 2022-01-08.
- Python Docs. 2022d. "Classes." Section 9, The Python Tutorial, v3.10.1, January 7. Accessed 2022-01-08.
- Python Docs. 2022e. "functools — Higher-order functions and operations on callable objects." The Python Standard Library, v3.10.1, January 7. Accessed 2022-01-08.
- Python Docs. 2022f. "Built-in Functions." The Python Standard Library, v3.10.1, January 7. Accessed 2022-01-08.
- Python Docs. 2022g. "dataclasses — Data Classes." The Python Standard Library, v3.10.1, January 13. Accessed 2022-01-13.
- Salgado, Pablo Galindo. 2021. "What’s New in Python 3.10." Release Notes, The Python Software Foundation, October 4. Accessed 2022-01-08.
- Souza, Fernando. 2020. "Python Dunder Methods." Level Up Coding, GitConnected, June 29. Accessed 2022-01-08.
- StackOverflow. 2008. "What are metaclasses in Python?" StackOverflow, September 19. Accessed 2022-01-08.
- StackOverflow. 2011. "Python __call__ special method practical example." StackOverflow, April 28. Accessed 2022-01-08.
- StackOverflow. 2014. "Difference between python set and dict internally." StackOverflow, June 11. Updated 2019-08-20. Accessed 2022-01-08.
- The Python Software Foundation. 2000. "Python 1.6." Release Notes, The Python Software Foundation, September 5. Accessed 2022-01-08.
- The Python Software Foundation. 2021. "Python Documentation by Version." The Python Software Foundation, December 6. Accessed 2022-01-08.
- Wikipedia. 2021a. "Object-oriented programming." Wikipedia, December 31. Accessed 2022-01-08.
- Wikipedia. 2021b. "Method overriding." Wikipedia, December 7. Accessed 2022-01-08.
- van Rossum, Guido. 2001. "PEP 252 -- Making Types Look More Like Classes." Python.org, April 19. Updated 2021-12-21. Accessed 2021-11-21.
- van Rossum, Guido. 2008. "What’s New In Python 3.0." Release Notes, v3.0, The Python Software Foundation, December 4. Accessed 2022-01-08.
- van Rossum, Guido, and David Ascher. 2000. "PEP 207 -- Rich Comparisons." Python.org, July 25. Updated 2021-02-09. Accessed 2022-01-08.