Why doesn't the namedtuple module use a metaclass to create nt class objects?
Question:
I spent some time investigating the collections.namedtuple module a few weeks ago. The module uses a factory function which populates the dynamic data (the name of the new namedtuple class, and the class attribute names) into a very large string. Then exec is executed with the string (which represents the code) as the argument, and the new class is returned.
Does anyone know why it was done this way, when there is a specific tool for this kind of thing readily available, i.e. the metaclass? I haven’t tried to do it myself, but it seems like everything that is happening in the namedtuple module could have been easily accomplished using a namedtuple metaclass, like so:
class namedtuple(type):
etc etc.
EDIT: Humbly suggest reading the answer I wrote several years later, which is a bit down the page.
Answers:
There are some hints in the issue 3974. The author proposed a new way to create named tuples, which was rejected with the following comments:
It seems the benefit of the original version is that it’s faster,
thanks to hardcoding critical methods.
– Antoine Pitrou
There is nothing unholy about using exec. Earlier versions used other
approaches and they proved unnecessarily complex and had unexpected
problems. It is a key feature for named tuples that they are exactly
equivalent to a hand-written class. – Raymond Hettinger
Additionally, here is the part of the description of the original namedtuple recipe:
… the recipe has evolved to its current exec-style where we get all
of Python’s high-speed builtin argument checking for free. The new
style of building and exec-ing a template made both the __new__ and
__repr__ functions faster and cleaner than in previous versions of this recipe.
If you’re looking for some alternative implementations:
-
abstract base class + mix-in for named tuples recipe by Jan Kaliszewski
-
metaclass-based implementation by Aaron Iles (see his blog post)
As a sidenote: The other objection I see most often against using exec is that some locations (read companies) disable it for security reasons.
Besides an advanced Enum and NamedConstant, the aenum library* also has NamedTuple which is metaclass-based.
* aenum is written by the author of enum and the enum34 backport.
Here is another approach.
""" Subclass of tuple with named fields """
from operator import itemgetter
from inspect import signature
class MetaTuple(type):
""" metaclass for NamedTuple """
def __new__(mcs, name, bases, namespace):
cls = type.__new__(mcs, name, bases, namespace)
names = signature(cls._signature).parameters.keys()
for i, key in enumerate(names):
setattr(cls, key, property(itemgetter(i)))
return cls
class NamedTuple(tuple, metaclass=MetaTuple):
""" Subclass of tuple with named fields """
@staticmethod
def _signature():
" Override in subclass "
def __new__(cls, *args):
new = super().__new__(cls, *args)
if len(new) == len(signature(cls._signature).parameters):
return new
return new._signature(*new)
if __name__ == '__main__':
class Point(NamedTuple):
" Simple test "
@staticmethod
def _signature(x, y, z): # pylint: disable=arguments-differ
" Three coordinates "
print(Point((1, 2, 4)))
If this approach has any virtue at all, it’s the simplicity. It would be simpler yet without NamedTuple.__new__, which serves only the purpose of enforcing the element count. Without that, it happily allows additional anonymous elements past the named ones, and the primary effect of omitting elements is the IndexError on omitted elements when accessing them by name (with a little work that could be translated to an AttributeError). The error message for an incorrect element count is a bit strange, but it gets the point across. I wouldn’t expect this to work with Python 2.
There is room for further complication, such as a __repr__ method. I have no idea how the performance compares to other implementations (caching the signature length might help), but I much prefer the calling convention as compared to the native namedtuple implementation.
Coming back to this question after many years of experience: below are a couple of other reasons that none of the other answers hit upon*.
Metaclass, or metaprogramming?
A fundamental question that is passed over by the question "why not just use a metaclass?!?" is: what is the purpose of nt?
The purpose is not merely to create a class factory. If it were that, the metaclass would be nearly perfect. The real purpose of namedtuple is not just the end functionality, but automatically producing classes that are simple and easy to understand in every way, as if written by hand by an experienced professional. And this requires meta programming— automatic generation not of a class, but of code. These are two different things. It is very similar to the newer dataclasses module, which writes methods for you (rather than writing an entire class, like namedtuple).
Only 1 metaclass per class allowed
A class can only have 1 metaclass. The metaclass acts as the factory that creates the class, and it isn’t possible to mix factories together willy nilly. You must create either a "combinatory factory" that knows how to call the multiple factories in the correct order, or a "child factory" that knows about the "parent factory", and uses it correctly.
If the namedtuple used its own metaclass, inheritance involving any other metaclass would break:
>>> class M1(type): ...
...
>>> class M2(type): ...
...
>>> class C1(metaclass=M1): ...
...
>>> class C2(metaclass=M2): ...
...
>>> class C(C1, C2): ...
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
Instead, if you wanted to have your own metaclass and inherit from a namedtuple class, you’d have to use some sort of so-called namedtuple_meta metaclass to do that:
from namedtuple import namedtuple_meta # pretending this exists
class MyMeta(type): ...
class MyMetaWithNT(namedtuple_meta, MyMeta): ...
class C(metaclass=MyMetaWithNT): ...
..or just inherit the custom metaclass from namedtuple_meta directly:
class MyMeta(namedtuple_meta): ...
class C(metaclass=MyMeta): ...
This looks easy at first, but writing your own metaclass that plays nicely with some (complicated) nt metaclass could become problematic very quickly. This limitation would probably not come up all THAT often, but often enough that it would hinder the usage of namedtuple. So it is definitely an advantage to have all namedtuple classes be of the type type, and removing the complexity of a custom metaclass.
* Raymond Hettinger’s comment does hint at it:
It is a key feature for named tuples that they are exactly equivalent to a hand-written class.
I spent some time investigating the collections.namedtuple module a few weeks ago. The module uses a factory function which populates the dynamic data (the name of the new namedtuple class, and the class attribute names) into a very large string. Then exec is executed with the string (which represents the code) as the argument, and the new class is returned.
Does anyone know why it was done this way, when there is a specific tool for this kind of thing readily available, i.e. the metaclass? I haven’t tried to do it myself, but it seems like everything that is happening in the namedtuple module could have been easily accomplished using a namedtuple metaclass, like so:
class namedtuple(type):
etc etc.
EDIT: Humbly suggest reading the answer I wrote several years later, which is a bit down the page.
There are some hints in the issue 3974. The author proposed a new way to create named tuples, which was rejected with the following comments:
It seems the benefit of the original version is that it’s faster,
thanks to hardcoding critical methods.
– Antoine Pitrou
There is nothing unholy about using exec. Earlier versions used other
approaches and they proved unnecessarily complex and had unexpected
problems. It is a key feature for named tuples that they are exactly
equivalent to a hand-written class. – Raymond Hettinger
Additionally, here is the part of the description of the original namedtuple recipe:
… the recipe has evolved to its current exec-style where we get all
of Python’s high-speed builtin argument checking for free. The new
style of building and exec-ing a template made both the __new__ and
__repr__ functions faster and cleaner than in previous versions of this recipe.
If you’re looking for some alternative implementations:
-
abstract base class + mix-in for named tuples recipe by Jan Kaliszewski
-
metaclass-based implementation by Aaron Iles (see his blog post)
As a sidenote: The other objection I see most often against using exec is that some locations (read companies) disable it for security reasons.
Besides an advanced Enum and NamedConstant, the aenum library* also has NamedTuple which is metaclass-based.
* aenum is written by the author of enum and the enum34 backport.
Here is another approach.
""" Subclass of tuple with named fields """
from operator import itemgetter
from inspect import signature
class MetaTuple(type):
""" metaclass for NamedTuple """
def __new__(mcs, name, bases, namespace):
cls = type.__new__(mcs, name, bases, namespace)
names = signature(cls._signature).parameters.keys()
for i, key in enumerate(names):
setattr(cls, key, property(itemgetter(i)))
return cls
class NamedTuple(tuple, metaclass=MetaTuple):
""" Subclass of tuple with named fields """
@staticmethod
def _signature():
" Override in subclass "
def __new__(cls, *args):
new = super().__new__(cls, *args)
if len(new) == len(signature(cls._signature).parameters):
return new
return new._signature(*new)
if __name__ == '__main__':
class Point(NamedTuple):
" Simple test "
@staticmethod
def _signature(x, y, z): # pylint: disable=arguments-differ
" Three coordinates "
print(Point((1, 2, 4)))
If this approach has any virtue at all, it’s the simplicity. It would be simpler yet without NamedTuple.__new__, which serves only the purpose of enforcing the element count. Without that, it happily allows additional anonymous elements past the named ones, and the primary effect of omitting elements is the IndexError on omitted elements when accessing them by name (with a little work that could be translated to an AttributeError). The error message for an incorrect element count is a bit strange, but it gets the point across. I wouldn’t expect this to work with Python 2.
There is room for further complication, such as a __repr__ method. I have no idea how the performance compares to other implementations (caching the signature length might help), but I much prefer the calling convention as compared to the native namedtuple implementation.
Coming back to this question after many years of experience: below are a couple of other reasons that none of the other answers hit upon*.
Metaclass, or metaprogramming?
A fundamental question that is passed over by the question "why not just use a metaclass?!?" is: what is the purpose of nt?
The purpose is not merely to create a class factory. If it were that, the metaclass would be nearly perfect. The real purpose of namedtuple is not just the end functionality, but automatically producing classes that are simple and easy to understand in every way, as if written by hand by an experienced professional. And this requires meta programming— automatic generation not of a class, but of code. These are two different things. It is very similar to the newer dataclasses module, which writes methods for you (rather than writing an entire class, like namedtuple).
Only 1 metaclass per class allowed
A class can only have 1 metaclass. The metaclass acts as the factory that creates the class, and it isn’t possible to mix factories together willy nilly. You must create either a "combinatory factory" that knows how to call the multiple factories in the correct order, or a "child factory" that knows about the "parent factory", and uses it correctly.
If the namedtuple used its own metaclass, inheritance involving any other metaclass would break:
>>> class M1(type): ...
...
>>> class M2(type): ...
...
>>> class C1(metaclass=M1): ...
...
>>> class C2(metaclass=M2): ...
...
>>> class C(C1, C2): ...
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
Instead, if you wanted to have your own metaclass and inherit from a namedtuple class, you’d have to use some sort of so-called namedtuple_meta metaclass to do that:
from namedtuple import namedtuple_meta # pretending this exists
class MyMeta(type): ...
class MyMetaWithNT(namedtuple_meta, MyMeta): ...
class C(metaclass=MyMetaWithNT): ...
..or just inherit the custom metaclass from namedtuple_meta directly:
class MyMeta(namedtuple_meta): ...
class C(metaclass=MyMeta): ...
This looks easy at first, but writing your own metaclass that plays nicely with some (complicated) nt metaclass could become problematic very quickly. This limitation would probably not come up all THAT often, but often enough that it would hinder the usage of namedtuple. So it is definitely an advantage to have all namedtuple classes be of the type type, and removing the complexity of a custom metaclass.
* Raymond Hettinger’s comment does hint at it:
It is a key feature for named tuples that they are exactly equivalent to a hand-written class.