What is the correct way to override the __dir__ method?
Question:
This question is meant to be more about __dir__
than about numpy
.
I have a subclass of numpy.recarray
(in python 2.7, numpy 1.6.2), and I noticed recarray
‘s field names are not listed when dir
ing the object (and therefore ipython’s autocomplete doesn’t work).
Trying to fix it, I tried overriding __dir__
in my subclass, like this:
def __dir__(self):
return sorted(set(
super(MyRecArray, self).__dir__() +
self.__dict__.keys() + self.dtype.fields.keys()))
which resulted with: AttributeError: 'super' object has no attribute '__dir__'
.
(I found here this should actually work in python 3.3…)
As a workaround, I tried:
def __dir__(self):
return sorted(set(
dir(type(self)) +
self.__dict__.keys() + self.dtype.fields.keys()))
As far as I can tell, this one works, but of course, not as elegantly.
Questions:
- Is the latter solution correct in my case, i.e. for a subclass of
recarray
?
- Is there a way to make it work in the general case? It seems to me it wouldn’t work with multiple inheritance (breaking the
super
-call chain), and of course, for objects with no __dict__
…
- Do you know why
recarray
does not support listing its field names to begin with? mere oversight?
Answers:
-
and 3: Yes your solution is correct. recarray
does not define __dir__
simply because the default implementation was okay, so they didn’t bother implementing it, and numpy
‘s devs did not design the class to be subclassed, so I don’t see why they should have bothered.
It’s often a bad idea to subclass built-in types or classes that are not specifically designed for inheritance, thus I’d suggest you to use delegation/composition instead of inheritance, except if there is a particular reason(e.g. you want to pass it to a numpy
function that excplicitly checks with isinstance
).
-
No. As you pointed out in python3 they changed the implementation so that there is an object.__dir__
, but on other python versions I can’t see anything that you can do. Also, again, using recarray
with multiple-inheritance is simply crazy, things will break. Multiple-inheritance should be carefully designed, and usually classes are specifically designed to be used with it(e.g. mix-ins). So I wouldn’t bother treating this case, since whoever tries it will be bitten by other problems.
I don’t see why you should care for classes that do not have __dict__
… since your subclass has it how should it break? When you’ll change the subclass implementation, e.g. using __slots__
you could easily change the __dir__
also. If you want to avoid redefining __dir__
you can simply define a function that checks for __dict__
then for __slots__
etc. Note however that attributes can be generated in subtle ways with __getattr__
and __getattribute__
and thus you simply can’t reliably catch all of them.
Python 2.7+, 3.3+ class mixin that simplifies implementation of __dir__ method in subclasses. Hope it will help. Gist.
import six
class DirMixIn:
""" Mix-in to make implementing __dir__ method in subclasses simpler
"""
def __dir__(self):
if six.PY3:
return super(DirMixIn, self).__dir__()
else:
# code is based on
# http://www.quora.com/How-dir-is-implemented-Is-there-any-PEP-related-to-that
def get_attrs(obj):
import types
if not hasattr(obj, '__dict__'):
return [] # slots only
if not isinstance(obj.__dict__, (dict, types.DictProxyType)):
raise TypeError("%s.__dict__ is not a dictionary"
"" % obj.__name__)
return obj.__dict__.keys()
def dir2(obj):
attrs = set()
if not hasattr(obj, '__bases__'):
# obj is an instance
if not hasattr(obj, '__class__'):
# slots
return sorted(get_attrs(obj))
klass = obj.__class__
attrs.update(get_attrs(klass))
else:
# obj is a class
klass = obj
for cls in klass.__bases__:
attrs.update(get_attrs(cls))
attrs.update(dir2(cls))
attrs.update(get_attrs(obj))
return list(attrs)
return dir2(self)
Have you tried:
def __dir__(self):
return sorted(set(
dir(super(MyRecArray, self)) +
self.__dict__.keys() + self.dtype.fields.keys()))
This question is meant to be more about __dir__
than about numpy
.
I have a subclass of numpy.recarray
(in python 2.7, numpy 1.6.2), and I noticed recarray
‘s field names are not listed when dir
ing the object (and therefore ipython’s autocomplete doesn’t work).
Trying to fix it, I tried overriding __dir__
in my subclass, like this:
def __dir__(self):
return sorted(set(
super(MyRecArray, self).__dir__() +
self.__dict__.keys() + self.dtype.fields.keys()))
which resulted with: AttributeError: 'super' object has no attribute '__dir__'
.
(I found here this should actually work in python 3.3…)
As a workaround, I tried:
def __dir__(self):
return sorted(set(
dir(type(self)) +
self.__dict__.keys() + self.dtype.fields.keys()))
As far as I can tell, this one works, but of course, not as elegantly.
Questions:
- Is the latter solution correct in my case, i.e. for a subclass of
recarray
? - Is there a way to make it work in the general case? It seems to me it wouldn’t work with multiple inheritance (breaking the
super
-call chain), and of course, for objects with no__dict__
… - Do you know why
recarray
does not support listing its field names to begin with? mere oversight?
-
and 3: Yes your solution is correct.
recarray
does not define__dir__
simply because the default implementation was okay, so they didn’t bother implementing it, andnumpy
‘s devs did not design the class to be subclassed, so I don’t see why they should have bothered.It’s often a bad idea to subclass built-in types or classes that are not specifically designed for inheritance, thus I’d suggest you to use delegation/composition instead of inheritance, except if there is a particular reason(e.g. you want to pass it to a
numpy
function that excplicitly checks withisinstance
). -
No. As you pointed out in python3 they changed the implementation so that there is an
object.__dir__
, but on other python versions I can’t see anything that you can do. Also, again, usingrecarray
with multiple-inheritance is simply crazy, things will break. Multiple-inheritance should be carefully designed, and usually classes are specifically designed to be used with it(e.g. mix-ins). So I wouldn’t bother treating this case, since whoever tries it will be bitten by other problems.I don’t see why you should care for classes that do not have
__dict__
… since your subclass has it how should it break? When you’ll change the subclass implementation, e.g. using__slots__
you could easily change the__dir__
also. If you want to avoid redefining__dir__
you can simply define a function that checks for__dict__
then for__slots__
etc. Note however that attributes can be generated in subtle ways with__getattr__
and__getattribute__
and thus you simply can’t reliably catch all of them.
Python 2.7+, 3.3+ class mixin that simplifies implementation of __dir__ method in subclasses. Hope it will help. Gist.
import six
class DirMixIn:
""" Mix-in to make implementing __dir__ method in subclasses simpler
"""
def __dir__(self):
if six.PY3:
return super(DirMixIn, self).__dir__()
else:
# code is based on
# http://www.quora.com/How-dir-is-implemented-Is-there-any-PEP-related-to-that
def get_attrs(obj):
import types
if not hasattr(obj, '__dict__'):
return [] # slots only
if not isinstance(obj.__dict__, (dict, types.DictProxyType)):
raise TypeError("%s.__dict__ is not a dictionary"
"" % obj.__name__)
return obj.__dict__.keys()
def dir2(obj):
attrs = set()
if not hasattr(obj, '__bases__'):
# obj is an instance
if not hasattr(obj, '__class__'):
# slots
return sorted(get_attrs(obj))
klass = obj.__class__
attrs.update(get_attrs(klass))
else:
# obj is a class
klass = obj
for cls in klass.__bases__:
attrs.update(get_attrs(cls))
attrs.update(dir2(cls))
attrs.update(get_attrs(obj))
return list(attrs)
return dir2(self)
Have you tried:
def __dir__(self):
return sorted(set(
dir(super(MyRecArray, self)) +
self.__dict__.keys() + self.dtype.fields.keys()))