Overloading Numpy functions?
Question:
I am hoping for some clarification on overloading Numpy universal functions in class methods.
To illustrate, here is a class myreal with an overloaded cos method. This overloaded method calls cos imported from the math module.
from math import cos
class myreal:
def __init__(self,x):
self.x = x
def cos(self):
return self.__class__(cos(self.x))
def __str__(self):
return self.x.__str__()
x = myreal(3.14)
y = myreal.cos(x)
print(x,y)
This works as expected and results in values
3.14 -0.9999987317275395
And, as expected, simply trying
z = cos(x)
results in an error TypeError: must be real number, not myreal, since outside of the myreal class, cos expects a float argument.
But surprisingly (and here is my question), if I now import cos from numpy, I can call cos(x) as a function, rather than as a method of the myreal class. In other words, this now works:
from numpy import cos
z = cos(x)
So it seems that the myreal.cos() method is now able to overload the Numpy global function cos(x). Is this "multipledispatch" behavior included by design?
Checking the type of the Numpy cos(x) reveals that it is of type `numpy.ufunc’, which suggests an explanation involving Numpy universal functions.
Any clarification about what is going on here would be very interesting and helpful.
Answers:
np.cos given a numeric dtype array (or anything that becomes that):
In [246]: np.cos(np.array([1,2,np.pi]))
Out[246]: array([ 0.54030231, -0.41614684, -1. ])
But if I give it an object dtype array, I get an error:
In [248]: np.cos(np.array([1,2,np.pi,None]))
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
AttributeError: 'int' object has no attribute 'cos'
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
Input In [248], in <cell line: 1>()
----> 1 np.cos(np.array([1,2,np.pi,None]))
TypeError: loop of ufunc does not support argument 0 of type int which has no callable cos method
With the None element the array is object dtype:
In [249]: np.array([1,2,np.pi,None])
Out[249]: array([1, 2, 3.141592653589793, None], dtype=object)
ufunc when given an object dtype array, iterates through the array and tries to use each element’s "method". For something like np.add, it looks for the .__add__ method. For functions like cos (and exp and sqrt) it looks for a method of the same name. Usually this fails because most objects don’t have a cos method. In your case, it didn’t fail – because you defined a cos method.
Try np.sin to see the error.
I wouldn’t call this overloading. It’s just a quirk of how numpy handles object dtype arrays.
myreal.cos is not able to overload the numpy cos. You are calling the numpy cos all the time, except inside myreal or when you write myreal.cos. It seems that numpy functions are friendly, and when numpy cos receives an object it doesn’t know, instead of an error message, it tries to call cos on the object.
from numpy import cos
class myreal:
def __init__(self, x):
self.x = x
def cos(self, myarg=None):
print(f'myreal-cos myarg: {myarg}')
return f'myreal-cos-{self.x}'
def __str__(self):
return self.x.__str__()
Now when we run cos, it’s numpy cos, not ours:
cos(5)
0.28366218546322625
Let’s try to use myarg:
cos(5, myarg='a')
TypeError: cos() got an unexpected keyword argument 'myarg'
It’s still numpy cos, so it doesn’t know the myarg keyword.
cos(myreal(2))
myreal-cos args: None
'myreal-cos-2'
This works, because numpy cos calls our myreal.cos. But it doesn’t work with the myarg keyword.
If we try sin:
sin(myreal(2))
TypeError: loop of ufunc does not support argument 0 of type myreal which has no callable sin method
This error message seems to make sense. numpy sin tries to call myreal sin, but it doesn’t exist.
This friendly feature is useful when we create an array:
A = np.array([myreal(i) for i in range(4)])
array([[<__main__.myreal object at 0x7fc8c9429cf0>,
<__main__.myreal object at 0x7fc8c939b970>],
[<__main__.myreal object at 0x7fc8c939b0a0>,
<__main__.myreal object at 0x7fc8c939a8c0>]], dtype=object)
Let’s call our myreal.cos function on that array:
myreal.cos(A)
----> 9 return f'myreal-cos-{self.x}'
AttributeError: 'numpy.ndarray' object has no attribute 'x'
It doesn’t work, because we didn’t prepare our function to work on arrays. But numpy.cos can, and for each element in the array it will call our myreal.cos:
cos(A)
myreal-cos myarg: None
myreal-cos myarg: None
myreal-cos myarg: None
myreal-cos myarg: None
array(['myreal-cos-0', 'myreal-cos-1', 'myreal-cos-2', 'myreal-cos-3'],
dtype=object)
The answers above both clarified my misunderstanding. Basically, a quirk in Numpy is what gives the impression that myreal class methods cos, sin etc are "overloading" the Numpy functions cos, sin and so on.
Rather than exploiting this quirk in Numpy, however, it seems the more correct way to overload the cos function so it accepts a myreal is to use the singledispatch mechanism.
My original class definitions remain unchanged, but a "cos" function is added to a dispatch registery that is searched when looking for the appropriate function/argument to call.
from functools import singledispatch
class myreal:
def __init__(self,x):
self.x = x
def cos(self):
return myreal(cos(self.x))
def __str__(self):
return self.x.__str__()
@singledispatch
def cos(x):
import math
return math.cos(x)
cos.register(myreal,myreal.cos);
print(cos.registry.keys())
x = myreal(3.14)
y = cos(x)
print(y)
The output is :
dict_keys([<class 'object'>, <class '__main__.myreal'>])
-0.9999987317275395
This also removes a dependence on Numpy (it if it not needed).
Here is a version of the above that uses Numpy arrays (inspired by @AndrzejO, above).
from functools import singledispatch
from numpy import array
class myreal:
def __init__(self,x):
self.x = x
def cos(self):
return self.__class__(cos(self.x))
def __repr__(self):
return self.__str__()
def __str__(self):
return self.x.__str__()
@singledispatch
def cos(x):
import numpy
return numpy.cos(x)
cos.register(myreal,myreal.cos);
x = array([myreal(i) for i in range(5)])
y = cos(x)
print(y)
The output is :
[1.0 0.5403023058681398 -0.4161468365471424 -0.9899924966004454
-0.6536436208636119]
Any comments would be appreciated.
I am hoping for some clarification on overloading Numpy universal functions in class methods.
To illustrate, here is a class myreal with an overloaded cos method. This overloaded method calls cos imported from the math module.
from math import cos
class myreal:
def __init__(self,x):
self.x = x
def cos(self):
return self.__class__(cos(self.x))
def __str__(self):
return self.x.__str__()
x = myreal(3.14)
y = myreal.cos(x)
print(x,y)
This works as expected and results in values
3.14 -0.9999987317275395
And, as expected, simply trying
z = cos(x)
results in an error TypeError: must be real number, not myreal, since outside of the myreal class, cos expects a float argument.
But surprisingly (and here is my question), if I now import cos from numpy, I can call cos(x) as a function, rather than as a method of the myreal class. In other words, this now works:
from numpy import cos
z = cos(x)
So it seems that the myreal.cos() method is now able to overload the Numpy global function cos(x). Is this "multipledispatch" behavior included by design?
Checking the type of the Numpy cos(x) reveals that it is of type `numpy.ufunc’, which suggests an explanation involving Numpy universal functions.
Any clarification about what is going on here would be very interesting and helpful.
np.cos given a numeric dtype array (or anything that becomes that):
In [246]: np.cos(np.array([1,2,np.pi]))
Out[246]: array([ 0.54030231, -0.41614684, -1. ])
But if I give it an object dtype array, I get an error:
In [248]: np.cos(np.array([1,2,np.pi,None]))
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
AttributeError: 'int' object has no attribute 'cos'
The above exception was the direct cause of the following exception:
TypeError Traceback (most recent call last)
Input In [248], in <cell line: 1>()
----> 1 np.cos(np.array([1,2,np.pi,None]))
TypeError: loop of ufunc does not support argument 0 of type int which has no callable cos method
With the None element the array is object dtype:
In [249]: np.array([1,2,np.pi,None])
Out[249]: array([1, 2, 3.141592653589793, None], dtype=object)
ufunc when given an object dtype array, iterates through the array and tries to use each element’s "method". For something like np.add, it looks for the .__add__ method. For functions like cos (and exp and sqrt) it looks for a method of the same name. Usually this fails because most objects don’t have a cos method. In your case, it didn’t fail – because you defined a cos method.
Try np.sin to see the error.
I wouldn’t call this overloading. It’s just a quirk of how numpy handles object dtype arrays.
myreal.cos is not able to overload the numpy cos. You are calling the numpy cos all the time, except inside myreal or when you write myreal.cos. It seems that numpy functions are friendly, and when numpy cos receives an object it doesn’t know, instead of an error message, it tries to call cos on the object.
from numpy import cos
class myreal:
def __init__(self, x):
self.x = x
def cos(self, myarg=None):
print(f'myreal-cos myarg: {myarg}')
return f'myreal-cos-{self.x}'
def __str__(self):
return self.x.__str__()
Now when we run cos, it’s numpy cos, not ours:
cos(5)
0.28366218546322625
Let’s try to use myarg:
cos(5, myarg='a')
TypeError: cos() got an unexpected keyword argument 'myarg'
It’s still numpy cos, so it doesn’t know the myarg keyword.
cos(myreal(2))
myreal-cos args: None
'myreal-cos-2'
This works, because numpy cos calls our myreal.cos. But it doesn’t work with the myarg keyword.
If we try sin:
sin(myreal(2))
TypeError: loop of ufunc does not support argument 0 of type myreal which has no callable sin method
This error message seems to make sense. numpy sin tries to call myreal sin, but it doesn’t exist.
This friendly feature is useful when we create an array:
A = np.array([myreal(i) for i in range(4)])
array([[<__main__.myreal object at 0x7fc8c9429cf0>,
<__main__.myreal object at 0x7fc8c939b970>],
[<__main__.myreal object at 0x7fc8c939b0a0>,
<__main__.myreal object at 0x7fc8c939a8c0>]], dtype=object)
Let’s call our myreal.cos function on that array:
myreal.cos(A)
----> 9 return f'myreal-cos-{self.x}'
AttributeError: 'numpy.ndarray' object has no attribute 'x'
It doesn’t work, because we didn’t prepare our function to work on arrays. But numpy.cos can, and for each element in the array it will call our myreal.cos:
cos(A)
myreal-cos myarg: None
myreal-cos myarg: None
myreal-cos myarg: None
myreal-cos myarg: None
array(['myreal-cos-0', 'myreal-cos-1', 'myreal-cos-2', 'myreal-cos-3'],
dtype=object)
The answers above both clarified my misunderstanding. Basically, a quirk in Numpy is what gives the impression that myreal class methods cos, sin etc are "overloading" the Numpy functions cos, sin and so on.
Rather than exploiting this quirk in Numpy, however, it seems the more correct way to overload the cos function so it accepts a myreal is to use the singledispatch mechanism.
My original class definitions remain unchanged, but a "cos" function is added to a dispatch registery that is searched when looking for the appropriate function/argument to call.
from functools import singledispatch
class myreal:
def __init__(self,x):
self.x = x
def cos(self):
return myreal(cos(self.x))
def __str__(self):
return self.x.__str__()
@singledispatch
def cos(x):
import math
return math.cos(x)
cos.register(myreal,myreal.cos);
print(cos.registry.keys())
x = myreal(3.14)
y = cos(x)
print(y)
The output is :
dict_keys([<class 'object'>, <class '__main__.myreal'>])
-0.9999987317275395
This also removes a dependence on Numpy (it if it not needed).
Here is a version of the above that uses Numpy arrays (inspired by @AndrzejO, above).
from functools import singledispatch
from numpy import array
class myreal:
def __init__(self,x):
self.x = x
def cos(self):
return self.__class__(cos(self.x))
def __repr__(self):
return self.__str__()
def __str__(self):
return self.x.__str__()
@singledispatch
def cos(x):
import numpy
return numpy.cos(x)
cos.register(myreal,myreal.cos);
x = array([myreal(i) for i in range(5)])
y = cos(x)
print(y)
The output is :
[1.0 0.5403023058681398 -0.4161468365471424 -0.9899924966004454
-0.6536436208636119]
Any comments would be appreciated.