With numpy, I can do a simple matrix multiplication like this:
a = numpy.ones((3, 2)) b = numpy.ones((2, 1)) result = a.dot(b)
However, this does not work with PyTorch:
a = torch.ones((3, 2)) b = torch.ones((2, 1)) result = torch.dot(a, b)
This code throws the following error:
RuntimeError: 1D tensors expected, but got 2D and 2D tensors
How do I perform matrix multiplication in PyTorch?
torch.dot() behaves differently to
np.dot(). There’s been some discussion about what would be desirable here. Specifically,
torch.dot() treats both
b as 1D vectors (irrespective of their original shape) and computes their inner product. The error is thrown because this behaviour makes your
a a vector of length 6 and your
b a vector of length 2; hence their inner product can’t be computed. For matrix multiplication in PyTorch, use
np.dot() in contrast is more flexible; it computes the inner product for 1D arrays and performs matrix multiplication for 2D arrays.
torch.matmul performs matrix multiplications if both arguments are
2D and computes their dot product if both arguments are
1D. For inputs of such dimensions, its behaviour is the same as
np.dot. It also lets you do broadcasting or
matrix x matrix,
matrix x vector and
vector x vector operations in batches.
# 1D inputs, same as torch.dot a = torch.rand(n) b = torch.rand(n) torch.matmul(a, b) # torch.Size() # 2D inputs, same as torch.mm a = torch.rand(m, k) b = torch.rand(k, j) torch.matmul(a, b) # torch.Size([m, j])
To perform a matrix (rank 2 tensor) multiplication, use any of the following equivalent ways:
AB = A.mm(B) AB = torch.mm(A, B) AB = torch.matmul(A, B) AB = A @ B # Python 3.5+ only
There are a few subtleties. From the PyTorch documentation:
torch.mmdoes not broadcast. For broadcasting matrix products,
For instance, you cannot multiply two 1-dimensional vectors with
torch.mm, nor multiply batched matrices (rank 3). To this end, you should use the more versatile
torch.matmul. For an extensive list of the broadcasting behaviours of
torch.matmul, see the documentation.
For element-wise multiplication, you can simply do (if A and B have the same shape)
A * B # element-wise matrix multiplication (Hadamard product)
torch.mm(a, b) or
Both are same.
>>> torch.mm <built-in method mm of type object at 0x11712a870> >>> torch.matmul <built-in method matmul of type object at 0x11712a870>
There’s one more option that may be good to know.
@ operator. @Simon H.
>>> a = torch.randn(2, 3) >>> b = torch.randn(3, 4) >>> a@b tensor([[ 0.6176, -0.6743, 0.5989, -0.1390], [ 0.8699, -0.3445, 1.4122, -0.5826]]) >>> a.mm(b) tensor([[ 0.6176, -0.6743, 0.5989, -0.1390], [ 0.8699, -0.3445, 1.4122, -0.5826]]) >>> a.matmul(b) tensor([[ 0.6176, -0.6743, 0.5989, -0.1390], [ 0.8699, -0.3445, 1.4122, -0.5826]])
The three give the same results.
You can use "@" for computing a dot product between two tensors in pytorch.
a = torch.tensor([[1,2], [3,4]]) b = torch.tensor([[5,6], [7,8]]) c = a@b #For dot product c d = a*b #For elementwise multiplication d