Python – Torchmetric SSIM depends on batch size

Question

sry for the quick question, I just want to know wether I found a bug or if I do not understand something here. I got the following sample, where I print the ssim of the torchmetrics library of two tensors with the batchsize 8 and the single calculated values mean.

Why are they not the same?

ssim = StructuralSimilarityIndexMeasure(kernel_size=(5, 5))

A = torch.zeros([8, 1, 500, 500])
B = torch.randn([8, 1, 500, 500])

print(ssim(A, B))

ssim1 = ssim(A[0].unsqueeze(0), B[0].unsqueeze(0))
ssim2 = ssim(A[1].unsqueeze(0), B[1].unsqueeze(0))
ssim3 = ssim(A[2].unsqueeze(0), B[2].unsqueeze(0))
ssim4 = ssim(A[3].unsqueeze(0), B[3].unsqueeze(0))
ssim5 = ssim(A[4].unsqueeze(0), B[4].unsqueeze(0))
ssim6 = ssim(A[5].unsqueeze(0), B[5].unsqueeze(0))
ssim7 = ssim(A[6].unsqueeze(0), B[6].unsqueeze(0))
ssim8 = ssim(A[7].unsqueeze(0), B[7].unsqueeze(0))

print((ssim1 + ssim2 + ssim3 + ssim4 + ssim5 + ssim6 + ssim7 + ssim8) / 8)

Console output:

tensor(0.0404)
tensor(0.0340)

Python version 3.8.10

TrochMetrics version 0.9.1

PyTroch version 1.10.1+cu113

Or is this a git issue?

Asked By: CaipiDE

||

Source

Answer 1

The difference comes from the data_range parameter. Please refer to the documentations:

data_range: Range of the image. If ``None``, it is determined from the image (max - min)

https://torchmetrics.readthedocs.io/en/stable/image/structural_similarity.html#structural-similarity-index-measure-ssim

By default it is None so the batch and individual examples of batch will have different data_range derived from the data:

if data_range is None:
    data_range = max(preds.max() - preds.min(), target.max() - target.min())

https://github.com/Lightning-AI/metrics/blob/19355a9d2c51b3b39b311694d7b8e6856a73eae6/src/torchmetrics/functional/image/ssim.py#L129

If you set specific data_range like this:

ssim = StructuralSimilarityIndexMeasure(kernel_size=(5, 5), data_range=255)

The results will be the same.

Answered By: u1234x1234

Answer 2

@u1234x1234 is right, that solved it, did not think of that thanks.

I will later post a solution of the sample of how to redefine the SSIM metric in the example to always adjust the data_range accordingly

Answered By: CaipiDE

Python – Torchmetric SSIM depends on batch size

Question:

Answers: