Pytorch: Receiving same test evaluations each round

Question

I have a problem with my federated learning setup. I have some weights which I want to evaluate (test) each round using PyTorch. For now im interested in the loss and accuracy of the model.
The weights are a list of numpy arrays.
The model im using is the following:

class Net(nn.Module):
    """Model (simple CNN adapted from 'PyTorch: A 60 Minute Blitz')"""

    def __init__(self) -> None:
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.fc3(x)

and the methods which produce the results:

def load_testset():
    """Load CIFAR-10 (test set)."""
    trf = Compose([ToTensor(), Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    testset = CIFAR10("./data", train=False, download=True, transform=trf)
    return DataLoader(testset), testset

def get_AccuracyAndLoss(weights):
    # Load the existing weights list
    weights_list = weights

    for i, weights in enumerate(weights_list):
        layer_name = 'layer_' + str(i)
        setattr(Net(), layer_name, nn.Parameter(torch.from_numpy(weights)))

    # Load model and data (simple CNN, CIFAR-10)
    net = Net().to(DEVICE)
    testloader, test_set = load_testset()

    criterion = torch.nn.CrossEntropyLoss()
    correct, total, loss = 0, 0, 0.0
    net.eval()
    with torch.no_grad():
        for images, labels in testloader:
            images, labels = images.to(DEVICE), labels.to(DEVICE)
            outputs = net(images)

            loss += criterion(outputs, labels).item()
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    print('Test_loss: %.3f, accuracy: %.2f' % (loss/len(testloader), correct / total))

The issue is that i get the same results each round , which is around Test_loss: 2.306, accuracy: 0.10 and i dont know whether the model is incorrect or the way the weights get inserted. Im fairly new to deep learning so unsophisticated answers are appreciated.

Most of the code is based of the pytorch deep learning tutorial here : https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html

Im using python 3.10

Asked By: kyro121

||

Source

Answer 1

From the code you provided, I think your model is not correctly initialized.
Since each time you use Net() you are creating a new model with random weights, so your for loop is just creating a completely new model for each layer, initialize one layer of each model correctly, and then drop the model since it is not used again. After the for loop, you are creating another new model with all random weights and are using it for validation.

To fix it, you should :

First create a model with net = Net().to(DEVICE)
Then, do your for loop to initialize correctly each layer of this model with setattr(net, layer_name, nn.Parameters(...))

By doing it in this order, the result should be better, assuming you are training your model between each validation.

If you want, it could also be easier to use load_state_dict (see this tutorial on pytorch website) to avoid having to browse each layer of your model.

Answered By: tgaudier

Pytorch: Receiving same test evaluations each round

Question:

Answers: