TypeError: unsupported operand type(s) for /: 'SequenceClassifierOutput' and 'int'

Question

I am using hugginface library to train a bert model on classification problem.

    model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=10) 

    def training_step(self, batch, batch_nb):
       sequence, label = batch
       input_ids, attention_mask, labels = self.prepare_batch(sequence=sequence, label=label)
       loss = self.model(input_ids=input_ids,
                      attention_mask=attention_mask,
                      labels=labels) 
       tensorboard_logs = {'train_loss': loss}

I am getting the following error just before the training starts:

in training_step
closure_loss = closure_loss / self.trainer.accumulate_grad_batches 
TypeError: unsupported operand type(s) for /: 'SequenceClassifierOutput' and 'int'

I am using pytorch-lightning

Asked By: Chan Wing

||

Source

Answer 1

Calling self.model() returns an object of type SequenceClassifierOutput.
To access the loss you need to call it’s loss attribute:
Replace

loss = self.model(input_ids=input_ids,
                      attention_mask=attention_mask,
                      labels=labels)

by

output = self.model(input_ids=input_ids,
                      attention_mask=attention_mask,
                      labels=labels) 
loss = output.loss

Answered By: JulienBr

TypeError: unsupported operand type(s) for /: 'SequenceClassifierOutput' and 'int'

Question:

Answers: