softmax_loss function: Turn the loop into matrix operation

Question

I am now learning the stanford cs231n course. When completing the softmax_loss function, I found it is not easy to write in a full-vectorized type, especially dealing with the dw term. Below is my code. Can somebody optimize the code. Would be appreciated.

def softmax_loss_vectorized(W, X, y, reg):

  loss = 0.0
  dW = np.zeros_like(W)


  num_train = X.shape[0]
  num_classes = W.shape[1]

  scores = X.dot(W)
  scores -= np.max(scores, axis = 1)[:, np.newaxis]
  exp_scores = np.exp(scores)
  sum_exp_scores = np.sum(exp_scores, axis = 1)
  correct_class_score = scores[range(num_train), y]

  loss = np.sum(np.log(sum_exp_scores)) - np.sum(correct_class_score)

  exp_scores = exp_scores / sum_exp_scores[:,np.newaxis]

  # **maybe here can be rewroten into matrix operations** 
  for i in xrange(num_train):
    dW += exp_scores[i] * X[i][:,np.newaxis]
    dW[:, y[i]] -= X[i]

  loss /= num_train
  loss += 0.5 * reg * np.sum( W*W )
  dW /= num_train
  dW += reg * W


  return loss, dW

Asked By: luoshao23

||

Source

Answer 1

Here’s a vectorized implementation below. But I suggest you try to spend a little bit more time and get to the solution yourself. The idea is to construct a matrix with all softmax values and subtract -1 from the correct elements.

def softmax_loss_vectorized(W, X, y, reg):
  num_train = X.shape[0]

  scores = X.dot(W)
  scores -= np.max(scores)
  correct_scores = scores[np.arange(num_train), y]

  # Compute the softmax per correct scores in bulk, and sum over its logs.
  exponents = np.exp(scores)
  sums_per_row = np.sum(exponents, axis=1)
  softmax_array = np.exp(correct_scores) / sums_per_row
  information_array = -np.log(softmax_array)
  loss = np.mean(information_array)

  # Compute the softmax per whole scores matrix, which gives the matrix for X rows coefficients.
  # Their linear combination is algebraically dot product X transpose.
  all_softmax_matrix = (exponents.T / sums_per_row).T
  grad_coeff = np.zeros_like(scores)
  grad_coeff[np.arange(num_train), y] = -1
  grad_coeff += all_softmax_matrix
  dW = np.dot(X.T, grad_coeff) / num_train

  # Regularization
  loss += 0.5 * reg * np.sum(W * W)
  dW += reg * W

  return loss, dW

Answered By: Maxim

Answer 2

It may be a naive question, but why do we have to calculate a corrected score? ( correct_scores = scores[np.arange(num_train), y] )?

Answered By: Alexandre Laborie

softmax_loss function: Turn the loop into matrix operation

Question:

Answers: