What is "metrics" in Keras?
Question:
It is not yet clear for me what metrics are (as given in the code below). What exactly are they evaluating? Why do we need to define them in the model? Why we can have multiple metrics in one model? And more importantly what is the mechanics behind all this?
Any scientific reference is also appreciated.
model.compile(loss='mean_squared_error',
optimizer='sgd',
metrics=['mae', 'acc'])
Answers:
As in keras metrics page described:
A metric is a function that is used to judge the performance of your
model
Metrics are frequently used with early stopping callback to terminate training and avoid overfitting
Reference: Keras Metrics Documentation
As given in the documentation page of keras metrics, a metric judges the performance of your model. The metrics argument in the compile method holds the list of metrics that needs to be evaluated by the model during its training and testing phases.
Metrics like:
-
binary_accuracy
-
categorical_accuracy
-
sparse_categorical_accuracy
-
top_k_categorical_accuracy and
-
sparse_top_k_categorical_accuracy
are the available metric functions that are supplied in the metrics parameter when the model is compiled.
Metric functions are customizable as well. When multiple metrics need to be evaluated it is passed in the form of a dictionary or a list.
One important resource you should refer for diving deep into metrics can be found here
So in order to understand what metrics are, it’s good to start by understanding what a loss function is. Neural networks are mostly trained using gradient methods by an iterative process of decreasing a loss function.
A loss is designed to have two crucial properties – first, the smaller its value is, the better your model fits your data, and second, it should be differentiable. So, knowing this, we could fully define what a metric is: it’s a function that, given predicted values and ground truth values from examples, provides you with a scalar measure of a “fitness” of your model, to the data you have. So, as you may see, a loss function is a metric, but the opposite doesn’t always hold. To understand these differences, let’s look at the most common examples of metrics usage:
-
Measure a performance of your network using non-differentiable functions: e.g. accuracy is not differentiable (not even continuous) so you cannot directly optimize your network w.r.t. to it. However, you could use it in order to choose the model with the best accuracy.
-
Obtain values of different loss functions when your final loss is a combination of a few of them: Let’s assume that your loss has a regularization term which measures how your weights differ from 0, and a term which measures the fitness of your model. In this case, you could use metrics in order to have a separate track of how the fitness of your model changes across epochs.
-
Track a measure with respect to which you don’t want to directly optimize your model: so – let’s assume that you are solving a multidimensional regression problem where you are mostly concerned about mse, but at the same time you are interested in how a cosine-distance of your solution is changing in time. Then, it’s the best to use metrics.
I hope that the explanation presented above made obvious what metrics are used for, and why you could use multiple metrics in one model. So now, let’s say a few words about mechanics of their usage in keras. There are two ways of computing them while training:
-
Using metrics defined while compilation: this is what you directly asked. In this case, keras is defining a separate tensor for each metric you defined, to have it computed while training. This usually makes computation faster, but this comes at a cost of additional compilations, and the fact that metrics should be defined in terms of keras.backend functions.
-
Using keras.callback: It is nice that you can use Callbacks in order to compute your metrics. As each callback has a default attribute of model, you could compute a variety of metrics using model.predict or model parameters while training. Moreover, it makes it possible to compute it, not only epoch-wise, but also batch-wise, or training-wise. This comes at a cost of slower computations, and more complicated logic – as you need to define metrics on your own.
Here you can find a list of available metrics, as well as an example on how you could define your own.
From an implementation point of view, losses and metrics are actually identical functions in Keras:
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow.keras as Keras
>>> print(Keras.losses.mean_squared_error == Keras.metrics.mean_squared_error)
True
>>> print(Keras.losses.poisson == Keras.metrics.poisson)
True
Loss helps find the best solution your model can produce. Metric actually tells us how good it is. Imagine, we found the regression line (that has the least minimum squared error). Is that a good enough solution? This is what the metric will answer(considering the shape and spread of data, ideally!).
It is not yet clear for me what metrics are (as given in the code below). What exactly are they evaluating? Why do we need to define them in the model? Why we can have multiple metrics in one model? And more importantly what is the mechanics behind all this?
Any scientific reference is also appreciated.
model.compile(loss='mean_squared_error',
optimizer='sgd',
metrics=['mae', 'acc'])
As in keras metrics page described:
A metric is a function that is used to judge the performance of your
model
Metrics are frequently used with early stopping callback to terminate training and avoid overfitting
Reference: Keras Metrics Documentation
As given in the documentation page of keras metrics, a metric judges the performance of your model. The metrics argument in the compile method holds the list of metrics that needs to be evaluated by the model during its training and testing phases.
Metrics like:
-
binary_accuracy -
categorical_accuracy -
sparse_categorical_accuracy -
top_k_categorical_accuracyand -
sparse_top_k_categorical_accuracy
are the available metric functions that are supplied in the metrics parameter when the model is compiled.
Metric functions are customizable as well. When multiple metrics need to be evaluated it is passed in the form of a dictionary or a list.
One important resource you should refer for diving deep into metrics can be found here
So in order to understand what metrics are, it’s good to start by understanding what a loss function is. Neural networks are mostly trained using gradient methods by an iterative process of decreasing a loss function.
A loss is designed to have two crucial properties – first, the smaller its value is, the better your model fits your data, and second, it should be differentiable. So, knowing this, we could fully define what a metric is: it’s a function that, given predicted values and ground truth values from examples, provides you with a scalar measure of a “fitness” of your model, to the data you have. So, as you may see, a loss function is a metric, but the opposite doesn’t always hold. To understand these differences, let’s look at the most common examples of metrics usage:
-
Measure a performance of your network using non-differentiable functions: e.g. accuracy is not differentiable (not even continuous) so you cannot directly optimize your network w.r.t. to it. However, you could use it in order to choose the model with the best accuracy.
-
Obtain values of different loss functions when your final loss is a combination of a few of them: Let’s assume that your loss has a regularization term which measures how your weights differ from
0, and a term which measures the fitness of your model. In this case, you could usemetricsin order to have a separate track of how the fitness of your model changes across epochs. -
Track a measure with respect to which you don’t want to directly optimize your model: so – let’s assume that you are solving a multidimensional regression problem where you are mostly concerned about
mse, but at the same time you are interested in how acosine-distanceof your solution is changing in time. Then, it’s the best to usemetrics.
I hope that the explanation presented above made obvious what metrics are used for, and why you could use multiple metrics in one model. So now, let’s say a few words about mechanics of their usage in keras. There are two ways of computing them while training:
-
Using
metricsdefined while compilation: this is what you directly asked. In this case,kerasis defining a separate tensor for each metric you defined, to have it computed while training. This usually makes computation faster, but this comes at a cost of additional compilations, and the fact that metrics should be defined in terms ofkeras.backendfunctions. -
Using
keras.callback: It is nice that you can useCallbacksin order to compute your metrics. As each callback has a default attribute ofmodel, you could compute a variety of metrics usingmodel.predictor model parameters while training. Moreover, it makes it possible to compute it, not only epoch-wise, but also batch-wise, or training-wise. This comes at a cost of slower computations, and more complicated logic – as you need to define metrics on your own.
Here you can find a list of available metrics, as well as an example on how you could define your own.
From an implementation point of view, losses and metrics are actually identical functions in Keras:
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 23:09:28) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow.keras as Keras
>>> print(Keras.losses.mean_squared_error == Keras.metrics.mean_squared_error)
True
>>> print(Keras.losses.poisson == Keras.metrics.poisson)
True
Loss helps find the best solution your model can produce. Metric actually tells us how good it is. Imagine, we found the regression line (that has the least minimum squared error). Is that a good enough solution? This is what the metric will answer(considering the shape and spread of data, ideally!).