UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples

As the error message states, the method used to get the F score is from the “Classification” part of sklearn – thus the talking about “labels”.

Do you have a regression problem? Sklearn provides a “F score” method for regression under the “feature selection” group: http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html

In case you do have a classification problem, @Shovalt’s answer seems correct to me.

The accepted answer explains already well why the warning occurs. If you simply want to control the warnings, one could use precision_recall_fscore_support. It offers a (semi-official) argument warn_for that could be used to mute the warnings.

(_, _, f1, _) = metrics.precision_recall_fscore_support(y_test, y_pred,
                                                        average='weighted', 
                                                        warn_for=tuple())

As mentioned already in some comments, use this with care.

the same problem also happened to me when i training my classification model.
the reason caused this problem is as what the warning message said “in labels with no predicated samples”, it will caused the zero-division when compute f1-score.
I found another solution when i read sklearn.metrics.f1_score doc, there is a note as follows:

When true positive + false positive == 0, precision is undefined; When true positive + false negative == 0, recall is undefined. In such cases, by default the metric will be set to 0, as will f-score, and UndefinedMetricWarning will be raised. This behavior can be modified with zero_division

the zero_division default value is "warn", you could set it to 0 or 1 to avoid UndefinedMetricWarning.
it works for me 😉 oh wait, there is another problem when i using zero_division, my sklearn report that no such keyword argument by using scikit-learn 0.21.3. Just update your sklearn to the latest version by running pip install scikit-learn -U

As I have noticed this error occurs under two circumstances,

1. If you have used train_test_split() to split your data, you have to make sure that you reset the index of the data (specially when taken using a pandas series object): y_train, y_test indices should be resetted. The problem is when you try to use one of the scores from sklearn.metrics such as; precision_score, this will try to match the shuffled indices of the y_test that you got from train_test_split().

so, either use np.array(y_test) for y_true in scores or y_test.reset_index(drop=True)

2. Then again you can still have this error if your predicted ‘True Positives’ is 0, which is used for precision, recall and f1_scores. You can visualize this using a confusion_matrix. If the classification is multilabel and you set param: average=’weighted’/micro/macro you will get an answer as long as the diagonal line in the matrix is not 0

Hope this helps.

I ended up here with the same error but after reading @Shovalt’s answer, I realized I was quite low in my test/train split. I had a large data set to start with but had split it down and one group was quite small. By making the sample size bigger, this warning went away and I got my f1 score.
From this

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.1, random_state=0)

to this

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

I checked, as Shovalt suggested, the difference between the sets of truth values and predictions in a multilabel case but it did not help me to solve my problem.

So, I searched into sklearn.metrics.precision_recall_fscore_support source code (which is called by f1_score) to check how it works.

The code triggering the warning is the following :

precision = _prf_divide(
    tp_sum, pred_sum, "precision", "predicted", average, warn_for, zero_division
)
recall = _prf_divide(
    tp_sum, true_sum, "recall", "true", average, warn_for, zero_division
)

• tpsum corresponds to TP (True Positives)
• pred_sum corresponds to TP + FP (False Positives)
• true_sum corresponds to TP + FN (False Negatives)
• first parameter of _prf_divide is numerator of division
• second parameter of _prf_divide is denominator of division

As soon as pred_sum or true_sum is equal to 0, it triggers the warning because division by 0 is not allowed.

In order to get these different values, use sklearn.metrics.multilabel_confusion_matrix. The result is a 3-dimensional array. You can see it as a list of 2×2 matrixes where each matrix represents the True Negatives (TN), False Positives (FP), False Negatives (FP) and the True Positives (TP) for each of your labels, structured as follow :

In my opinion, the problem should come from the model unability to predict some labels due to bad training or lack of samples.

This command works for me

sklearn.metrics.f1_score(y_true, y_pred,average='weighted',zero_division=0)