How does a multi-class SVC with ovo models make a prediction if there is a "tie" in votes?
Question:
Let’s say I have a multi-class set of data with 4 labels: [A, B, C, D] that I want to use an ovo SVC classifier to predict. I know that once I train, I’ll have 6 ovo models trained and can use
svc.decision_function() to see the set of decision function outputs. Those functions are organized as
[AB, AC, AD, BC, BD, CD]
with the first letter being the positive case, and the second letter being the negative case. As I understand it, prediction takes these decision function scores and produces a set of "votes", the mode of which is the final prediction. Something like:
[AB, AC, AD, BC, BD, CD] #Model Organization
[1.4, 0.2, 0.6, 3, -2.2, -0.4] #Decision function output
[A, A, A, B, D, D] #Prediction for each model
The mode is A, so the svc.predict() function should output A. My question is what if the decision function output looks like this instead (I just flipped the sign of the 2nd entry but pretend its a new output):
[AB, AC, AD, BC, BD, CD] #Model Organization
[1.4, -0.2, 0.5, 3, -2.2, -0.4] #Decision function output
[A, C, A, B, D, D] #Prediction for each model
Now both A or D are the mode, so how would SVC break the tie and make the prediction? Speculatively I would sum the values of the decision function output (wrt the direction of the label) for each label, since the distance from the decision surface indicates to me how decisively the model predicted the label in each case. That would make D in the example. But I could also see using the specific score for the "AD" model. Regardless, not sure how SVC does it.
Answers:
It breaks ties for the lowest-value class. It’s hard to track that down, because it appears to be happening at the level of libsvm rather than at the sklearn level. Luckily, an example arising from an earlier issue provides a dataset where we can see the action. Here’s the original (the plot looks the same after changing decision_function_shape to ovo):
The middle triangular region gets classified along with the lower-left region, which is class 0. The decisions of the ovo models are tied with one vote for each class in that region.
Swapping the names of class 0 and 2 (e.g. by taking y=2-y) produces:
The triangular region is now classified along with the top-left region, which is newly named as zero.
Let’s say I have a multi-class set of data with 4 labels: [A, B, C, D] that I want to use an ovo SVC classifier to predict. I know that once I train, I’ll have 6 ovo models trained and can use
svc.decision_function() to see the set of decision function outputs. Those functions are organized as
[AB, AC, AD, BC, BD, CD]
with the first letter being the positive case, and the second letter being the negative case. As I understand it, prediction takes these decision function scores and produces a set of "votes", the mode of which is the final prediction. Something like:
[AB, AC, AD, BC, BD, CD] #Model Organization
[1.4, 0.2, 0.6, 3, -2.2, -0.4] #Decision function output
[A, A, A, B, D, D] #Prediction for each model
The mode is A, so the svc.predict() function should output A. My question is what if the decision function output looks like this instead (I just flipped the sign of the 2nd entry but pretend its a new output):
[AB, AC, AD, BC, BD, CD] #Model Organization
[1.4, -0.2, 0.5, 3, -2.2, -0.4] #Decision function output
[A, C, A, B, D, D] #Prediction for each model
Now both A or D are the mode, so how would SVC break the tie and make the prediction? Speculatively I would sum the values of the decision function output (wrt the direction of the label) for each label, since the distance from the decision surface indicates to me how decisively the model predicted the label in each case. That would make D in the example. But I could also see using the specific score for the "AD" model. Regardless, not sure how SVC does it.
It breaks ties for the lowest-value class. It’s hard to track that down, because it appears to be happening at the level of libsvm rather than at the sklearn level. Luckily, an example arising from an earlier issue provides a dataset where we can see the action. Here’s the original (the plot looks the same after changing decision_function_shape to ovo):
The middle triangular region gets classified along with the lower-left region, which is class 0. The decisions of the ovo models are tied with one vote for each class in that region.
Swapping the names of class 0 and 2 (e.g. by taking y=2-y) produces:
The triangular region is now classified along with the top-left region, which is newly named as zero.