Why does calling the KFold generator with shuffle give the same indices?

Question

With sklearn, when you create a new KFold object and shuffle is true, it’ll produce a different, newly randomized fold indices. However, every generator from a given KFold object gives the same indices for each fold even when shuffle is true. Why does it work like this?

Example:

from sklearn.cross_validation import KFold
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4])
kf = KFold(4, n_folds=2, shuffle = True)

for fold in kf:
    print fold

print '---second round----'

for fold in kf:
    print fold

Output:

(array([2, 3]), array([0, 1]))
(array([0, 1]), array([2, 3]))
---second round----#same indices for the folds
(array([2, 3]), array([0, 1]))
(array([0, 1]), array([2, 3]))

This question was motivated by a comment on this answer. I decided to split it into a new question to prevent that answer from becoming too long.

Asked By: ilyas patanam

||

Source

Answer 1

A new iteration with the same KFold object will not reshuffle the indices, that only happens during instantiation of the object. KFold() never sees the data but knows number of samples so it uses that to shuffle the indices. From the code during instantiation of KFold:

if shuffle:
    rng = check_random_state(self.random_state)
    rng.shuffle(self.idxs)

Each time a generator is called to iterate through the indices of each fold, it will use same shuffled indices and divide them the same way.

Take a look at the code for the base class of KFold _PartitionIterator(with_metaclass(ABCMeta)) where __iter__ is defined. The __iter__ method in the base class calls _iter_test_indices in KFold to divide and yield the train and test indices for each fold.

Answered By: ilyas patanam

Answer 2

With new version of sklearn by calling from sklearn.model_selection import KFold, KFold generator with shuffle give the different indices:

import numpy as np
from sklearn.model_selection import KFold
X = np.array([[1, 2], [3, 4], [1, 2], [3, 4]])
y = np.array([1, 2, 3, 4])
kf = KFold(n_splits=3, shuffle=True)

print('---first round----')
for train_index, test_index in kf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    
print('---second round----')
for train_index, test_index in kf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)

Out:

---first round----
TRAIN: [2 3] TEST: [0 1]
TRAIN: [0 1 3] TEST: [2]
TRAIN: [0 1 2] TEST: [3]
---second round----
TRAIN: [0 1] TEST: [2 3]
TRAIN: [1 2 3] TEST: [0]
TRAIN: [0 2 3] TEST: [1]

Alternatively, the code below iteratively generates same result:

from sklearn.model_selection import KFold
np.random.seed(42)
data = np.random.choice([0, 1], 10, p=[0.5, 0.5])
kf = KFold(2, shuffle=True, random_state=2022)
list(kf.split(data))

Out:

[(array([0, 1, 6, 8, 9]), array([2, 3, 4, 5, 7])),
 (array([2, 3, 4, 5, 7]), array([0, 1, 6, 8, 9]))]

Answered By: ah bon

Why does calling the KFold generator with shuffle give the same indices?

Question:

Answers: