How to read large sample of images effectively without overloading ram memory?

I would load the first half of the dataset and then train the model on the first half of the dataset and then I would load the 2nd half and train the model on the 2nd part of the dataset. This does not influence the result.

The easiest way to split your dataset is to simply make a 2nd folder with the same structure with 50% of the dataset.

The pseudo-code for that method of training would look like this.:

1. load dataset 1
2. train the model with dataset 1
3. load dataset 2 but into the same variable as the first one to reuse the memory instead of creating a 2nd variable with the first one still in memory
4. train the model with dataset 2

A second option to decrease the memory size of your array you could use np.float16 instead of np.float32 but this would result in a more inaccurate model. The difference is data-dependent. So it could be 1-2% or even 5-10% so the only option without losing any accuracy is the option is described above.

EDIT

I am going to add the actual code.

import cv2 #pip install opencv-python
import os 

part1_of_dataset = os.listdir("Path_to_your_first_dataset")
part2_of_dataset = os.listdir("Path_to_your_second_dataset")

x_train = np.array([cv2.imread(image) for image in part1_of_dataset],dtype=np.float32)
x_train /=255.0
x_train_m = np.mean(x_train,axis=0)
x_train -= x_train_m

model.fit(x_train,y_train) #not the full training code just an example


x_train = np.array([cv2.imread(image) for image in part2_of_dataset],dtype=np.float32)
x_train /=255.0
x_train_m = np.mean(x_train,axis=0)
x_train -= x_train_m

model.fit(x_train,y_train) #not the full training code just an example