Large Numpy array causses Error when trying to load

Question

Edit : For those looking for answers, my file was corrupted hpauli suggested, knowing the shape the array should have, I opened the file using open(filenam) as append mode and appended 0 until there was a correct amount of data, then I split the files in 2 and loaded the first part, then splitted the second half and splitted itself into 2 more parts, etc… until I had recovered most of my data.

I’m making an ai using image recognition, so I recorded each frames of me playing into a numpy array. It worked just fine when the first time I exported all the images and got the 6 thousand of them. Now, I was recording a lot more data but suddently I get this erro with no change in my code or environement

Traceback (most recent call last):

File "D:DevFall-Guys-AI-RaceutilsCreateImages.py", line 6, in
>
data = np.load("D:/Dev/Fall-Guys-AI-Race/data/training_data.npy", allow_pickle=True)

File "D:Program FilesPython39libsite-packagesnumpylibnpyio.py", line 430, in load
>
return format.read_array(fid, allow_pickle=allow_pickle,

File "D:Program FilesPython39libsite-packagesnumpylibformat.py", line 786, in read_array
>
array.shape = shape

ValueError: cannot reshape array of size 2147483648 into shape (14460,224,224,3)

Here is my CreateImages.py :

import cv2, os
import numpy as np

listing = os.listdir("D:/Dev/Fall-Guys-AI-Race/data/")
for j in range(1):
    data = np.load("D:/Dev/Fall-Guys-AI-Race/data/training_data.npy", allow_pickle=True)
    targets = np.load("D:/Dev/Fall-Guys-AI-Race/data/target_data.npy", allow_pickle=True)

    print(f'Image Data Shape: {data.shape}')
    print(f'targets Shape: {targets.shape}')

    # Lets see how many of each type of move we have.
    unique_elements, counts = np.unique(targets, return_counts=True)

    # Store both data and targets in a list.
    # We may want to shuffle down the road.

    holder_list = []
    for i, image in enumerate(data):
        holder_list.append([data[i], targets[i]])

    count_up = 0
    count_left = 0
    count_right = 0
    count_jump = 0
    count_down = 0

    for data in holder_list:
        #writes data to image in correct folder, skipped because lots of lines:
        cv2.imwrite(f"*my_path*{count_left}.png", data[0])
    print("done")
    print(count_down, count_up, count_jump, count_left, count_right)

Thanks for the help

Edit: I cant even load the array (which is stored as a file) so i don’t think I can modify it

Asked By: AngelFire

||

Source

Answer 1

It appears that you are attempting to load numpy arrays, and the new arrays are larger than the previously loaded arrays.

The error notice informs you that the array size you are attempting to load cannot be moulded into the appropriate shape.

This might occur because the data size has expanded and the array cannot fit in memory, resulting in the error.

To fix this, consider

saving the data in smaller pieces or splitting the data into
different files and loading them independently.

Answered By: fatima raza

Large Numpy array causses Error when trying to load

Question:

Answers: