How can I efficiently filter and store maximum and minimum x,y pairs from a large list in Python?

Question

I have a loop function to calculate 2 values of the combinations of one list:

xyList = []
for i, j in itertools.combinations(mx, 2):
    x= *some calculation*
    y= *some calculation*
    xyList.append([x,y])

However, the combinations of the list are too many, there is always a MemoryError. Therefore, I tried to only keep the unique value:

    p=[x,y]
        if not (p in xyList):
            xyList.append(p)

But there is still a MemoryError. Now I’m thinking of only keeping the maximum y and minimum y for each x, i.e., the outline of this x, y scattered shape. But I cannot come up with an efficient way to do that. What is the possible solution?

Asked By: River

||

Source

Answer 1

You should probably make xyList a dictionary instead. Let’s say we call this xy_limits. Each value of xy_limits can be a list which keeps track of the minimum and maximum as its elements. When you encounter a y value that is outside this range, you can replace the correct value. Consider:

xy_limits = {}
for i, j in itertools.combinations(...):
    ...
    if x not in xy_limits:
        xy_limits[x] = [y, y]
        continue

    old_limits = xy_limits[x]
    if y < old_limits[0]:
        old_limits[0] = y
    elif y > old_limits[1]:
        old_limits[1] = y

Since old_limits is a list (a mutable type) modifications to that variable (e.g. by assigning one of its elements) will be seen in all occurrences of the list, such as that in the original xy_limits

Answered By: Pranav Hosangadi

How can I efficiently filter and store maximum and minimum x,y pairs from a large list in Python?

Question:

Answers: