How can I optimize this code to handle large values?

Question

I was working on a project, and I noticed that this for loop takes a very long time to run.

d = [0, 1, 2, 5, ..., 0, 0] # Max array size: 100000 (10^5)
for i in range(len(d)): 
  sums = sum(d[i:]) * (i + 1)
  if sums > max_sum:
    max_sum = sums
    max_idx = i + 1

Is there a way to optimize it so that it can handle large values like 10^5 as the value of len(d)?

Thanks!

Asked By: VMSM

||

Source

Answer 1

Your code exhibits quadratic behavior because you are repeatedly recomputing the same sums in your loop: sum(d[i:]) == d[i] + sum(d[i+1:]).

Start by computing the sum of d[0:] == d, then subtract each item from that sum as you iterate.

subsum = sum(d)
for i, item in enumerate(d):
    sums = subsum * (i+1)
    subsum -= item
    if sums > max_sum:
        max_sum = sums
        max_idx = i + 1

Answered By: chepner

Answer 2

One way to optimize this code would be to use the prefix sum technique.

First, we can create a list called prefix_sum which will store the cumulative sum of the elements in d up to each index. We can do this by iterating through d and adding the previous cumulative sum to the current element.

Then, in the for loop, instead of calculating the sum of d[i:] using the built-in sum() function, we can use the prefix_sum list to get the sum of d[i:] in constant time by subtracting prefix_sum[i-1] from prefix_sum[-1] (assuming i is not the first element of d).

Here is the optimized code:

import numpy as np
from collections import deque

d = np.arange(100000)
d = d.tolist()

# Calculate prefix sum
prefix_sum = [0] * (len(d) + 1)
for i in range(1, len(d) + 1):
    prefix_sum[i] = prefix_sum[i - 1] + d[i - 1]

# Initialize the rolling sum and the deque
max_sum = 0
max_idx = 1
rolling_sum = sum(d)
subarray = deque(d)

for i in range(1, len(d)):
    # Update the rolling sum by subtracting the element at the current index
    # and adding the element at the next index
    rolling_sum = rolling_sum - d[i - 1] + d[i]

    # Update the deque by removing the first element and adding the next element
    subarray.popleft()
    subarray.append(d[i])

# Use the prefix sum array to compute the sum of the subarray in constant time
sums = (prefix_sum[len(d)] - prefix_sum[i]) * (i + 1)

# Update the maximum sum and index if necessary
if sums > max_sum:
    max_sum = sums
    max_idx = i + 1

# The maximum sum and index can now be accessed using the variables `max_sum` and `max_idx`

Answered By: Vatsal Patil

How can I optimize this code to handle large values?

Question:

Answers: