Split array of integers into subarrays with the biggest sum of difference between min and max
Question:
I’m trying to find the algorithm efficiently solving this problem:
Given an unsorted array of numbers, you need to divide it into several subarrays of length from a to b, so that the sum of differences between the minimum and maximum numbers in each of the subarrays is the greatest. The order of the numbers must be preserved.
Examples:
a = 3, b = 7
input: [5, 8, 4, 5, 1, 3, 5, 1, 3, 1]
answer: [[5, 8, 4], [5, 1, 3], [5, 1, 3, 1]] (diff sum is 12)
a = 3, b = 4
input: [1, 6, 2, 2, 5, 2, 8, 1, 5, 6]
answer: [[1, 6, 2], [2, 5, 2, 8], [1, 5, 6]] (diff sum is 16)
a = 4, b = 5
input: [5, 8, 4, 5, 1, 3, 5, 1, 3, 1, 2]
answer: splitting is impossible
The only solution I’ve come up with so far is trying all of the possible subarray combinations.
from collections import deque
def partition_array(numbers, min_len, max_len):
max_diff_subarray = None
queue = deque()
for end in range(min_len - 1, max_len):
if end < len(numbers):
diff = max(numbers[0:end + 1]) - min(numbers[0:end + 1])
queue.append(Subarray(previous=None, start=0, end=end, diff_sum=diff))
while queue:
subarray = queue.popleft()
if subarray.end == len(numbers) - 1:
if max_diff_subarray is None:
max_diff_subarray = subarray
elif max_diff_subarray.diff_sum < subarray.diff_sum:
max_diff_subarray = subarray
continue
start = subarray.end + 1
for end in range(start + min_len - 1, start + max_len):
if end < len(numbers):
diff = max(numbers[start:end + 1]) - min(numbers[start:end + 1])
queue.append(Subarray(previous=subarray, start=start, end=end, diff_sum=subarray.diff_sum + diff))
else:
break
return max_diff_subarray
class Subarray:
def __init__(self, previous=None, start=0, end=0, diff_sum=0):
self.previous = previous
self.start = start
self.end = end
self.diff_sum = diff_sum
numbers = [5, 8, 4, 5, 1, 3, 5, 1, 3, 1]
a = 3
b = 7
result = partition_array(numbers, a, b)
print(result.diff_sum)
Are there any more time efficient solutions?
Answers:
First let’s solve a simpler problem. Let’s run through an array, and give mins and maxes for all windows of fixed size.
def window_mins_maxes (size, array):
min_values = deque()
min_positions = deque()
max_values = deque()
max_positions = deque()
for i, value in enumerate(array):
if size <= i:
yield (i, min_values[0], max_values[0])
if min_positions[0] <= i - size:
min_values.popleft()
min_positions.popleft()
if max_positions[0] <= i - size:
max_values.popleft()
max_positions.popleft()
while 0 < len(min_values) and value <= min_values[-1]:
min_values.pop()
min_positions.pop()
min_values.append(value)
min_positions.append(i)
while 0 < len(max_values) and max_values[-1] <= value:
max_values.pop()
max_positions.pop()
max_values.append(value)
max_positions.append(i)
yield (len(array), min_values[0], max_values[0])
This clearly takes memory O(size). What’s less obvious is that it takes time O(n) to process an array of length n. But we can see that with amortized analysis. To each element we’ll attribute the cost of checking the possible value that is smaller than it, the cost of some later element checking that it should be removed, and the cost of being added. That accounts for all operations (though this isn’t the order that they happen) and is a fixed amount of work per element.
Also note that the memory needed for this part of the solution fits within O(n).
So far I’d consider this a well-known dynamic programming problem. Now let’s make it more challenging.
We will tackle the partition problem as a traditional dynamic programming problem. We’ll build up an array best_weight of the best partition to that point, and prev_index of the start of the previous partition ending just before that point.
To build it up, we’ll use the above algorithm to take a previous partition and add one of min_len to it. If it is better than the previous, we’ll save its information in those arrays. We’ll then scan forward from that partition and do that up to max_len. Then we move on to the next possible start of a partition.
When we’re done we’ll find the answer from that code.
Here is what that looks like:
def partition_array(numbers, min_len, max_len):
if max_len < min_len or len(numbers) < min_len:
return (None, None)
best_weight = [None for _ in numbers]
prev_index = [None for _ in numbers]
# Need an extra entry for off of the end of the array.
best_weight.append(None)
prev_index.append(None)
best_weight[0] = 0
for i, min_value, max_value in window_mins_maxes(min_len, numbers):
window_start_weight = best_weight[i - min_len]
if window_start_weight is not None:
j = i
while j - i < max_len - min_len and j < len(numbers):
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - min_len
if numbers[j] < min_value:
min_value = numbers[j]
if max_value < numbers[j]:
max_value = numbers[j]
j += 1
# And fill in the longest value.
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - min_len
if best_weight[-1] is None:
return (None, None)
else:
path = [len(numbers)]
while prev_index[path[-1]] is not None:
path.append(prev_index[path[-1]])
path = list(reversed(path))
partitioned = [numbers[path[i]:path[i+1]] for i in range(len(path)-1)]
return (best_weight[-1], partitioned)
Note that we do O(1) work for each possible start and length. And so that is time O((max_len + 1 - min_len)*n). And the data structures we used are all bounded above by O(n) in size. Giving the overall efficiency that I promised in the comments.
Now let’s test it.
print(partition_array([5, 8, 4, 5, 1, 3, 5, 1, 3, 1], 3, 7))
print(partition_array([1, 6, 2, 2, 5, 2, 8, 1, 5, 6], 3, 4))
print(partition_array([5, 8, 4, 5, 1, 3, 5, 1, 3, 1, 2], 4, 5))
And the output is:
(12, [[5, 8, 4], [5, 1, 3], [5, 1, 3, 1]])
(16, [[1, 6, 2], [2, 5, 2, 8], [1, 5, 6]])
(None, None)
Are there any more time efficient solutions?
Yes, there are.
The answer by btilly demonstrates that the algorithm presented in the code of the question which requires with array size exponential increasing time can be beaten by the dynamic programming approach with almost linear time dependency.
It is possible to do better than that? Is there still room for improvement?
Let’s start the journey with constructing a following test case where three different partitioning options deliver the same optimal result and the size of the list of integers allows reliable measurements of required computation time:
from time import perf_counter as T
from random import randint, seed
seed(2); arr=[]
for _ in range(100000): arr.append(randint(1,10000000))
sT=T(); result=partition_array(arr, 49999, 50001); eT=T()
print(eT-sT)
print("btl", result[0], result[1] , end='')
and run it using the code provided by btilly.
On my machine I am getting as output:
0.10844088008161634
btl 19998737 [0, 49999, 100000]
The btl ( shortcut for btilly ) code needs to arrive at a solution on my Linux Mint 21.2 Xfce Thinkpad Notebook running Python 3.10.12 about 0.1 seconds and uses the window_mins_maxes() function for the most work and therefore optimized it using a deque data type from the collections module. The code outputs the best sum and one of the possible multiple solutions for the partitioning of the list of integers.
Below another version of the window_mins_maxes() I named winMinMaxLargeSize() to distinguish it from much simpler winMinMax() code which works very fast on small integer lists, but gets slow on large ones:
def winMinMax(size, array):
for i in range(size, len(array)+1):
yield i, min(array[ i-size : i ]), max(array[ i -size : i])
def winMinMaxLargeSize(size, L):
""" Let's optimize min/max calculation by re-using already got values what starts to
make sense probably only at larger sizes (for example greater than 7).
This function moves a "Window" with width equal 'size' over the list 'L' calculating
the min/max values and remembering their list "L' indices. The indices are used then
to skip recalculating of min/max values in case the value "leaving" the "Window" has
no impact on the min/max values because in the range between min and max. The
variable names storing these indices are 'indxMin' and 'indxMax' """
# Lets calculate the min/max values for the first "Window" at the beginning of 'L' to
# have a reference start values for "moving" of the "Window" over the list 'L' :
prevMin, prevMax = min(L[0 : size ]), max(L[0: size])
lenL=len(L)
indxCounterMin=0; indxCounterMax=0
for i in range(size)[::-1] :
if indxCounterMin == 0 and L[i] == prevMin :
indxMin = i
indxCounterMin+= 1
if indxCounterMax == 0 and L[i] == prevMax :
indxMax = i
indxCounterMax+= 1
if indxCounterMax and indxCounterMin : # index positions of min/max are found
break # so we may exit the search loop .
#print(f"i={size=} , {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield size, prevMin, prevMax
for i in range(size+1, lenL+1):
if ( i - size - 1 ) == indxMin:
prevMin=min(L[ i - size : i ])
for indx in range( i - size, i )[::-1]:
if L[indx] == prevMin :
indxMin = indx
break
elif ( i - size - 1 ) == indxMax:
prevMax=max(L[ i - size : i ])
for indx in list(range(i - size , i ))[::-1]:
if L[indx] == prevMax :
indxMax = indx
break
else:
if L[ i - 1 ] <= prevMin :
indxMin = i - 1
if L[ i - 1 ] < prevMin :
prevMin = L[ i - 1 ]
elif L[ i - 1 ] >= prevMax :
indxMax = i - 1
if L[ i - 1 ] > prevMax :
prevMax = L[ i - 1 ]
#print(f" > {i=:2d}, {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield i, prevMin, prevMax
Replacing the function used in the "btl" code with the function above gives following output:
0.05169699003454298
btl-opt 19998737 [0, 49999, 100000]
Wow! With this "btl-opt" optimization of the code it was possible to cut the time down to a half and arrive at 0.05 seconds.
Is this now already the limit of what could be achieved, or is there still room for improvement?
Let’s leave the dynamic programming approach by side and use another one which targets the solution by "brute force" and checks all of the possible cases of partitioning the list. As a side-effect it spits out multiple solutions for the partitioning if there are any:
def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum(L, targetSum, n=None, m=None):
if n is None: n = 1
if m is None: m = targetSum
lenL = len(L)
# print(f"{targetSum=}, {L=}, {n=}, {m=}")
Stack = []
# Initialize the Stack with the starting point for each element in L
for i in range(lenL):
currentSum = L[ i ]
path = [ L[ i ] ]
start = 0 # Start from 0 allows revisiting of all items
Stack.append( (currentSum, path, start ) )
while Stack:
currentSum, path, start = Stack.pop()
# Check if the current path meets the criteria
if currentSum == targetSum and n <= len(path) <= m:
yield path
if currentSum > targetSum or len(path) > m:
continue
# ^ - NEXT please: stop exploring this path as it's not valid or complete
# Continue to build the path by adding elements from L, starting from 0 index
for i in range(len(L)): # Change start to 0 if you want to include all permutations
newSum = currentSum + L[ i ]
newPath = path + [ L[ i ] ]
Stack.append((newSum, newPath, 0)) # Start from 0 allows every possibility
# def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
splitsGenerator = allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
def arrSplit(Arr, m, n) :
targetSum=len(Arr)
L=list(range( m, n+1) )
bestSum = None
for split in splitsGenerator( L, targetSum ) :
#print("###", split)
startIndx = 0
endIndx = 0
currSum = 0
for indx in split :
endIndx += indx
currSum = currSum + max(Arr[startIndx : endIndx ]) - min(Arr[ startIndx : endIndx ])
startIndx=startIndx + endIndx
if bestSum is None:
bestSum = currSum
bestSplit = [ split ]
continue
if bestSum <= currSum :
if bestSum == currSum :
bestSplit.append(split)
else:
bestSum = currSum
bestSplit = [ split ]
return bestSum, bestSplit
targetSum = 10000
from time import perf_counter as T
from random import randint, seed
seed(2); arr=[]
for _ in range(100000): arr.append(randint(1,10000000))
sT=T(); result=arrSplit(arr, 49999, 50001); eT=T()
print(eT-sT)
print("oOo", result[0], result[1], end='')
The code outputs:
0.012507450999692082
oOo 19998737 [[50001, 49999], [50000, 50000], [49999, 50001]]
and beats the already very fast code we have started the journey with being ten times faster with 0.01 seconds.
Can this be still improved by a further order of magnitude? I am quite sure it can by more than that, so there is still plenty of room for improvement and for next answers with much faster code to continue the journey.
May the power of oOo be with you!
Just an alternative for btilly’s window_mins_maxes, also O(len(array)) time and O(size) space:
from itertools import accumulate
def window_mins_maxes(size, array):
leftmins = None
for stop in range(size, len(array) + 1):
if not leftmins:
left = array[stop-size : stop][::-1]
right = array[stop-1 : stop+size]
leftmins = list(accumulate(left, min))
leftmaxs = list(accumulate(left, max))
rightmins = accumulate(right, min)
rightmaxs = accumulate(right, max)
yield (
stop,
min(leftmins.pop(), next(rightmins)),
max(leftmaxs.pop(), next(rightmaxs))
)
The idea: If you want the minimum of some array range, split the range into a left part and a right part, and take the minimum of the left’s minimum and the right’s minimum.
For example for 10 elements and window size 4:
0 1 2 3 4 5 6 7 8 9
L L L L
R
L L L
R R
L L
R R R
L
R R R R
L L L L
R
...
The Ls mark the left part and the Rs mark the right part of the window. The right part is growing, so its minima are just its cumulative minima. The left part is shrinking, so its minima are it’s reversed cumulative suffix minima.
Whenever I don’t have left-part minima, I compute a chunk of them and reset the right part, as shown in the last step above.
I made L and R overlap so they’re never empty, not even when size=1.
While trying to find an algorithm for fast splitting an n -items long array of integers into sub-arrays of length between and including a and b with the biggest sum of differences between min and max values in each of the sub-arrays it is important to become aware of the fact that the complexity of the algorithm required to solve this problem highly depends on the parameters a and b which impact on the overall complexity is a way higher than the size of the array itself.
What is the time complexity of this problem expressed in terms of n, a and b and why is it important to be aware of it?
Becoming aware of it is important in order to see that as long as no algorithm allowing full linear time complexity becomes known here it will be necessary to use different algorithms to achieve good performance over various special ranges of possible values of a and b at given array size n.
If you have tried to use the algorithm provided by Claudio in the other answer (which claims to be ten times faster than the algorithm provided by btilly) with another values for a and b as these ones chosen by Claudio, you had already experienced its really bad performance compared to the dynamic programming approach in most of the other cases.
This is the point where knowing about the right formula for the time complexity of the problem helps to provide code which always runs faster.
Below the hopefully correct formula used in provided code for deciding about the choice of the appropriate algorithm to ensure best performance in all of the possible cases:
O( n * (b-a+1)**(n//a -1) ) ( ‘ ** ‘-> exponentiation ‘//’-> integer division )
In the code itself the (b-a+1)**(n//a) part of the formula is called complexityFactor and the threshold value used to choose one of the two provided algorithms for calculating the result is called complexityFactorThreshold
def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum(L, targetSum, n=None, m=None):
if n is None: n = 1
if m is None: m = targetSum
lenL = len(L)
# print(f"{targetSum=}, {L=}, {n=}, {m=}")
Stack = []
# Initialize the Stack with the starting point for each element in L
for i in range(lenL):
currentSum = L[ i ]
path = [ L[ i ] ]
start = 0 # Start from 0 allows revisiting of all items
Stack.append( (currentSum, path, start ) )
while Stack:
currentSum, path, start = Stack.pop()
# Check if the current path meets the criteria
if currentSum == targetSum and n <= len(path) <= m:
yield path
if currentSum > targetSum or len(path) > m:
continue
# ^ - NEXT please: stop exploring this path as it's not valid or complete
# Continue to build the path by adding elements from L, starting from 0 index
for i in range(len(L)): # Change start to 0 if you want to include all permutations
newSum = currentSum + L[ i ]
newPath = path + [ L[ i ] ]
Stack.append((newSum, newPath, 0)) # Start from 0 allows every possibility
# def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
splitsGenerator = allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
def lowComplexitySplit(Arr, m, n) :
targetSum=len(Arr)
L=list(range( m, n+1) )
bestSum = None
for split in splitsGenerator( L, targetSum ) :
#print("###", split)
startIndx = 0
endIndx = 0
currSum = 0
for indx in split :
endIndx += indx
currSum = currSum + max(Arr[startIndx : endIndx ]) - min(Arr[ startIndx : endIndx ])
startIndx=startIndx + endIndx
if bestSum is None:
bestSum = currSum
bestSplit = [ split ]
continue
if bestSum <= currSum :
if bestSum == currSum :
bestSplit.append(split)
else:
bestSum = currSum
bestSplit = [ split ]
return bestSum, bestSplit
def winMinMaxLargeSize(size, L):
""" oOosys 2024-04-23_21:30
This method of getting the min/max values within the over the list 'L' moved
chunk/interval/partition/section "Window" is saving the indices/positions of these
values in the list 'L' for later use. This allows skipping of recalculations of min/max
in cases where the integer value "leaving" the "Window" has no impact on the
min/max values because it fits into the range between min and max.
The variables storing the indices/positions are 'indxMin' and 'indxMax' and represent
the LAST position of the found min/max value if non-unique.
As Python suprisingly does not support simultaneous calculation of min/max and the
last position of obtained values some "trickery" is used to avoid some of the multiple
scans over the analysed "Window" in the process of getting both the min/max
values and their last indices/positions in the for them scanned list.
"""
# Lets calculate the min/max values for the first "Window" at the beginning of 'L' to
# get initial/start values for "moving" of the "Window" over the list 'L' :
prevMax, indxMax = max( zip(L[ 0 : size ], range( size )))
#prevMin, indxMin = min( zip(L[ 0 : size ], reversed(range( size )))); indxMin=size - indxMin - 1
prevMin=min(L[ 0 : size ]); indxMin = size - list(reversed(L[ 0 : size ])).index( prevMin ) - 1 # probably FASTER than in the line above
lenL=len(L)
#print(f"i={size=} , {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield size, prevMin, prevMax
for i in range(size+1, lenL+1):
if ( i - size - 1 ) == indxMin:
prevMin=min(L[ i - size : i ]); indxMin = i - list(reversed(L[ i - size : i ])).index( prevMin ) - 1 # appears to be the FASTEST ONE (see another approaches below)
# ---
#prevMin, indxMin = min( zip(L[ i - size : i ], range( size - 1, -1, -1 ))); indxMin = i - indxMin - 1
# ---
#prevMin, indxMin = min( zip(L[ i - size : i ], reversed( range( size )))); indxMin = i - indxMin - 1
# ---
#prevMin=min(L[ i - size : i ])
#for indx in range( i - size, i )[::-1]:
# if L[indx] == prevMin :
# indxMin = indx
# break
# ---
elif ( i - size - 1 ) == indxMax:
prevMax, indxMax = max( zip(L[ i - size : i ], range( size ))); indxMax = i - size + indxMax
#prevMax=max(L[ i - size : i ])
#for indx in list(range(i - size , i ))[::-1]:
# if L[indx] == prevMax :
# indxMax = indx
# break
else: # the value which has "left" the "Window" was a neutral one and it is possible to update values without re-calculating min/max over the "Window" width
if L[ i - 1 ] <= prevMin : # let's handle the case of necessary update of indxMin and eventually necessary update also of prevMin in one if block:
indxMin = i - 1
if L[ i - 1 ] < prevMin :
prevMin = L[ i - 1 ]
elif L[ i - 1 ] >= prevMax : # this is the case where the "incoming" value impacts indxMax and eventually the prevMax value
indxMax = i - 1
if L[ i - 1 ] > prevMax :
prevMax = L[ i - 1 ]
#print(f" > {i=:2d}, {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield i, prevMin, prevMax
def highComplexitySplit(listOfInts, minSplitWidth, maxSplitWidth):
if maxSplitWidth < minSplitWidth or len(listOfInts) < minSplitWidth:
return (None, None)
best_weight = [None for _ in listOfInts]
prev_index = [None for _ in listOfInts]
# Need an extra entry for off of the end of the array.
best_weight.append(None)
prev_index.append(None)
best_weight[0] = 0
for i, min_value, max_value in winMinMaxLargeSize(minSplitWidth, listOfInts):
#print(f" {i=:4d}, {min_value=:8d}, {max_value=:8d}", end=" >>> ")
window_start_weight = best_weight[i - minSplitWidth]
if window_start_weight is not None:
j = i
while j - i < maxSplitWidth - minSplitWidth and j < len(listOfInts):
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - minSplitWidth
if listOfInts[j] < min_value:
min_value = listOfInts[j]
if max_value < listOfInts[j]:
max_value = listOfInts[j]
j += 1
# And fill in the longest value.
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - minSplitWidth
if best_weight[-1] is None:
return (None, None)
else:
path = [len(listOfInts)]
while prev_index[path[-1]] is not None:
path.append(prev_index[path[-1]])
#path = list(reversed(path))
path = list(path)[::-1]
#partitioned = [listOfInts[path[i]:path[i+1]] for i in range(len(path)-1)]
#return (best_weight[-1], partitioned)
return (best_weight[-1], path )
def complexityFactorSplit(listOfInts, minSplitWidth, maxSplitWidth) :
n=len(listOfInts)
complexityFactor=( maxSplitWidth - minSplitWidth + 1 )**( n//minSplitWidth - 1)
complexityFactorThreshold = 7
if complexityFactor <= complexityFactorThreshold :
return lowComplexitySplit( listOfInts, minSplitWidth, maxSplitWidth )
else :
return highComplexitySplit( listOfInts, minSplitWidth, maxSplitWidth )
# ---
#n=100000; a=49970; b=50030
#n=100000; a=50000; b=50000
#n=100000; a=49995; b=50005
from time import perf_counter as T
from random import randint, seed
seed(2); arr=[]; n=100000
for _ in range(n): arr.append(randint(1,n**2))
print(f"{n=}")
for step in range(0, 11, 1):
a=50000 - step
b=50000 + step
sT1=T(); lowComplexitySplit(arr, a, b); eT1=T()
sT2=T(); highComplexitySplit(arr, a, b); eT2=T()
print(f"a/b:{a}/{b} lowC: {eT1-sT1:.3f},highC: {eT2-sT2:.3f} low/high:{100*(eT1-sT1)/(eT2-sT2): 5.0f}% , cF={( b - a + 1 )**( n//a - 1):3d} , btilly O(): O( n *{b+1-a: 4d} )")
for step in range(15, 60, 5):
a=50000 - step
b=50000 + step
sT1=T(); lowComplexitySplit(arr, a, b); eT1=T()
sT2=T(); highComplexitySplit(arr, a, b); eT2=T()
print(f"a/b:{a}/{b} lowC: {eT1-sT1:.3f},highC: {eT2-sT2:.3f} low/high:{100*(eT1-sT1)/(eT2-sT2): 5.0f}% , cF={( b - a + 1 )**( n//a - 1):3d} , btilly O(): O( n *{b+1-a: 4d} )")
which outputs:
n=100000
a/b:50000/50000 lowC: 0.005,highC: 0.050 low/high: 11% , cF= 1 , btilly O(): O( n * 1 )
a/b:49999/50001 lowC: 0.023,highC: 0.048 low/high: 47% , cF= 3 , btilly O(): O( n * 3 )
a/b:49998/50002 lowC: 0.031,highC: 0.051 low/high: 60% , cF= 5 , btilly O(): O( n * 5 )
a/b:49997/50003 lowC: 0.042,highC: 0.048 low/high: 88% , cF= 7 , btilly O(): O( n * 7 )
a/b:49996/50004 lowC: 0.055,highC: 0.048 low/high: 115% , cF= 9 , btilly O(): O( n * 9 )
a/b:49995/50005 lowC: 0.065,highC: 0.046 low/high: 141% , cF= 11 , btilly O(): O( n * 11 )
a/b:49994/50006 lowC: 0.076,highC: 0.046 low/high: 166% , cF= 13 , btilly O(): O( n * 13 )
a/b:49993/50007 lowC: 0.090,highC: 0.046 low/high: 195% , cF= 15 , btilly O(): O( n * 15 )
a/b:49992/50008 lowC: 0.100,highC: 0.047 low/high: 212% , cF= 17 , btilly O(): O( n * 17 )
a/b:49991/50009 lowC: 0.108,highC: 0.046 low/high: 233% , cF= 19 , btilly O(): O( n * 19 )
a/b:49990/50010 lowC: 0.120,highC: 0.047 low/high: 254% , cF= 21 , btilly O(): O( n * 21 )
a/b:49985/50015 lowC: 0.175,highC: 0.046 low/high: 378% , cF= 31 , btilly O(): O( n * 31 )
a/b:49980/50020 lowC: 0.238,highC: 0.047 low/high: 510% , cF= 41 , btilly O(): O( n * 41 )
a/b:49975/50025 lowC: 0.297,highC: 0.048 low/high: 620% , cF= 51 , btilly O(): O( n * 51 )
a/b:49970/50030 lowC: 0.367,highC: 0.048 low/high: 769% , cF= 61 , btilly O(): O( n * 61 )
a/b:49965/50035 lowC: 0.445,highC: 0.048 low/high: 935% , cF= 71 , btilly O(): O( n * 71 )
a/b:49960/50040 lowC: 0.531,highC: 0.048 low/high: 1100% , cF= 81 , btilly O(): O( n * 81 )
a/b:49955/50045 lowC: 0.625,highC: 0.048 low/high: 1300% , cF= 91 , btilly O(): O( n * 91 )
a/b:49950/50050 lowC: 0.730,highC: 0.048 low/high: 1512% , cF=101 , btilly O(): O( n * 101 )
a/b:49945/50055 lowC: 0.841,highC: 0.049 low/high: 1722% , cF=111 , btilly O(): O( n * 111 )
The output shows that the threshold value for cF (complexity Factor) of 7 will in the case of the example allow a right decision about the algorithm to use while running the complexityFactorSplit() function which then delivers fast results for both, high and low complexity sets of values for a and b.
Notice in the output that the by btilly provided formula for the O() complexity suggests growing complexity, but the code running time (highC) does not increase what proves that this formula is incorrect for both: describing the behavior of the code and describing the complexity of the problem itself.
There are up to now ( 2024-04-24 ) following main algorithmic approaches formulated as Python scripts in the question and all the other answers:
- brute force approach which fails to handle arrays of length beyond 35 ( in OP question )
- a dynamic programming approach able to handle large arrays ( first given answer by btilly )
- a dynamic programming btilly code based approach with improved core function for finding the maxima and minima in all "windows" with width a sliding over the array of numbers running twice as fast as the original version
- a brute force approach generating in first step a splitting which runs on some cases of a and b values ten times faster then the fastest code in the already mentioned approaches
- a mixed approach achieving overall faster results by switching to the faster algorithm running two to ten times faster than the first by btilly posted code.
Is it possible to provide an algorithm which runs faster than all of the already provided ones without any tricks with making the code as such running faster?
This answer shows that it is possible covering a much wider range of cases as the up to now fastest algorithm suitable for a small range of b and a values for given n being the size of the array with numbers.
Compared to the first posted fast dynamic programming based algorithm the below provided code runs 60 times faster and could probably not be improved as an algorithm (except there is some relationship valid because of the specifics of the weight function being the difference of max-min values within the chunk/partition/sub-list), so speeding it up will need faster running code implementing the algorithm.
Below the code showing the basic idea behind the algorithm rendering all the previous considerations about complexity of the problem obsolete.
Using this basic idea will make it with some further work on it possible to provide code covering all possible cases by hierarchical building upon this core "brick like" unit. It will then handle all of possible parameter sets for n, a, b and not only these which satisfy the condition n//a == 2 ( where // means integer division ) leading to better performance than the dynamic programming based approach and capable to provide multiple solutions for splittings if there are any.
def oneSplitPoint(Arr,a,b):
lenArr=len(Arr)
assert lenArr//a == 2
minL, maxL =min(Arr[:a]), max(Arr[:a])
minR, maxR =min(Arr[-a:]), max(Arr[-a:])
dL =[ maxL -minL ]
dR =[ maxR - minR ]
if a == b and a+b == n:
return ( dL[0]+dR[0], f"[ {a} ]" )
minSplitWidth = a ; maxSplitWidth = b
for indxL in range( minSplitWidth, maxSplitWidth ):
if Arr[indxL] > maxL : maxL = Arr[indxL]
if Arr[indxL] < minL : minL = Arr[indxL]
dL = dL + [ maxL -minL ]
indxR = lenArr - indxL - 1
if Arr[indxR] > maxR : maxR = Arr[indxR]
if Arr[indxR] < minR : minR = Arr[indxR]
dR = dR + [ maxR -minR ]
dR.reverse()
sumLR = []
for indx in range(maxSplitWidth - minSplitWidth + 1) :
sumLR = sumLR + [ dL[indx] + dR[indx] ]
maxSumLR = max( sumLR )
splitIndices = []; lastIndex = 0
while True:
try:
lastIndex = sumLR.index( maxSumLR , lastIndex ) + 1
splitIndices = splitIndices + [ minSplitWidth + lastIndex - 2 ]
except ValueError:
break
return maxSumLR, splitIndices
Below the output from running the above code against the improved dynamic programming algorithm approach showing cutting down of the time to 3% of the reference code. And because the reference code runs twice as fast as the first one posted it means an improvement of being over 60 times faster.
n=12000
a/b:6000/6000 lowC: 0.000,highC: 0.005 low/high: 11% , cF= 0 , btilly O(): O( n * 1 )
a/b:5800/6200 lowC: 0.002,highC: 0.040 low/high: 4% , cF= 1 , btilly O(): O( n * 401 )
a/b:5600/6400 lowC: 0.004,highC: 0.137 low/high: 3% , cF= 4 , btilly O(): O( n * 801 )
a/b:5400/6600 lowC: 0.009,highC: 0.301 low/high: 3% , cF= 9 , btilly O(): O( n * 1201 )
a/b:5200/6800 lowC: 0.017,highC: 0.530 low/high: 3% , cF= 16 , btilly O(): O( n * 1601 )
a/b:5000/7000 lowC: 0.025,highC: 0.828 low/high: 3% , cF= 25 , btilly O(): O( n * 2001 )
a/b:4800/7200 lowC: 0.039,highC: 1.188 low/high: 3% , cF= 36 , btilly O(): O( n * 2401 )
I have adjusted the complexity factor to a formula which mirrors the running time and it seems that the time complexity of this approach is rough estimated proportional to the square of (b-a+1) i.e. O(n(b-a+1)^2) .
I’m trying to find the algorithm efficiently solving this problem:
Given an unsorted array of numbers, you need to divide it into several subarrays of length from a to b, so that the sum of differences between the minimum and maximum numbers in each of the subarrays is the greatest. The order of the numbers must be preserved.
Examples:
a = 3, b = 7 input: [5, 8, 4, 5, 1, 3, 5, 1, 3, 1] answer: [[5, 8, 4], [5, 1, 3], [5, 1, 3, 1]] (diff sum is 12) a = 3, b = 4 input: [1, 6, 2, 2, 5, 2, 8, 1, 5, 6] answer: [[1, 6, 2], [2, 5, 2, 8], [1, 5, 6]] (diff sum is 16) a = 4, b = 5 input: [5, 8, 4, 5, 1, 3, 5, 1, 3, 1, 2] answer: splitting is impossible
The only solution I’ve come up with so far is trying all of the possible subarray combinations.
from collections import deque
def partition_array(numbers, min_len, max_len):
max_diff_subarray = None
queue = deque()
for end in range(min_len - 1, max_len):
if end < len(numbers):
diff = max(numbers[0:end + 1]) - min(numbers[0:end + 1])
queue.append(Subarray(previous=None, start=0, end=end, diff_sum=diff))
while queue:
subarray = queue.popleft()
if subarray.end == len(numbers) - 1:
if max_diff_subarray is None:
max_diff_subarray = subarray
elif max_diff_subarray.diff_sum < subarray.diff_sum:
max_diff_subarray = subarray
continue
start = subarray.end + 1
for end in range(start + min_len - 1, start + max_len):
if end < len(numbers):
diff = max(numbers[start:end + 1]) - min(numbers[start:end + 1])
queue.append(Subarray(previous=subarray, start=start, end=end, diff_sum=subarray.diff_sum + diff))
else:
break
return max_diff_subarray
class Subarray:
def __init__(self, previous=None, start=0, end=0, diff_sum=0):
self.previous = previous
self.start = start
self.end = end
self.diff_sum = diff_sum
numbers = [5, 8, 4, 5, 1, 3, 5, 1, 3, 1]
a = 3
b = 7
result = partition_array(numbers, a, b)
print(result.diff_sum)
Are there any more time efficient solutions?
First let’s solve a simpler problem. Let’s run through an array, and give mins and maxes for all windows of fixed size.
def window_mins_maxes (size, array):
min_values = deque()
min_positions = deque()
max_values = deque()
max_positions = deque()
for i, value in enumerate(array):
if size <= i:
yield (i, min_values[0], max_values[0])
if min_positions[0] <= i - size:
min_values.popleft()
min_positions.popleft()
if max_positions[0] <= i - size:
max_values.popleft()
max_positions.popleft()
while 0 < len(min_values) and value <= min_values[-1]:
min_values.pop()
min_positions.pop()
min_values.append(value)
min_positions.append(i)
while 0 < len(max_values) and max_values[-1] <= value:
max_values.pop()
max_positions.pop()
max_values.append(value)
max_positions.append(i)
yield (len(array), min_values[0], max_values[0])
This clearly takes memory O(size). What’s less obvious is that it takes time O(n) to process an array of length n. But we can see that with amortized analysis. To each element we’ll attribute the cost of checking the possible value that is smaller than it, the cost of some later element checking that it should be removed, and the cost of being added. That accounts for all operations (though this isn’t the order that they happen) and is a fixed amount of work per element.
Also note that the memory needed for this part of the solution fits within O(n).
So far I’d consider this a well-known dynamic programming problem. Now let’s make it more challenging.
We will tackle the partition problem as a traditional dynamic programming problem. We’ll build up an array best_weight of the best partition to that point, and prev_index of the start of the previous partition ending just before that point.
To build it up, we’ll use the above algorithm to take a previous partition and add one of min_len to it. If it is better than the previous, we’ll save its information in those arrays. We’ll then scan forward from that partition and do that up to max_len. Then we move on to the next possible start of a partition.
When we’re done we’ll find the answer from that code.
Here is what that looks like:
def partition_array(numbers, min_len, max_len):
if max_len < min_len or len(numbers) < min_len:
return (None, None)
best_weight = [None for _ in numbers]
prev_index = [None for _ in numbers]
# Need an extra entry for off of the end of the array.
best_weight.append(None)
prev_index.append(None)
best_weight[0] = 0
for i, min_value, max_value in window_mins_maxes(min_len, numbers):
window_start_weight = best_weight[i - min_len]
if window_start_weight is not None:
j = i
while j - i < max_len - min_len and j < len(numbers):
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - min_len
if numbers[j] < min_value:
min_value = numbers[j]
if max_value < numbers[j]:
max_value = numbers[j]
j += 1
# And fill in the longest value.
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - min_len
if best_weight[-1] is None:
return (None, None)
else:
path = [len(numbers)]
while prev_index[path[-1]] is not None:
path.append(prev_index[path[-1]])
path = list(reversed(path))
partitioned = [numbers[path[i]:path[i+1]] for i in range(len(path)-1)]
return (best_weight[-1], partitioned)
Note that we do O(1) work for each possible start and length. And so that is time O((max_len + 1 - min_len)*n). And the data structures we used are all bounded above by O(n) in size. Giving the overall efficiency that I promised in the comments.
Now let’s test it.
print(partition_array([5, 8, 4, 5, 1, 3, 5, 1, 3, 1], 3, 7))
print(partition_array([1, 6, 2, 2, 5, 2, 8, 1, 5, 6], 3, 4))
print(partition_array([5, 8, 4, 5, 1, 3, 5, 1, 3, 1, 2], 4, 5))
And the output is:
(12, [[5, 8, 4], [5, 1, 3], [5, 1, 3, 1]])
(16, [[1, 6, 2], [2, 5, 2, 8], [1, 5, 6]])
(None, None)
Are there any more time efficient solutions?
Yes, there are.
The answer by btilly demonstrates that the algorithm presented in the code of the question which requires with array size exponential increasing time can be beaten by the dynamic programming approach with almost linear time dependency.
It is possible to do better than that? Is there still room for improvement?
Let’s start the journey with constructing a following test case where three different partitioning options deliver the same optimal result and the size of the list of integers allows reliable measurements of required computation time:
from time import perf_counter as T
from random import randint, seed
seed(2); arr=[]
for _ in range(100000): arr.append(randint(1,10000000))
sT=T(); result=partition_array(arr, 49999, 50001); eT=T()
print(eT-sT)
print("btl", result[0], result[1] , end='')
and run it using the code provided by btilly.
On my machine I am getting as output:
0.10844088008161634
btl 19998737 [0, 49999, 100000]
The btl ( shortcut for btilly ) code needs to arrive at a solution on my Linux Mint 21.2 Xfce Thinkpad Notebook running Python 3.10.12 about 0.1 seconds and uses the window_mins_maxes() function for the most work and therefore optimized it using a deque data type from the collections module. The code outputs the best sum and one of the possible multiple solutions for the partitioning of the list of integers.
Below another version of the window_mins_maxes() I named winMinMaxLargeSize() to distinguish it from much simpler winMinMax() code which works very fast on small integer lists, but gets slow on large ones:
def winMinMax(size, array):
for i in range(size, len(array)+1):
yield i, min(array[ i-size : i ]), max(array[ i -size : i])
def winMinMaxLargeSize(size, L):
""" Let's optimize min/max calculation by re-using already got values what starts to
make sense probably only at larger sizes (for example greater than 7).
This function moves a "Window" with width equal 'size' over the list 'L' calculating
the min/max values and remembering their list "L' indices. The indices are used then
to skip recalculating of min/max values in case the value "leaving" the "Window" has
no impact on the min/max values because in the range between min and max. The
variable names storing these indices are 'indxMin' and 'indxMax' """
# Lets calculate the min/max values for the first "Window" at the beginning of 'L' to
# have a reference start values for "moving" of the "Window" over the list 'L' :
prevMin, prevMax = min(L[0 : size ]), max(L[0: size])
lenL=len(L)
indxCounterMin=0; indxCounterMax=0
for i in range(size)[::-1] :
if indxCounterMin == 0 and L[i] == prevMin :
indxMin = i
indxCounterMin+= 1
if indxCounterMax == 0 and L[i] == prevMax :
indxMax = i
indxCounterMax+= 1
if indxCounterMax and indxCounterMin : # index positions of min/max are found
break # so we may exit the search loop .
#print(f"i={size=} , {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield size, prevMin, prevMax
for i in range(size+1, lenL+1):
if ( i - size - 1 ) == indxMin:
prevMin=min(L[ i - size : i ])
for indx in range( i - size, i )[::-1]:
if L[indx] == prevMin :
indxMin = indx
break
elif ( i - size - 1 ) == indxMax:
prevMax=max(L[ i - size : i ])
for indx in list(range(i - size , i ))[::-1]:
if L[indx] == prevMax :
indxMax = indx
break
else:
if L[ i - 1 ] <= prevMin :
indxMin = i - 1
if L[ i - 1 ] < prevMin :
prevMin = L[ i - 1 ]
elif L[ i - 1 ] >= prevMax :
indxMax = i - 1
if L[ i - 1 ] > prevMax :
prevMax = L[ i - 1 ]
#print(f" > {i=:2d}, {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield i, prevMin, prevMax
Replacing the function used in the "btl" code with the function above gives following output:
0.05169699003454298
btl-opt 19998737 [0, 49999, 100000]
Wow! With this "btl-opt" optimization of the code it was possible to cut the time down to a half and arrive at 0.05 seconds.
Is this now already the limit of what could be achieved, or is there still room for improvement?
Let’s leave the dynamic programming approach by side and use another one which targets the solution by "brute force" and checks all of the possible cases of partitioning the list. As a side-effect it spits out multiple solutions for the partitioning if there are any:
def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum(L, targetSum, n=None, m=None):
if n is None: n = 1
if m is None: m = targetSum
lenL = len(L)
# print(f"{targetSum=}, {L=}, {n=}, {m=}")
Stack = []
# Initialize the Stack with the starting point for each element in L
for i in range(lenL):
currentSum = L[ i ]
path = [ L[ i ] ]
start = 0 # Start from 0 allows revisiting of all items
Stack.append( (currentSum, path, start ) )
while Stack:
currentSum, path, start = Stack.pop()
# Check if the current path meets the criteria
if currentSum == targetSum and n <= len(path) <= m:
yield path
if currentSum > targetSum or len(path) > m:
continue
# ^ - NEXT please: stop exploring this path as it's not valid or complete
# Continue to build the path by adding elements from L, starting from 0 index
for i in range(len(L)): # Change start to 0 if you want to include all permutations
newSum = currentSum + L[ i ]
newPath = path + [ L[ i ] ]
Stack.append((newSum, newPath, 0)) # Start from 0 allows every possibility
# def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
splitsGenerator = allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
def arrSplit(Arr, m, n) :
targetSum=len(Arr)
L=list(range( m, n+1) )
bestSum = None
for split in splitsGenerator( L, targetSum ) :
#print("###", split)
startIndx = 0
endIndx = 0
currSum = 0
for indx in split :
endIndx += indx
currSum = currSum + max(Arr[startIndx : endIndx ]) - min(Arr[ startIndx : endIndx ])
startIndx=startIndx + endIndx
if bestSum is None:
bestSum = currSum
bestSplit = [ split ]
continue
if bestSum <= currSum :
if bestSum == currSum :
bestSplit.append(split)
else:
bestSum = currSum
bestSplit = [ split ]
return bestSum, bestSplit
targetSum = 10000
from time import perf_counter as T
from random import randint, seed
seed(2); arr=[]
for _ in range(100000): arr.append(randint(1,10000000))
sT=T(); result=arrSplit(arr, 49999, 50001); eT=T()
print(eT-sT)
print("oOo", result[0], result[1], end='')
The code outputs:
0.012507450999692082
oOo 19998737 [[50001, 49999], [50000, 50000], [49999, 50001]]
and beats the already very fast code we have started the journey with being ten times faster with 0.01 seconds.
Can this be still improved by a further order of magnitude? I am quite sure it can by more than that, so there is still plenty of room for improvement and for next answers with much faster code to continue the journey.
May the power of oOo be with you!
Just an alternative for btilly’s window_mins_maxes, also O(len(array)) time and O(size) space:
from itertools import accumulate
def window_mins_maxes(size, array):
leftmins = None
for stop in range(size, len(array) + 1):
if not leftmins:
left = array[stop-size : stop][::-1]
right = array[stop-1 : stop+size]
leftmins = list(accumulate(left, min))
leftmaxs = list(accumulate(left, max))
rightmins = accumulate(right, min)
rightmaxs = accumulate(right, max)
yield (
stop,
min(leftmins.pop(), next(rightmins)),
max(leftmaxs.pop(), next(rightmaxs))
)
The idea: If you want the minimum of some array range, split the range into a left part and a right part, and take the minimum of the left’s minimum and the right’s minimum.
For example for 10 elements and window size 4:
0 1 2 3 4 5 6 7 8 9
L L L L
R
L L L
R R
L L
R R R
L
R R R R
L L L L
R
...
The Ls mark the left part and the Rs mark the right part of the window. The right part is growing, so its minima are just its cumulative minima. The left part is shrinking, so its minima are it’s reversed cumulative suffix minima.
Whenever I don’t have left-part minima, I compute a chunk of them and reset the right part, as shown in the last step above.
I made L and R overlap so they’re never empty, not even when size=1.
While trying to find an algorithm for fast splitting an n -items long array of integers into sub-arrays of length between and including a and b with the biggest sum of differences between min and max values in each of the sub-arrays it is important to become aware of the fact that the complexity of the algorithm required to solve this problem highly depends on the parameters a and b which impact on the overall complexity is a way higher than the size of the array itself.
What is the time complexity of this problem expressed in terms of n, a and b and why is it important to be aware of it?
Becoming aware of it is important in order to see that as long as no algorithm allowing full linear time complexity becomes known here it will be necessary to use different algorithms to achieve good performance over various special ranges of possible values of a and b at given array size n.
If you have tried to use the algorithm provided by Claudio in the other answer (which claims to be ten times faster than the algorithm provided by btilly) with another values for a and b as these ones chosen by Claudio, you had already experienced its really bad performance compared to the dynamic programming approach in most of the other cases.
This is the point where knowing about the right formula for the time complexity of the problem helps to provide code which always runs faster.
Below the hopefully correct formula used in provided code for deciding about the choice of the appropriate algorithm to ensure best performance in all of the possible cases:
O( n * (b-a+1)**(n//a -1) ) ( ‘ ** ‘-> exponentiation ‘//’-> integer division )
In the code itself the (b-a+1)**(n//a) part of the formula is called complexityFactor and the threshold value used to choose one of the two provided algorithms for calculating the result is called complexityFactorThreshold
def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum(L, targetSum, n=None, m=None):
if n is None: n = 1
if m is None: m = targetSum
lenL = len(L)
# print(f"{targetSum=}, {L=}, {n=}, {m=}")
Stack = []
# Initialize the Stack with the starting point for each element in L
for i in range(lenL):
currentSum = L[ i ]
path = [ L[ i ] ]
start = 0 # Start from 0 allows revisiting of all items
Stack.append( (currentSum, path, start ) )
while Stack:
currentSum, path, start = Stack.pop()
# Check if the current path meets the criteria
if currentSum == targetSum and n <= len(path) <= m:
yield path
if currentSum > targetSum or len(path) > m:
continue
# ^ - NEXT please: stop exploring this path as it's not valid or complete
# Continue to build the path by adding elements from L, starting from 0 index
for i in range(len(L)): # Change start to 0 if you want to include all permutations
newSum = currentSum + L[ i ]
newPath = path + [ L[ i ] ]
Stack.append((newSum, newPath, 0)) # Start from 0 allows every possibility
# def allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
splitsGenerator = allArrangementsOfIntegerItemsInLsummingUpTOtargetSum
def lowComplexitySplit(Arr, m, n) :
targetSum=len(Arr)
L=list(range( m, n+1) )
bestSum = None
for split in splitsGenerator( L, targetSum ) :
#print("###", split)
startIndx = 0
endIndx = 0
currSum = 0
for indx in split :
endIndx += indx
currSum = currSum + max(Arr[startIndx : endIndx ]) - min(Arr[ startIndx : endIndx ])
startIndx=startIndx + endIndx
if bestSum is None:
bestSum = currSum
bestSplit = [ split ]
continue
if bestSum <= currSum :
if bestSum == currSum :
bestSplit.append(split)
else:
bestSum = currSum
bestSplit = [ split ]
return bestSum, bestSplit
def winMinMaxLargeSize(size, L):
""" oOosys 2024-04-23_21:30
This method of getting the min/max values within the over the list 'L' moved
chunk/interval/partition/section "Window" is saving the indices/positions of these
values in the list 'L' for later use. This allows skipping of recalculations of min/max
in cases where the integer value "leaving" the "Window" has no impact on the
min/max values because it fits into the range between min and max.
The variables storing the indices/positions are 'indxMin' and 'indxMax' and represent
the LAST position of the found min/max value if non-unique.
As Python suprisingly does not support simultaneous calculation of min/max and the
last position of obtained values some "trickery" is used to avoid some of the multiple
scans over the analysed "Window" in the process of getting both the min/max
values and their last indices/positions in the for them scanned list.
"""
# Lets calculate the min/max values for the first "Window" at the beginning of 'L' to
# get initial/start values for "moving" of the "Window" over the list 'L' :
prevMax, indxMax = max( zip(L[ 0 : size ], range( size )))
#prevMin, indxMin = min( zip(L[ 0 : size ], reversed(range( size )))); indxMin=size - indxMin - 1
prevMin=min(L[ 0 : size ]); indxMin = size - list(reversed(L[ 0 : size ])).index( prevMin ) - 1 # probably FASTER than in the line above
lenL=len(L)
#print(f"i={size=} , {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield size, prevMin, prevMax
for i in range(size+1, lenL+1):
if ( i - size - 1 ) == indxMin:
prevMin=min(L[ i - size : i ]); indxMin = i - list(reversed(L[ i - size : i ])).index( prevMin ) - 1 # appears to be the FASTEST ONE (see another approaches below)
# ---
#prevMin, indxMin = min( zip(L[ i - size : i ], range( size - 1, -1, -1 ))); indxMin = i - indxMin - 1
# ---
#prevMin, indxMin = min( zip(L[ i - size : i ], reversed( range( size )))); indxMin = i - indxMin - 1
# ---
#prevMin=min(L[ i - size : i ])
#for indx in range( i - size, i )[::-1]:
# if L[indx] == prevMin :
# indxMin = indx
# break
# ---
elif ( i - size - 1 ) == indxMax:
prevMax, indxMax = max( zip(L[ i - size : i ], range( size ))); indxMax = i - size + indxMax
#prevMax=max(L[ i - size : i ])
#for indx in list(range(i - size , i ))[::-1]:
# if L[indx] == prevMax :
# indxMax = indx
# break
else: # the value which has "left" the "Window" was a neutral one and it is possible to update values without re-calculating min/max over the "Window" width
if L[ i - 1 ] <= prevMin : # let's handle the case of necessary update of indxMin and eventually necessary update also of prevMin in one if block:
indxMin = i - 1
if L[ i - 1 ] < prevMin :
prevMin = L[ i - 1 ]
elif L[ i - 1 ] >= prevMax : # this is the case where the "incoming" value impacts indxMax and eventually the prevMax value
indxMax = i - 1
if L[ i - 1 ] > prevMax :
prevMax = L[ i - 1 ]
#print(f" > {i=:2d}, {indxMin=}, {indxMax=}, {prevMin=}, {prevMax=}")
yield i, prevMin, prevMax
def highComplexitySplit(listOfInts, minSplitWidth, maxSplitWidth):
if maxSplitWidth < minSplitWidth or len(listOfInts) < minSplitWidth:
return (None, None)
best_weight = [None for _ in listOfInts]
prev_index = [None for _ in listOfInts]
# Need an extra entry for off of the end of the array.
best_weight.append(None)
prev_index.append(None)
best_weight[0] = 0
for i, min_value, max_value in winMinMaxLargeSize(minSplitWidth, listOfInts):
#print(f" {i=:4d}, {min_value=:8d}, {max_value=:8d}", end=" >>> ")
window_start_weight = best_weight[i - minSplitWidth]
if window_start_weight is not None:
j = i
while j - i < maxSplitWidth - minSplitWidth and j < len(listOfInts):
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - minSplitWidth
if listOfInts[j] < min_value:
min_value = listOfInts[j]
if max_value < listOfInts[j]:
max_value = listOfInts[j]
j += 1
# And fill in the longest value.
new_weight = window_start_weight + max_value - min_value
if best_weight[j] is None or best_weight[j] < new_weight:
best_weight[j] = new_weight
prev_index[j] = i - minSplitWidth
if best_weight[-1] is None:
return (None, None)
else:
path = [len(listOfInts)]
while prev_index[path[-1]] is not None:
path.append(prev_index[path[-1]])
#path = list(reversed(path))
path = list(path)[::-1]
#partitioned = [listOfInts[path[i]:path[i+1]] for i in range(len(path)-1)]
#return (best_weight[-1], partitioned)
return (best_weight[-1], path )
def complexityFactorSplit(listOfInts, minSplitWidth, maxSplitWidth) :
n=len(listOfInts)
complexityFactor=( maxSplitWidth - minSplitWidth + 1 )**( n//minSplitWidth - 1)
complexityFactorThreshold = 7
if complexityFactor <= complexityFactorThreshold :
return lowComplexitySplit( listOfInts, minSplitWidth, maxSplitWidth )
else :
return highComplexitySplit( listOfInts, minSplitWidth, maxSplitWidth )
# ---
#n=100000; a=49970; b=50030
#n=100000; a=50000; b=50000
#n=100000; a=49995; b=50005
from time import perf_counter as T
from random import randint, seed
seed(2); arr=[]; n=100000
for _ in range(n): arr.append(randint(1,n**2))
print(f"{n=}")
for step in range(0, 11, 1):
a=50000 - step
b=50000 + step
sT1=T(); lowComplexitySplit(arr, a, b); eT1=T()
sT2=T(); highComplexitySplit(arr, a, b); eT2=T()
print(f"a/b:{a}/{b} lowC: {eT1-sT1:.3f},highC: {eT2-sT2:.3f} low/high:{100*(eT1-sT1)/(eT2-sT2): 5.0f}% , cF={( b - a + 1 )**( n//a - 1):3d} , btilly O(): O( n *{b+1-a: 4d} )")
for step in range(15, 60, 5):
a=50000 - step
b=50000 + step
sT1=T(); lowComplexitySplit(arr, a, b); eT1=T()
sT2=T(); highComplexitySplit(arr, a, b); eT2=T()
print(f"a/b:{a}/{b} lowC: {eT1-sT1:.3f},highC: {eT2-sT2:.3f} low/high:{100*(eT1-sT1)/(eT2-sT2): 5.0f}% , cF={( b - a + 1 )**( n//a - 1):3d} , btilly O(): O( n *{b+1-a: 4d} )")
which outputs:
n=100000
a/b:50000/50000 lowC: 0.005,highC: 0.050 low/high: 11% , cF= 1 , btilly O(): O( n * 1 )
a/b:49999/50001 lowC: 0.023,highC: 0.048 low/high: 47% , cF= 3 , btilly O(): O( n * 3 )
a/b:49998/50002 lowC: 0.031,highC: 0.051 low/high: 60% , cF= 5 , btilly O(): O( n * 5 )
a/b:49997/50003 lowC: 0.042,highC: 0.048 low/high: 88% , cF= 7 , btilly O(): O( n * 7 )
a/b:49996/50004 lowC: 0.055,highC: 0.048 low/high: 115% , cF= 9 , btilly O(): O( n * 9 )
a/b:49995/50005 lowC: 0.065,highC: 0.046 low/high: 141% , cF= 11 , btilly O(): O( n * 11 )
a/b:49994/50006 lowC: 0.076,highC: 0.046 low/high: 166% , cF= 13 , btilly O(): O( n * 13 )
a/b:49993/50007 lowC: 0.090,highC: 0.046 low/high: 195% , cF= 15 , btilly O(): O( n * 15 )
a/b:49992/50008 lowC: 0.100,highC: 0.047 low/high: 212% , cF= 17 , btilly O(): O( n * 17 )
a/b:49991/50009 lowC: 0.108,highC: 0.046 low/high: 233% , cF= 19 , btilly O(): O( n * 19 )
a/b:49990/50010 lowC: 0.120,highC: 0.047 low/high: 254% , cF= 21 , btilly O(): O( n * 21 )
a/b:49985/50015 lowC: 0.175,highC: 0.046 low/high: 378% , cF= 31 , btilly O(): O( n * 31 )
a/b:49980/50020 lowC: 0.238,highC: 0.047 low/high: 510% , cF= 41 , btilly O(): O( n * 41 )
a/b:49975/50025 lowC: 0.297,highC: 0.048 low/high: 620% , cF= 51 , btilly O(): O( n * 51 )
a/b:49970/50030 lowC: 0.367,highC: 0.048 low/high: 769% , cF= 61 , btilly O(): O( n * 61 )
a/b:49965/50035 lowC: 0.445,highC: 0.048 low/high: 935% , cF= 71 , btilly O(): O( n * 71 )
a/b:49960/50040 lowC: 0.531,highC: 0.048 low/high: 1100% , cF= 81 , btilly O(): O( n * 81 )
a/b:49955/50045 lowC: 0.625,highC: 0.048 low/high: 1300% , cF= 91 , btilly O(): O( n * 91 )
a/b:49950/50050 lowC: 0.730,highC: 0.048 low/high: 1512% , cF=101 , btilly O(): O( n * 101 )
a/b:49945/50055 lowC: 0.841,highC: 0.049 low/high: 1722% , cF=111 , btilly O(): O( n * 111 )
The output shows that the threshold value for cF (complexity Factor) of 7 will in the case of the example allow a right decision about the algorithm to use while running the complexityFactorSplit() function which then delivers fast results for both, high and low complexity sets of values for a and b.
Notice in the output that the by btilly provided formula for the O() complexity suggests growing complexity, but the code running time (highC) does not increase what proves that this formula is incorrect for both: describing the behavior of the code and describing the complexity of the problem itself.
There are up to now ( 2024-04-24 ) following main algorithmic approaches formulated as Python scripts in the question and all the other answers:
- brute force approach which fails to handle arrays of length beyond 35 ( in OP question )
- a dynamic programming approach able to handle large arrays ( first given answer by btilly )
- a dynamic programming btilly code based approach with improved core function for finding the maxima and minima in all "windows" with width a sliding over the array of numbers running twice as fast as the original version
- a brute force approach generating in first step a splitting which runs on some cases of a and b values ten times faster then the fastest code in the already mentioned approaches
- a mixed approach achieving overall faster results by switching to the faster algorithm running two to ten times faster than the first by btilly posted code.
Is it possible to provide an algorithm which runs faster than all of the already provided ones without any tricks with making the code as such running faster?
This answer shows that it is possible covering a much wider range of cases as the up to now fastest algorithm suitable for a small range of b and a values for given n being the size of the array with numbers.
Compared to the first posted fast dynamic programming based algorithm the below provided code runs 60 times faster and could probably not be improved as an algorithm (except there is some relationship valid because of the specifics of the weight function being the difference of max-min values within the chunk/partition/sub-list), so speeding it up will need faster running code implementing the algorithm.
Below the code showing the basic idea behind the algorithm rendering all the previous considerations about complexity of the problem obsolete.
Using this basic idea will make it with some further work on it possible to provide code covering all possible cases by hierarchical building upon this core "brick like" unit. It will then handle all of possible parameter sets for n, a, b and not only these which satisfy the condition n//a == 2 ( where // means integer division ) leading to better performance than the dynamic programming based approach and capable to provide multiple solutions for splittings if there are any.
def oneSplitPoint(Arr,a,b):
lenArr=len(Arr)
assert lenArr//a == 2
minL, maxL =min(Arr[:a]), max(Arr[:a])
minR, maxR =min(Arr[-a:]), max(Arr[-a:])
dL =[ maxL -minL ]
dR =[ maxR - minR ]
if a == b and a+b == n:
return ( dL[0]+dR[0], f"[ {a} ]" )
minSplitWidth = a ; maxSplitWidth = b
for indxL in range( minSplitWidth, maxSplitWidth ):
if Arr[indxL] > maxL : maxL = Arr[indxL]
if Arr[indxL] < minL : minL = Arr[indxL]
dL = dL + [ maxL -minL ]
indxR = lenArr - indxL - 1
if Arr[indxR] > maxR : maxR = Arr[indxR]
if Arr[indxR] < minR : minR = Arr[indxR]
dR = dR + [ maxR -minR ]
dR.reverse()
sumLR = []
for indx in range(maxSplitWidth - minSplitWidth + 1) :
sumLR = sumLR + [ dL[indx] + dR[indx] ]
maxSumLR = max( sumLR )
splitIndices = []; lastIndex = 0
while True:
try:
lastIndex = sumLR.index( maxSumLR , lastIndex ) + 1
splitIndices = splitIndices + [ minSplitWidth + lastIndex - 2 ]
except ValueError:
break
return maxSumLR, splitIndices
Below the output from running the above code against the improved dynamic programming algorithm approach showing cutting down of the time to 3% of the reference code. And because the reference code runs twice as fast as the first one posted it means an improvement of being over 60 times faster.
n=12000
a/b:6000/6000 lowC: 0.000,highC: 0.005 low/high: 11% , cF= 0 , btilly O(): O( n * 1 )
a/b:5800/6200 lowC: 0.002,highC: 0.040 low/high: 4% , cF= 1 , btilly O(): O( n * 401 )
a/b:5600/6400 lowC: 0.004,highC: 0.137 low/high: 3% , cF= 4 , btilly O(): O( n * 801 )
a/b:5400/6600 lowC: 0.009,highC: 0.301 low/high: 3% , cF= 9 , btilly O(): O( n * 1201 )
a/b:5200/6800 lowC: 0.017,highC: 0.530 low/high: 3% , cF= 16 , btilly O(): O( n * 1601 )
a/b:5000/7000 lowC: 0.025,highC: 0.828 low/high: 3% , cF= 25 , btilly O(): O( n * 2001 )
a/b:4800/7200 lowC: 0.039,highC: 1.188 low/high: 3% , cF= 36 , btilly O(): O( n * 2401 )
I have adjusted the complexity factor to a formula which mirrors the running time and it seems that the time complexity of this approach is rough estimated proportional to the square of (b-a+1) i.e. O(n(b-a+1)^2) .