Is there a way to speed up permutations of permutations?

Question

I’m trying to generate permutations of 5 numbers from a list (numset). Then I choose two of the five numbers and test them (another permutation). This appears to work but is very inefficient and time consuming with a large numset list (100k+ entries). I’m wondering if there is a way to optimize this?

from itertools import permutations

numbers2 = []

# Get all permutations of length 5 so we have five numbers to test.
numbers5 = permutations(numset, 5)

def testfive():
  # Choose two of the numbers from the 5 number list and test them.
  for i in list(numbers5):
      numbers2 = permutations(numbers5, 2)
      for x in list(numbers2):
        if (test(x)!=True):
          break
      # Do something if the test is successful.

testfive()

Here is how this should operate:

I start with a large number set (numset) like: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 …]

I pick five numbers, without repeating: [1, 2, 3, 4, 5]

Then I pick two of these: [2, 4] or [2, 6] or [2, 8] and so on.

Then I start over with five more numbers: [1, 2, 3, 4, 6]

I do this over, and over, and over again. I’m using Colab, and it runs out of RAM. I know this is a brute force way of going through numbers, but I’m wondering if there is a tool to optimize this that I’m missing?

Asked By: Alligator

||

Source

Answer 1

I would avoid calling the list function because that would compute the list, load the whole list into your RAM and iterate through it after it was created and numbers5 never gets deleted in your code.

Here is some (pseudo)code to show you a better way of doing this. Everything should be computed when it’s needed and discarded after its usage. A generator does exactly this, like explained here. permutations is a generator because it yields one result on every next() call. And I assume you want to select two numbers randomly:

from itertools import permutations
from random import choices

#numset = range(8) # for testing

def testfive():
  # Choose two of the numbers from the 5 number list and test them.
  for l in permutations(numset, 5): # is discarded after use
        if test(choices(l, k=2)):
          # Success
      
testfive()

But as a side note: If you do this with 100k entries, then you have 100000! / (100000-5)! = 9.999 000 035 × 10^24 5-permutations.
For further optimization, you could take a look at Cython. It can be much faster.

Answered By: Confused Learner

Is there a way to speed up permutations of permutations?

Question:

Answers: