Python itertool variations, memory maximum reached

Question

I am currently looking to generate a list of numbers with a specific number of digits, my code currently as follows:

| Python 2.7 |

import itertools

inp = raw_input('Number of digits to write?:')
inp = int(inp)
inp2 = raw_input('File name?:')
inp2 = inp2 + '.txt'
variants = ["".join(item) for item in itertools.product("0123456789", repeat=inp)]

variant = open(inp2, 'w')

for number in variants:
    variant.write("%sn" % number)

As you can see i am trying to produce multiple different files with the output placed line by line for each new number.

I know that the list may be of issue, as it is storing all the possible numbers in memory within that list. My question is: With the digits being more then 7, there is a memory issue, how would I go about reducing the memory needed or put multiple files together to generate a list with the same type of data.

would a for loop work to in essence ‘append’ two lists together ( say for the 4 digit file and the 5 digit file to create essentially a 9 digit file ) without the use of this specific itertools implementation?

Perhaps some sort of recursion? ( I still do not understand how to write recursive functions etc. Im a noob when it comes to Programing in general)

Asked By: slackakracka

||

Source

Answer 1

just use the iterator as its intended to be used … the whole point of iterators is to not store everything in memory at once …

variants = itertools.product("0123456789", repeat=inp)

variant = open(inp2, 'w')

for number in variants:
    variant.write("%sn" % (" ".join(number))

alternatively you could use a generator instead that would be functionally equivelent

variants = ("".join(item) for item in itertools.product("0123456789", repeat=inp)) #now its a generator expression


with open("outfile","wb") as variant:
    for number in variants:
        variant.write("%sn"%number)

as pointed out you could do this much easier as

max_value = 10**n
with open("outfile","wb") as variant:
    for number in xrange(max_value):
        variant.write("{0:09d}n".format(number))

Answered By: Joran Beasley

Answer 2

When you construct the variants list, you’re putting all of the strings into memory. When you have 10^7 or more strings in memory, it makes sense that you would start running low on it. What you’ll want to do is go through the iterator and write out your strings one at a time. This begins after you get your input.

variants = itertools.product('0123456789',repeat=inp)
outfile = open(inp2,'w')
for group in variants:
   outfile.write("%sn" % (''.join(group)))

Answered By: Brien

Python itertool variations, memory maximum reached

Question:

Answers: