How to store itertools.chain and use it more than once?

Question

I would like to use itertools.chain for efficient concatenation of lists (memoization), but I need to be able to read (or map, etc.) the result multiple times. This example illustrates the problem:

import itertools
a = itertools.chain([1, 2], [3, 4])
print list(a) # => [1, 2, 3, 4]
print list(a) # => []

What is the best way to avoid this problem?

Asked By: jtbandes

||

Source

Answer 1

As with all generators, you’ll need to convert it to a list and store that result instead:

a = list(a)

This is a fundamental principle of generators, they are expected to produce their sequence only once.

Moreover, you cannot simply store a generator for memoization purposes, as the underlying lists could change. In almost all memoization use-cases, you should store the list instead; a generator is usually only a means of efficiently transforming or filtering the underlying sequences, and does not represent the data you want to memoize itself. It’s as if you are storing a function, not it’s output. In your specific case, if all what you are doing is using chain() to concatenate existing lists, store those lists directly instead.

Note that this enables generators to produce endless sequences, so be careful with that you convert to a list.

Answered By: Martijn Pieters

Answer 2

Try itertools.tee:

import itertools
a = itertools.chain([1, 2], [3, 4])
a, b = itertools.tee(a)
print list(b) # => [1, 2, 3, 4]
a, b = itertools.tee(a)
print list(b) # => [1, 2, 3, 4]

Answered By: georg

How to store itertools.chain and use it more than once?

Question:

Answers: