# How to iterate over a list in chunks

## Question:

I have a Python script which takes as input a list of integers, which I need to work with four integers at a time. Unfortunately, I don’t have control of the input, or I’d have it passed in as a list of four-element tuples. Currently, I’m iterating over it this way:

``````for i in range(0, len(ints), 4):
# dummy op for example code
foo += ints[i] * ints[i + 1] + ints[i + 2] * ints[i + 3]
``````

It looks a lot like "C-think", though, which makes me suspect there’s a more pythonic way of dealing with this situation. The list is discarded after iterating, so it needn’t be preserved. Perhaps something like this would be better?

``````while ints:
foo += ints[0] * ints[1] + ints[2] * ints[3]
ints[0:4] = []
``````

Still doesn’t quite "feel" right, though. :-/

Related question: How do you split a list into evenly sized chunks in Python?

There doesn’t seem to be a pretty way to do this. Here is a page that has a number of methods, including:

``````def split_seq(seq, size):
newseq = []
splitsize = 1.0/size*len(seq)
for i in range(size):
newseq.append(seq[int(round(i*splitsize)):int(round((i+1)*splitsize))])
return newseq
``````

In your second method, I would advance to the next group of 4 by doing this:

``````ints = ints[4:]
``````

However, I haven’t done any performance measurement so I don’t know which one might be more efficient.

Having said that, I would usually choose the first method. It’s not pretty, but that’s often a consequence of interfacing with the outside world.

``````import itertools
def chunks(iterable,size):
it = iter(iterable)
chunk = tuple(itertools.islice(it,size))
while chunk:
yield chunk
chunk = tuple(itertools.islice(it,size))

# though this will throw ValueError if the length of ints
# isn't a multiple of four:
for x1,x2,x3,x4 in chunks(ints,4):
foo += x1 + x2 + x3 + x4

for chunk in chunks(ints,4):
foo += sum(chunk)
``````

Another way:

``````import itertools
def chunks2(iterable,size,filler=None):
it = itertools.chain(iterable,itertools.repeat(filler,size-1))
chunk = tuple(itertools.islice(it,size))
while len(chunk) == size:
yield chunk
chunk = tuple(itertools.islice(it,size))

# x2, x3 and x4 could get the value 0 if the length is not
# a multiple of 4.
for x1,x2,x3,x4 in chunks2(ints,4,0):
foo += x1 + x2 + x3 + x4
``````
``````chunk_size = 4
for i in range(0, len(ints), chunk_size):
chunk = ints[i:i+chunk_size]
# process chunk of size <= chunk_size
``````
``````def chunker(seq, size):
return (seq[pos:pos + size] for pos in range(0, len(seq), size))
``````

Works with any sequence:

``````text = "I am a very, very helpful text"

for group in chunker(text, 7):
print(repr(group),)
# 'I am a ' 'very, v' 'ery hel' 'pful te' 'xt'

print('|'.join(chunker(text, 10)))
# I am a ver|y, very he|lpful text

animals = ['cat', 'dog', 'rabbit', 'duck', 'bird', 'cow', 'gnu', 'fish']

for group in chunker(animals, 3):
print(group)
# ['cat', 'dog', 'rabbit']
# ['duck', 'bird', 'cow']
# ['gnu', 'fish']
``````

If the list is large, the highest-performing way to do this will be to use a generator:

``````def get_chunk(iterable, chunk_size):
result = []
for item in iterable:
result.append(item)
if len(result) == chunk_size:
yield tuple(result)
result = []
if len(result) > 0:
yield tuple(result)

for x in get_chunk([1,2,3,4,5,6,7,8,9,10], 3):
print x

(1, 2, 3)
(4, 5, 6)
(7, 8, 9)
(10,)
``````

If the lists are the same size, you can combine them into lists of 4-tuples with `zip()`. For example:

``````# Four lists of four elements each.

l1 = range(0, 4)
l2 = range(4, 8)
l3 = range(8, 12)
l4 = range(12, 16)

for i1, i2, i3, i4 in zip(l1, l2, l3, l4):
...
``````

Here’s what the `zip()` function produces:

``````>>> print l1
[0, 1, 2, 3]
>>> print l2
[4, 5, 6, 7]
>>> print l3
[8, 9, 10, 11]
>>> print l4
[12, 13, 14, 15]
>>> print zip(l1, l2, l3, l4)
[(0, 4, 8, 12), (1, 5, 9, 13), (2, 6, 10, 14), (3, 7, 11, 15)]
``````

If the lists are large, and you don’t want to combine them into a bigger list, use `itertools.izip()`, which produces an iterator, rather than a list.

``````from itertools import izip

for i1, i2, i3, i4 in izip(l1, l2, l3, l4):
...
``````
``````from itertools import izip_longest

def chunker(iterable, chunksize, filler):
return izip_longest(*[iter(iterable)]*chunksize, fillvalue=filler)
``````

Modified from the Recipes section of Python’s `itertools` docs:

``````from itertools import zip_longest

def grouper(iterable, n, fillvalue=None):
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)
``````

Example

``````grouper('ABCDEFG', 3, 'x')  # --> 'ABC' 'DEF' 'Gxx'
``````

Note: on Python 2 use `izip_longest` instead of `zip_longest`.

Since nobody’s mentioned it yet here’s a `zip()` solution:

``````>>> def chunker(iterable, chunksize):
...     return zip(*[iter(iterable)]*chunksize)
``````

It works only if your sequence’s length is always divisible by the chunk size or you don’t care about a trailing chunk if it isn’t.

Example:

``````>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9')]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8')]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
``````

Or using itertools.izip to return an iterator instead of a list:

``````>>> from itertools import izip
>>> def chunker(iterable, chunksize):
...     return izip(*[iter(iterable)]*chunksize)
``````

``````>>> from itertools import chain, izip, repeat
>>> def chunker(iterable, chunksize, fillvalue=None):
...     it   = chain(iterable, repeat(fillvalue, chunksize-1))
...     args = [it] * chunksize
...     return izip(*args)
``````

``````>>> def chunker(iterable, chunksize):
...   return map(None,*[iter(iterable)]*chunksize)
``````

Example:

``````>>> s = '1234567890'
>>> chunker(s, 3)
[('1', '2', '3'), ('4', '5', '6'), ('7', '8', '9'), ('0', None, None)]
>>> chunker(s, 4)
[('1', '2', '3', '4'), ('5', '6', '7', '8'), ('9', '0', None, None)]
>>> chunker(s, 5)
[('1', '2', '3', '4', '5'), ('6', '7', '8', '9', '0')]
``````

The ideal solution for this problem works with iterators (not just sequences). It should also be fast.

This is the solution provided by the documentation for itertools:

``````def grouper(n, iterable, fillvalue=None):
#"grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
return itertools.izip_longest(fillvalue=fillvalue, *args)
``````

Using ipython’s `%timeit` on my mac book air, I get 47.5 us per loop.

However, this really doesn’t work for me since the results are padded to be even sized groups. A solution without the padding is slightly more complicated. The most naive solution might be:

``````def grouper(size, iterable):
i = iter(iterable)
while True:
out = []
try:
for _ in range(size):
out.append(i.next())
except StopIteration:
yield out
break

yield out
``````

Simple, but pretty slow: 693 us per loop

The best solution I could come up with uses `islice` for the inner loop:

``````def grouper(size, iterable):
it = iter(iterable)
while True:
group = tuple(itertools.islice(it, None, size))
if not group:
break
yield group
``````

With the same dataset, I get 305 us per loop.

Unable to get a pure solution any faster than that, I provide the following solution with an important caveat: If your input data has instances of `filldata` in it, you could get wrong answer.

``````def grouper(n, iterable, fillvalue=None):
#"grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
# itertools.zip_longest on Python 3
for x in itertools.izip_longest(*args, fillvalue=fillvalue):
if x[-1] is fillvalue:
yield tuple(v for v in x if v is not fillvalue)
else:
yield x
``````

I really don’t like this answer, but it is significantly faster. 124 us per loop

1) Easily understandable
2) Works on any iterable, not just sequences (some of the above answers will choke on filehandles)
3) Does not load the chunk into memory all at once
4) Does not make a chunk-long list of references to the same iterator in memory
5) No padding of fill values at the end of the list

That being said, I haven’t timed it so it might be slower than some of the more clever methods, and some of the advantages may be irrelevant given the use case.

``````def chunkiter(iterable, size):
def inneriter(first, iterator, size):
yield first
for _ in xrange(size - 1):
yield iterator.next()
it = iter(iterable)
while True:
yield inneriter(it.next(), it, size)

In [2]: i = chunkiter('abcdefgh', 3)
In [3]: for ii in i:
for c in ii:
print c,
print ''
...:
a b c
d e f
g h
``````

Update:
A couple of drawbacks due to the fact the inner and outer loops are pulling values from the same iterator:
1) continue doesn’t work as expected in the outer loop – it just continues on to the next item rather than skipping a chunk. However, this doesn’t seem like a problem as there’s nothing to test in the outer loop.
2) break doesn’t work as expected in the inner loop – control will wind up in the inner loop again with the next item in the iterator. To skip whole chunks, either wrap the inner iterator (ii above) in a tuple, e.g. `for c in tuple(ii)`, or set a flag and exhaust the iterator.

Similar to other proposals, but not exactly identical, I like doing it this way, because it’s simple and easy to read:

``````it = iter([1, 2, 3, 4, 5, 6, 7, 8, 9])
for chunk in zip(it, it, it, it):
print chunk

>>> (1, 2, 3, 4)
>>> (5, 6, 7, 8)
``````

This way you won’t get the last partial chunk. If you want to get `(9, None, None, None)` as last chunk, just use `izip_longest` from `itertools`.

Using little functions and things really doesn’t appeal to me; I prefer to just use slices:

``````data = [...]
chunk_size = 10000 # or whatever
chunks = [data[i:i+chunk_size] for i in xrange(0,len(data),chunk_size)]
for chunk in chunks:
...
``````

I needed a solution that would also work with sets and generators. I couldn’t come up with anything very short and pretty, but it’s quite readable at least.

``````def chunker(seq, size):
res = []
for el in seq:
res.append(el)
if len(res) == size:
yield res
res = []
if res:
yield res
``````

List:

``````>>> list(chunker([i for i in range(10)], 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
``````

Set:

``````>>> list(chunker(set([i for i in range(10)]), 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
``````

Generator:

``````>>> list(chunker((i for i in range(10)), 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
``````
``````def group_by(iterable, size):
"""Group an iterable into lists that don't exceed the size given.

>>> group_by([1,2,3,4,5], 2)
[[1, 2], [3, 4], [5]]

"""
sublist = []

for index, item in enumerate(iterable):
if index > 0 and index % size == 0:
yield sublist
sublist = []

sublist.append(item)

if sublist:
yield sublist
``````

Another approach would be to use the two-argument form of `iter`:

``````from itertools import islice

def group(it, size):
it = iter(it)
return iter(lambda: tuple(islice(it, size)), ())
``````

``````from itertools import islice, chain, repeat

return iter(lambda: tuple(islice(it, size)), (pad,) * size)
``````

These can even be combined for optional padding:

``````_no_pad = object()
it = iter(it)
sentinel = ()
else:
return iter(lambda: tuple(islice(it, size)), sentinel)
``````

You can use partition or chunks function from funcy library:

``````from funcy import partition

for a, b, c, d in partition(4, ints):
foo += a * b * c * d
``````

These functions also has iterator versions `ipartition` and `ichunks`, which will be more efficient in this case.

You can also peek at their implementation.

One-liner, adhoc solution to iterate over a list `x` in chunks of size `4`

``````for a, b, c, d in zip(x[0::4], x[1::4], x[2::4], x[3::4]):
... do something with a, b, c and d ...
``````

To avoid all conversions to a list `import itertools` and:

``````>>> for k, g in itertools.groupby(xrange(35), lambda x: x/10):
...     list(g)
``````

Produces:

``````...
0 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
1 [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
2 [20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
3 [30, 31, 32, 33, 34]
>>>
``````

I checked `groupby` and it doesn’t convert to list or use `len` so I (think) this will delay resolution of each value until it is actually used. Sadly none of the available answers (at this time) seemed to offer this variation.

Obviously if you need to handle each item in turn nest a for loop over g:

``````for k,g in itertools.groupby(xrange(35), lambda x: x/10):
for i in g:
# do what you need to do with individual items
# now do what you need to do with the whole group
``````

My specific interest in this was the need to consume a generator to submit changes in batches of up to 1000 to the gmail API:

``````    messages = a_generator_which_would_not_be_smart_as_a_list
for idx, batch in groupby(messages, lambda x: x/1000):
batch_request = BatchHttpRequest()
for message in batch:
http = httplib2.Http()
self.credentials.authorize(http)
batch_request.execute(http=http)
``````

With NumPy it’s simple:

``````ints = array([1, 2, 3, 4, 5, 6, 7, 8])
for int1, int2 in ints.reshape(-1, 2):
print(int1, int2)
``````

output:

``````1 2
3 4
5 6
7 8
``````

At first, I designed it to split strings into substrings to parse string containing hex.
Today I turned it into complex, but still simple generator.

``````def chunker(iterable, size, reductor, condition):
it = iter(iterable)
def chunk_generator():
return (next(it) for _ in range(size))
chunk = reductor(chunk_generator())
while condition(chunk):
yield chunk
chunk = reductor(chunk_generator())
``````

# Arguments:

### Obvious ones

• `iterable` is any iterable / iterator / generator containg / generating / iterating over input data,
• `size` is, of course, size of chunk you want get,

### More interesting

• `reductor` is a callable, which receives generator iterating over content of chunk.
I’d expect it to return sequence or string, but I don’t demand that.

You can pass as this argument for example `list`, `tuple`, `set`, `frozenset`,
or anything fancier. I’d pass this function, returning string
(provided that `iterable` contains / generates / iterates over strings):

``````def concatenate(iterable):
return ''.join(iterable)
``````

Note that `reductor` can cause closing generator by raising exception.

• `condition` is a callable which receives anything what `reductor` returned.
It decides to approve & yield it (by returning anything evaluating to `True`),
or to decline it & finish generator’s work (by returning anything other or raising exception).

When number of elements in `iterable` is not divisible by `size`, when `it` gets exhausted, `reductor` will receive generator generating less elements than `size`.
Let’s call these elements lasts elements.

I invited two functions to pass as this argument:

• `lambda x:x` – the lasts elements will be yielded.

• `lambda x: len(x)==<size>` – the lasts elements will be rejected.
replace `<size>` using number equal to `size`

About solution gave by `J.F. Sebastian` here:

``````def chunker(iterable, chunksize):
return zip(*[iter(iterable)]*chunksize)
``````

It’s clever, but has one disadvantage – always return tuple. How to get string instead?
Of course you can write `''.join(chunker(...))`, but the temporary tuple is constructed anyway.

You can get rid of the temporary tuple by writing own `zip`, like this:

``````class IteratorExhausted(Exception):
pass

def translate_StopIteration(iterable, to=IteratorExhausted):
for i in iterable:
yield i
raise to # StopIteration would get ignored because this is generator,
# but custom exception can leave the generator.

def custom_zip(*iterables, reductor=tuple):
iterators = tuple(map(translate_StopIteration, iterables))
while True:
try:
yield reductor(next(i) for i in iterators)
except IteratorExhausted: # when any of iterators get exhausted.
break
``````

Then

``````def chunker(data, size, reductor=tuple):
return custom_zip(*[iter(data)]*size, reductor=reductor)
``````

Example usage:

``````>>> for i in chunker('12345', 2):
...     print(repr(i))
...
('1', '2')
('3', '4')
>>> for i in chunker('12345', 2, ''.join):
...     print(repr(i))
...
'12'
'34'
``````

Here is a chunker without imports that supports generators:

``````def chunks(seq, size):
it = iter(seq)
while True:
ret = tuple(next(it) for _ in range(size))
if len(ret) == size:
yield ret
else:
raise StopIteration()
``````

Example of use:

``````>>> def foo():
...     i = 0
...     while True:
...         i += 1
...         yield i
...
>>> c = chunks(foo(), 3)
>>> c.next()
(1, 2, 3)
>>> c.next()
(4, 5, 6)
>>> list(chunks('abcdefg', 2))
[('a', 'b'), ('c', 'd'), ('e', 'f')]
``````

It is easy to make `itertools.groupby` work for you to get an iterable of iterables, without creating any temporary lists:

``````groupby(iterable, (lambda x,y: (lambda z: x.next()/y))(count(),100))
``````

Don’t get put off by the nested lambdas, outer lambda runs just once to put `count()` generator and the constant `100` into the scope of the inner lambda.

I use this to send chunks of rows to mysql.

``````for k,v in groupby(bigdata, (lambda x,y: (lambda z: x.next()/y))(count(),100))):
cursor.executemany(sql, v)
``````
``````def chunker(iterable, n):
"""Yield iterable in chunk sizes.

>>> chunks = chunker('ABCDEF', n=4)
>>> chunks.next()
['A', 'B', 'C', 'D']
>>> chunks.next()
['E', 'F']
"""
it = iter(iterable)
while True:
chunk = []
for i in range(n):
try:
chunk.append(next(it))
except StopIteration:
yield chunk
raise StopIteration
yield chunk

if __name__ == '__main__':
import doctest

doctest.testmod()
``````

I like this approach. It feels simple and not magical and supports all iterable types and doesn’t require imports.

``````def chunk_iter(iterable, chunk_size):
it = iter(iterable)
while True:
chunk = tuple(next(it) for _ in range(chunk_size))
if not chunk:
break
yield chunk
``````

Quite pythonic here (you may also inline the body of the `split_groups` function)

``````import itertools
def split_groups(iter_in, group_size):
return ((x for _, x in item) for _, item in itertools.groupby(enumerate(iter_in), key=lambda x: x[0] // group_size))

for x, y, z, w in split_groups(range(16), 4):
foo += x * y + z * w
``````

This answer splits a list of strings, f.ex. to achieve PEP8-line length compliance:

``````def split(what, target_length=79):
'''splits list of strings into sublists, each
having string length at most 79'''
out = [[]]
while what:
if len("', '".join(out[-1])) + len(what[0]) < target_length:
out[-1].append(what.pop(0))
else:
if not out[-1]: # string longer than target_length
out[-1] = [what.pop(0)]
out.append([])
return out
``````

Use as

``````>>> split(['deferred_income', 'long_term_incentive', 'restricted_stock_deferred', 'shared_receipt_with_poi', 'loan_advances', 'from_messages', 'other', 'director_fees', 'bonus', 'total_stock_value', 'from_poi_to_this_person', 'from_this_person_to_poi', 'restricted_stock', 'salary', 'total_payments', 'exercised_stock_options'], 75)
[['deferred_income', 'long_term_incentive', 'restricted_stock_deferred'], ['shared_receipt_with_poi', 'loan_advances', 'from_messages', 'other'], ['director_fees', 'bonus', 'total_stock_value', 'from_poi_to_this_person'], ['from_this_person_to_poi', 'restricted_stock', 'salary', 'total_payments'], ['exercised_stock_options']]
``````

I never want my chunks padded, so that requirement is essential. I find that the ability to work on any iterable is also requirement. Given that, I decided to extend on the accepted answer, https://stackoverflow.com/a/434411/1074659.

Performance takes a slight hit in this approach if padding is not wanted due to the need to compare and filter the padded values. However, for large chunk sizes, this utility is very performant.

``````#!/usr/bin/env python3
from itertools import zip_longest

_UNDEFINED = object()

def chunker(iterable, chunksize, fillvalue=_UNDEFINED):
"""
Collect data into chunks and optionally pad it.

Performance worsens as `chunksize` approaches 1.

Inspired by:
https://docs.python.org/3/library/itertools.html#itertools-recipes

"""
args = [iter(iterable)] * chunksize
chunks = zip_longest(*args, fillvalue=fillvalue)
yield from (
filter(lambda val: val is not _UNDEFINED, chunk)
if chunk[-1] is _UNDEFINED
else chunk
for chunk in chunks
) if fillvalue is _UNDEFINED else chunks
``````

If you don’t mind using an external package you could use `iteration_utilities.grouper` from `iteration_utilties` 1. It supports all iterables (not just sequences):

``````from iteration_utilities import grouper
seq = list(range(20))
for group in grouper(seq, 4):
print(group)
``````

which prints:

``````(0, 1, 2, 3)
(4, 5, 6, 7)
(8, 9, 10, 11)
(12, 13, 14, 15)
(16, 17, 18, 19)
``````

In case the length isn’t a multiple of the groupsize it also supports filling (the incomplete last group) or truncating (discarding the incomplete last group) the last one:

``````from iteration_utilities import grouper
seq = list(range(17))
for group in grouper(seq, 4):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
# (16,)

for group in grouper(seq, 4, fillvalue=None):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
# (16, None, None, None)

for group in grouper(seq, 4, truncate=True):
print(group)
# (0, 1, 2, 3)
# (4, 5, 6, 7)
# (8, 9, 10, 11)
# (12, 13, 14, 15)
``````

## Benchmarks

I also decided to compare the run-time of a few of the mentioned approaches. It’s a log-log plot grouping into groups of “10” elements based on a list of varying size. For qualitative results: Lower means faster:

At least in this benchmark the `iteration_utilities.grouper` performs best. Followed by the approach of Craz.

The benchmark was created with `simple_benchmark`1. The code used to run this benchmark was:

``````import iteration_utilities
import itertools
from itertools import zip_longest

def consume_all(it):
return iteration_utilities.consume(it, None)

import simple_benchmark
b = simple_benchmark.BenchmarkBuilder()

def grouper(l, n):
return consume_all(iteration_utilities.grouper(l, n))

def Craz_inner(iterable, n, fillvalue=None):
args = [iter(iterable)] * n
return zip_longest(*args, fillvalue=fillvalue)

def Craz(iterable, n, fillvalue=None):
return consume_all(Craz_inner(iterable, n, fillvalue))

def nosklo_inner(seq, size):
return (seq[pos:pos + size] for pos in range(0, len(seq), size))

def nosklo(seq, size):
return consume_all(nosklo_inner(seq, size))

def SLott_inner(ints, chunk_size):
for i in range(0, len(ints), chunk_size):
yield ints[i:i+chunk_size]

def SLott(ints, chunk_size):
return consume_all(SLott_inner(ints, chunk_size))

def MarkusJarderot1_inner(iterable,size):
it = iter(iterable)
chunk = tuple(itertools.islice(it,size))
while chunk:
yield chunk
chunk = tuple(itertools.islice(it,size))

def MarkusJarderot1(iterable,size):
return consume_all(MarkusJarderot1_inner(iterable,size))

def MarkusJarderot2_inner(iterable,size,filler=None):
it = itertools.chain(iterable,itertools.repeat(filler,size-1))
chunk = tuple(itertools.islice(it,size))
while len(chunk) == size:
yield chunk
chunk = tuple(itertools.islice(it,size))

def MarkusJarderot2(iterable,size):
return consume_all(MarkusJarderot2_inner(iterable,size))

def argument_provider():
for exp in range(2, 20):
size = 2**exp
yield size, simple_benchmark.MultiArgument([[0] * size, 10])

r = b.run()
``````

1 Disclaimer: I’m the author of the libraries `iteration_utilities` and `simple_benchmark`.

I am hoping that by turning an iterator out of a list i am not simply copying a slice of the list. Generators can be sliced and they will automatically still be a generator, while lists will be sliced into huge chunks of 1000 entries, which is less efficient.

``````def iter_group(iterable, batch_size:int):
length = len(iterable)
start = batch_size*-1
end = 0
while(end < length):
start += batch_size
end += batch_size
if type(iterable) == list:
yield (iterable[i] for i in range(start,min(length-1,end)))
else:
yield iterable[start:end]
``````

Usage:

``````items = list(range(1,1251))

for item_group in iter_group(items, 1000):
for item in item_group:
print(item)
``````

Unless I misses something, the following simple solution with generator expressions has not been mentioned. It assumes that both the size and the number of chunks are known (which is often the case), and that no padding is required:

``````def chunks(it, n, m):
"""Make an iterator over m first chunks of size n.
"""
it = iter(it)
# Chunks are presented as tuples.
return (tuple(next(it) for _ in range(n)) for _ in range(m))
``````

Why not use list comprehension

``````l = [1 , 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]
n = 4
filler = 0
fills = len(l) % n
chunks = ((l + [filler] * fills)[x * n:x * n + n] for x in range(int((len(l) + n - 1)/n)))
print(chunks)

[[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 0]]
``````

Since Python 3.8 you can use the walrus `:=` operator and `itertools.islice`.

``````from itertools import islice

list_ = [i for i in range(10, 100)]

def chunker(it, size):
iterator = iter(it)
while chunk := list(islice(iterator, size)):
print(chunk)
``````
``````In [2]: chunker(list_, 10)
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29]
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39]
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59]
[60, 61, 62, 63, 64, 65, 66, 67, 68, 69]
[70, 71, 72, 73, 74, 75, 76, 77, 78, 79]
[80, 81, 82, 83, 84, 85, 86, 87, 88, 89]
[90, 91, 92, 93, 94, 95, 96, 97, 98, 99]

``````

The more-itertools package has chunked method which does exactly that:

``````import more_itertools
for s in more_itertools.chunked(range(9), 4):
print(s)
``````

Prints

``````[0, 1, 2, 3]
[4, 5, 6, 7]
[8]
``````

`chunked` returns the items in a list. If you’d prefer iterables, use ichunked.

here is my go works on lists,iters and range … lazily :

``````def chunker(it,size):
rv = []
for i,el in enumerate(it,1) :
rv.append(el)
if i % size == 0 :
yield rv
rv = []
if rv : yield rv
``````

``````In [95]: list(chunker(range(9),2) )
Out[95]: [[0, 1], [2, 3], [4, 5], [6, 7], [8]]

In [96]: list(chunker([1,2,3,4,5],2) )
Out[96]: [[1, 2], [3, 4], [5]]

In [97]: list(chunker(iter(range(9)),2) )
Out[97]: [[0, 1], [2, 3], [4, 5], [6, 7], [8]]

In [98]: list(chunker(range(9),25) )
Out[98]: [[0, 1, 2, 3, 4, 5, 6, 7, 8]]

In [99]: list(chunker(range(9),1) )
Out[99]: [[0], [1], [2], [3], [4], [5], [6], [7], [8]]

In [101]: %timeit list(chunker(range(101),2) )
11.3 µs ± 68.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
``````

In my special case, I needed to pad the items to repeat the last element until it reaches the size, so I changed this answer to suit my needs.

Example of needed input output with size 4:

``````Input = [1,2,3,4,5,6,7,8]
Output= [[1,2,3,4], [5,6,7,8]]

Input = [[1,2,3,4,5,6,7]]
Output= [[1,2,3,4], [5,6,7,7]]

Input = [1,2,3,4,5]
Output= [[1,2,3,4], [5,5,5,5]]
``````
``````def chunker(seq, size):
res = []
for el in seq:
res.append(el)
if len(res) == size:
yield res
res = []
if res:
res = res + (size - len(res)) * [res[-1]]
yield res
``````