Can I create a "view" on a Python list?
Question:
I have a large list l
. I want to create a view from element 4 to 6. I can do it with sequence slice.
>>> l = range(10)
>>> lv = l[3:6]
>>> lv
[3, 4, 5]
However lv
is a copy of a slice of l
. If I change the underlying list, lv
does not reflect the change.
>>> l[4] = -1
>>> lv
[3, 4, 5]
Vice versa I want modification on lv
reflect in l
as well. Other than that the list size are not going to be changed.
I’m not looking forward to build a big class to do this. I’m just hoping other Python gurus may know some hidden language trick. Ideally I hope it can be like pointer arithmetic in C:
int lv[] = l + 3;
Answers:
You can do that by creating your own generator using the original list reference.
l = [1,2,3,4,5]
lv = (l[i] for i in range(1,4))
lv.next() # 2
l[2]=-1
lv.next() # -1
lv.next() # 4
However this being a generator, you can only go through the list once, forwards and it will explode if you remove more elements than you requested with range
.
There is no “list slice” class in the Python standard library (nor is one built-in). So, you do need a class, though it need not be big — especially if you’re content with a “readonly” and “compact” slice. E.g.:
import collections
class ROListSlice(collections.Sequence):
def __init__(self, alist, start, alen):
self.alist = alist
self.start = start
self.alen = alen
def __len__(self):
return self.alen
def adj(self, i):
if i<0: i += self.alen
return i + self.start
def __getitem__(self, i):
return self.alist[self.adj(i)]
This has some limitations (doesn’t support “slicing a slice”) but for most purposes might be OK.
To make this sequence r/w you need to add __setitem__
, __delitem__
, and insert
:
class ListSlice(ROListSlice):
def __setitem__(self, i, v):
self.alist[self.adj(i)] = v
def __delitem__(self, i, v):
del self.alist[self.adj(i)]
self.alen -= 1
def insert(self, i, v):
self.alist.insert(self.adj(i), v)
self.alen += 1
You could edit: not do something like
shiftedlist = type('ShiftedList',
(list,),
{"__getitem__": lambda self, i: list.__getitem__(self, i + 3)}
)([1, 2, 3, 4, 5, 6])
Being essentially a one-liner, it’s not very Pythonic, but that’s the basic gist.
edit: I’ve belatedly realized that this doesn’t work because list()
will essentially do a shallow copy of the list it’s passed. So this will end up being more or less the same as just slicing the list. Actually less, due to a missing override of __len__
. You’ll need to use a proxy class; see Mr. Martelli’s answer for the details.
Perhaps just use a numpy array:
In [19]: import numpy as np
In [20]: l=np.arange(10)
Basic slicing numpy arrays returns a view, not a copy:
In [21]: lv=l[3:6]
In [22]: lv
Out[22]: array([3, 4, 5])
Altering l
affects lv
:
In [23]: l[4]=-1
In [24]: lv
Out[24]: array([ 3, -1, 5])
And altering lv
affects l
:
In [25]: lv[1]=4
In [26]: l
Out[26]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Edit: The object argument must be an object that supports the buffer call interface (such as strings, arrays, and buffers).
– so no, sadly.
I think buffer type is what you are looking for.
Pasting example from linked page:
>>> s = bytearray(1000000) # a million zeroed bytes
>>> t = buffer(s, 1) # slice cuts off the first byte
>>> s[1] = 5 # set the second element in s
>>> t[0] # which is now also the first element in t!
'x05'
As soon as you will take a slice from a list, you will be creating a new list. Ok, it will contain same objects so as long as objects of the list are concerned it would be the same, but if you modify a slice the original list is unchanged.
If you really want to create a modifiable view, you could imagine a new class based on collection.MutableSequence
This could be a starting point for a full featured sub list – it correctly processes slice indexes, but at least is lacking specification for negative indexes processing:
class Sublist(collections.MutableSequence):
def __init__(self, ls, beg, end):
self.ls = ls
self.beg = beg
self.end = end
def __getitem__(self, i):
self._valid(i)
return self.ls[self._newindex(i)]
def __delitem__(self, i):
self._valid(i)
del self.ls[self._newindex(i)]
def insert(self, i, x):
self._valid(i)
self.ls.insert(i+ self.beg, x)
def __len__(self):
return self.end - self.beg
def __setitem__(self, i, x):
self.ls[self._newindex(i)] = x
def _valid(self, i):
if isinstance(i, slice):
self._valid(i.start)
self._valid(i.stop)
elif isinstance(i, int):
if i<0 or i>=self.__len__():
raise IndexError()
else:
raise TypeError()
def _newindex(self, i):
if isinstance(i, slice):
return slice(self.beg + i.start, self.beg + i.stop, i.step)
else:
return i + self.beg
Example:
>>> a = list(range(10))
>>> s = Sublist(a, 3, 8)
>>> s[2:4]
[5, 6]
>>> s[2] = 15
>>> a
[0, 1, 2, 3, 4, 15, 6, 7, 8, 9]
If you are going to be accessing the “view” sequentially then you can just use itertools.islice(..)You can see the documentation for more info.
l = [1, 2, 3, 4, 5]
d = [1:3] #[2, 3]
d = itertools.islice(2, 3) # iterator yielding -> 2, 3
You can’t access individual elements to change them in the slice and if you do change the list you have to re-call isclice(..).
Subclass the more_itertools.SequenceView
to affect views by mutating sequences and vice versa.
Code
import more_itertools as mit
class SequenceView(mit.SequenceView):
"""Overload assignments in views."""
def __setitem__(self, index, item):
self._target[index] = item
Demo
>>> seq = list(range(10))
>>> view = SequenceView(seq)
>>> view
SequenceView([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> # Mutate Sequence -> Affect View
>>> seq[6] = -1
>>> view[5:8]
[5, -1, 7]
>>> # Mutate View -> Affect Sequence
>>> view[5] = -2
>>> seq[5:8]
[-2, -1, 7]
more_itertools
is a third-party library. Install via > pip install more_itertools
.
https://gist.github.com/mathieucaroff/0cf094325fb5294fb54c6a577f05a2c1
Above link is a solution based on python 3 range ability to be sliced and
indexed in constant time.
It supports slicing, equality comparsion, string casting (__str__
), and
reproducers (__repr__
), but doesn’t support assigment.
Creating a SliceableSequenceView of a SliceableSequenceView won’t slow down
access times as this case is detected.
sequenceView.py
# stackoverflow.com/q/3485475/can-i-create-a-view-on-a-python-list
try:
from collections.abc import Sequence
except ImportError:
from collections import Sequence # pylint: disable=no-name-in-module
class SliceableSequenceView(Sequence):
"""
A read-only sequence which allows slicing without copying the viewed list.
Supports negative indexes.
Usage:
li = list(range(100))
s = SliceableSequenceView(li)
u = SliceableSequenceView(li, slice(1,7,2))
v = s[1:7:2]
w = s[-99:-93:2]
li[1] += 10
assert li[1:7:2] == list(u) == list(v) == list(w)
"""
__slots__ = "seq range".split()
def __init__(self, seq, sliced=None):
"""
Accept any sequence (such as lists, strings or ranges).
"""
if sliced is None:
sliced = slice(len(seq))
ls = looksSliceable = True
ls = ls and hasattr(seq, "seq") and isinstance(seq.seq, Sequence)
ls = ls and hasattr(seq, "range") and isinstance(seq.range, range)
looksSliceable = ls
if looksSliceable:
self.seq = seq.seq
self.range = seq.range[sliced]
else:
self.seq = seq
self.range = range(len(seq))[sliced]
def __len__(self):
return len(self.range)
def __getitem__(self, i):
if isinstance(i, slice):
return SliceableSequenceView(self.seq, i)
return self.seq[self.range[i]]
def __str__(self):
r = self.range
s = slice(r.start, r.stop, r.step)
return str(self.seq[s])
def __repr__(self):
r = self.range
s = slice(r.start, r.stop, r.step)
return "SliceableSequenceView({!r})".format(self.seq[s])
def equal(self, otherSequence):
if self is otherSequence:
return True
if len(self) != len(otherSequence):
return False
for v, w in zip(self, otherSequence):
if v != w:
return False
return True
It’s actually not too difficult to implement this yourself using range
.* You can slice a range and it does all of the complicated arithmetic for you:
>>> range(20)[10:]
range(10, 20)
>>> range(10, 20)[::2]
range(10, 20, 2)
>>> range(10, 20, 2)[::-3]
range(18, 8, -6)
So you just need a class of object that contains a reference to the original sequence, and a range. Here is the code for such a class (not too big, I hope):
class SequenceView:
def __init__(self, sequence, range_object=None):
if range_object is None:
range_object = range(len(sequence))
self.range = range_object
self.sequence = sequence
def __getitem__(self, key):
if type(key) == slice:
return SequenceView(self.sequence, self.range[key])
else:
return self.sequence[self.range[key]]
def __setitem__(self, key, value):
self.sequence[self.range[key]] = value
def __len__(self):
return len(self.range)
def __iter__(self):
for i in self.range:
yield self.sequence[i]
def __repr__(self):
return f"SequenceView({self.sequence!r}, {self.range!r})"
def __str__(self):
if type(self.sequence) == str:
return ''.join(self)
elif type(self.sequence) in (list, tuple):
return str(type(self.sequence)(self))
else:
return repr(self)
(This was bodged together in about 5 minutes, so make sure you test it thoroughly before using it anywhere important.)
Usage:
>>> p = list(range(10))
>>> q = SequenceView(p)[3:6]
>>> print(q)
[3, 4, 5]
>>> q[1] = -1
>>> print(q)
[3, -1, 5]
>>> print(p)
[0, 1, 2, 3, -1, 5, 6, 7, 8, 9]
* in Python 3
I have a large list l
. I want to create a view from element 4 to 6. I can do it with sequence slice.
>>> l = range(10)
>>> lv = l[3:6]
>>> lv
[3, 4, 5]
However lv
is a copy of a slice of l
. If I change the underlying list, lv
does not reflect the change.
>>> l[4] = -1
>>> lv
[3, 4, 5]
Vice versa I want modification on lv
reflect in l
as well. Other than that the list size are not going to be changed.
I’m not looking forward to build a big class to do this. I’m just hoping other Python gurus may know some hidden language trick. Ideally I hope it can be like pointer arithmetic in C:
int lv[] = l + 3;
You can do that by creating your own generator using the original list reference.
l = [1,2,3,4,5]
lv = (l[i] for i in range(1,4))
lv.next() # 2
l[2]=-1
lv.next() # -1
lv.next() # 4
However this being a generator, you can only go through the list once, forwards and it will explode if you remove more elements than you requested with range
.
There is no “list slice” class in the Python standard library (nor is one built-in). So, you do need a class, though it need not be big — especially if you’re content with a “readonly” and “compact” slice. E.g.:
import collections
class ROListSlice(collections.Sequence):
def __init__(self, alist, start, alen):
self.alist = alist
self.start = start
self.alen = alen
def __len__(self):
return self.alen
def adj(self, i):
if i<0: i += self.alen
return i + self.start
def __getitem__(self, i):
return self.alist[self.adj(i)]
This has some limitations (doesn’t support “slicing a slice”) but for most purposes might be OK.
To make this sequence r/w you need to add __setitem__
, __delitem__
, and insert
:
class ListSlice(ROListSlice):
def __setitem__(self, i, v):
self.alist[self.adj(i)] = v
def __delitem__(self, i, v):
del self.alist[self.adj(i)]
self.alen -= 1
def insert(self, i, v):
self.alist.insert(self.adj(i), v)
self.alen += 1
You could edit: not do something like
shiftedlist = type('ShiftedList',
(list,),
{"__getitem__": lambda self, i: list.__getitem__(self, i + 3)}
)([1, 2, 3, 4, 5, 6])
Being essentially a one-liner, it’s not very Pythonic, but that’s the basic gist.
edit: I’ve belatedly realized that this doesn’t work because list()
will essentially do a shallow copy of the list it’s passed. So this will end up being more or less the same as just slicing the list. Actually less, due to a missing override of __len__
. You’ll need to use a proxy class; see Mr. Martelli’s answer for the details.
Perhaps just use a numpy array:
In [19]: import numpy as np
In [20]: l=np.arange(10)
Basic slicing numpy arrays returns a view, not a copy:
In [21]: lv=l[3:6]
In [22]: lv
Out[22]: array([3, 4, 5])
Altering l
affects lv
:
In [23]: l[4]=-1
In [24]: lv
Out[24]: array([ 3, -1, 5])
And altering lv
affects l
:
In [25]: lv[1]=4
In [26]: l
Out[26]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
Edit: The object argument must be an object that supports the buffer call interface (such as strings, arrays, and buffers).
– so no, sadly.
I think buffer type is what you are looking for.
Pasting example from linked page:
>>> s = bytearray(1000000) # a million zeroed bytes
>>> t = buffer(s, 1) # slice cuts off the first byte
>>> s[1] = 5 # set the second element in s
>>> t[0] # which is now also the first element in t!
'x05'
As soon as you will take a slice from a list, you will be creating a new list. Ok, it will contain same objects so as long as objects of the list are concerned it would be the same, but if you modify a slice the original list is unchanged.
If you really want to create a modifiable view, you could imagine a new class based on collection.MutableSequence
This could be a starting point for a full featured sub list – it correctly processes slice indexes, but at least is lacking specification for negative indexes processing:
class Sublist(collections.MutableSequence):
def __init__(self, ls, beg, end):
self.ls = ls
self.beg = beg
self.end = end
def __getitem__(self, i):
self._valid(i)
return self.ls[self._newindex(i)]
def __delitem__(self, i):
self._valid(i)
del self.ls[self._newindex(i)]
def insert(self, i, x):
self._valid(i)
self.ls.insert(i+ self.beg, x)
def __len__(self):
return self.end - self.beg
def __setitem__(self, i, x):
self.ls[self._newindex(i)] = x
def _valid(self, i):
if isinstance(i, slice):
self._valid(i.start)
self._valid(i.stop)
elif isinstance(i, int):
if i<0 or i>=self.__len__():
raise IndexError()
else:
raise TypeError()
def _newindex(self, i):
if isinstance(i, slice):
return slice(self.beg + i.start, self.beg + i.stop, i.step)
else:
return i + self.beg
Example:
>>> a = list(range(10))
>>> s = Sublist(a, 3, 8)
>>> s[2:4]
[5, 6]
>>> s[2] = 15
>>> a
[0, 1, 2, 3, 4, 15, 6, 7, 8, 9]
If you are going to be accessing the “view” sequentially then you can just use itertools.islice(..)You can see the documentation for more info.
l = [1, 2, 3, 4, 5]
d = [1:3] #[2, 3]
d = itertools.islice(2, 3) # iterator yielding -> 2, 3
You can’t access individual elements to change them in the slice and if you do change the list you have to re-call isclice(..).
Subclass the more_itertools.SequenceView
to affect views by mutating sequences and vice versa.
Code
import more_itertools as mit
class SequenceView(mit.SequenceView):
"""Overload assignments in views."""
def __setitem__(self, index, item):
self._target[index] = item
Demo
>>> seq = list(range(10))
>>> view = SequenceView(seq)
>>> view
SequenceView([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> # Mutate Sequence -> Affect View
>>> seq[6] = -1
>>> view[5:8]
[5, -1, 7]
>>> # Mutate View -> Affect Sequence
>>> view[5] = -2
>>> seq[5:8]
[-2, -1, 7]
more_itertools
is a third-party library. Install via > pip install more_itertools
.
https://gist.github.com/mathieucaroff/0cf094325fb5294fb54c6a577f05a2c1
Above link is a solution based on python 3 range ability to be sliced and
indexed in constant time.
It supports slicing, equality comparsion, string casting (__str__
), and
reproducers (__repr__
), but doesn’t support assigment.
Creating a SliceableSequenceView of a SliceableSequenceView won’t slow down
access times as this case is detected.
sequenceView.py
# stackoverflow.com/q/3485475/can-i-create-a-view-on-a-python-list
try:
from collections.abc import Sequence
except ImportError:
from collections import Sequence # pylint: disable=no-name-in-module
class SliceableSequenceView(Sequence):
"""
A read-only sequence which allows slicing without copying the viewed list.
Supports negative indexes.
Usage:
li = list(range(100))
s = SliceableSequenceView(li)
u = SliceableSequenceView(li, slice(1,7,2))
v = s[1:7:2]
w = s[-99:-93:2]
li[1] += 10
assert li[1:7:2] == list(u) == list(v) == list(w)
"""
__slots__ = "seq range".split()
def __init__(self, seq, sliced=None):
"""
Accept any sequence (such as lists, strings or ranges).
"""
if sliced is None:
sliced = slice(len(seq))
ls = looksSliceable = True
ls = ls and hasattr(seq, "seq") and isinstance(seq.seq, Sequence)
ls = ls and hasattr(seq, "range") and isinstance(seq.range, range)
looksSliceable = ls
if looksSliceable:
self.seq = seq.seq
self.range = seq.range[sliced]
else:
self.seq = seq
self.range = range(len(seq))[sliced]
def __len__(self):
return len(self.range)
def __getitem__(self, i):
if isinstance(i, slice):
return SliceableSequenceView(self.seq, i)
return self.seq[self.range[i]]
def __str__(self):
r = self.range
s = slice(r.start, r.stop, r.step)
return str(self.seq[s])
def __repr__(self):
r = self.range
s = slice(r.start, r.stop, r.step)
return "SliceableSequenceView({!r})".format(self.seq[s])
def equal(self, otherSequence):
if self is otherSequence:
return True
if len(self) != len(otherSequence):
return False
for v, w in zip(self, otherSequence):
if v != w:
return False
return True
It’s actually not too difficult to implement this yourself using range
.* You can slice a range and it does all of the complicated arithmetic for you:
>>> range(20)[10:]
range(10, 20)
>>> range(10, 20)[::2]
range(10, 20, 2)
>>> range(10, 20, 2)[::-3]
range(18, 8, -6)
So you just need a class of object that contains a reference to the original sequence, and a range. Here is the code for such a class (not too big, I hope):
class SequenceView:
def __init__(self, sequence, range_object=None):
if range_object is None:
range_object = range(len(sequence))
self.range = range_object
self.sequence = sequence
def __getitem__(self, key):
if type(key) == slice:
return SequenceView(self.sequence, self.range[key])
else:
return self.sequence[self.range[key]]
def __setitem__(self, key, value):
self.sequence[self.range[key]] = value
def __len__(self):
return len(self.range)
def __iter__(self):
for i in self.range:
yield self.sequence[i]
def __repr__(self):
return f"SequenceView({self.sequence!r}, {self.range!r})"
def __str__(self):
if type(self.sequence) == str:
return ''.join(self)
elif type(self.sequence) in (list, tuple):
return str(type(self.sequence)(self))
else:
return repr(self)
(This was bodged together in about 5 minutes, so make sure you test it thoroughly before using it anywhere important.)
Usage:
>>> p = list(range(10))
>>> q = SequenceView(p)[3:6]
>>> print(q)
[3, 4, 5]
>>> q[1] = -1
>>> print(q)
[3, -1, 5]
>>> print(p)
[0, 1, 2, 3, -1, 5, 6, 7, 8, 9]
* in Python 3