Understanding __getitem__ method

The [] syntax for getting item by key or index is just syntax sugar.

When you evaluate a[i] Python calls a.__getitem__(i) (or type(a).__getitem__(a, i), but this distinction is about inheritance models and is not important here). Even if the class of a may not explicitly define this method, it is usually inherited from an ancestor class.

All the (Python 2.7) special method names and their semantics are listed here: https://docs.python.org/2.7/reference/datamodel.html#special-method-names

The magic method __getitem__ is basically used for accessing list items, dictionary entries, array elements etc. It is very useful for a quick lookup of instance attributes.

Here I am showing this with an example class Person that can be instantiated by ‘name’, ‘age’, and ‘dob’ (date of birth). The __getitem__ method is written in a way that one can access the indexed instance attributes, such as first or last name, day, month or year of the dob, etc.

import copy

# Constants that can be used to index date of birth's Date-Month-Year
D = 0; M = 1; Y = -1

class Person(object):
    def __init__(self, name, age, dob):
        self.name = name
        self.age = age
        self.dob = dob

    def __getitem__(self, indx):
        print ("Calling __getitem__")
        p = copy.copy(self)

        p.name = p.name.split(" ")[indx]
        p.dob = p.dob[indx] # or, p.dob = p.dob.__getitem__(indx)
        return p

Suppose one user input is as follows:

p = Person(name = 'Jonab Gutu', age = 20, dob=(1, 1, 1999))

With the help of __getitem__ method, the user can access the indexed attributes. e.g.,

print p[0].name # print first (or last) name
print p[Y].dob  # print (Date or Month or ) Year of the 'date of birth'

For readability and consistency. That question is part of why operator overloading exists, since __getitem__ is one of the functions that implement that.

If you get an unknown class, written by an unknown author, and you want to add its 3rd element to its 5th element, you can very well assume that obj[3] + obj[5] will work.

What would that line look like in a language that does not support operator overloading?? Probably something like obj.get(3).add(obj.get(5))?? Or maybe obj.index(3).plus(obj.index(5))??

The problem with the second approach is that (1) it’s much less readable and (2) you can’t guess, you have to look up the documentation.

A common library that uses this technique is the ’email’ module. It uses the __getitem__ method in the email.message.Message class, which in turn is inherited by MIME-related classes.

Then in the and all you need to get a valid MIME-type message with sane defaults is add your headers. There’s a lot more going on under the hood but the usage is simple.

message = MIMEText(message_text)
message['to'] = to
message['from'] = sender
message['subject'] = subject

As a side note, the __getitem__ method also allows you to turn your object into an iterable.

Example: if used with iter(), it can generate as many int squared values as you want:

class MyIterable:
    def __getitem__(self, index):
        return index ** 2


obj = MyIterable()
obj_iter = iter(obj)

for i in range(1000):
    print(next(obj_iter))

Django core has several interesting and nifty usages for magic methods, including __getitem__. These were my recent finds:

1. Django HTTP Request

   • When you submit GET/POST data in Django, it will be stored in Django’s request object as request.GET/request.POST dict. This dict is of type QueryDict which inherits from MultiValueDict.

   • When you submit data, say user_id=42, QueryDict will be stored/represented as:

     <QueryDict: {'user_id': ['42']}>

     So, the passed data becomes

     'user_id': ['42']

     instead of the intuitive

     'user_id': '42'

     MultiValueDict‘s docstring explains though why it needs to auto-convert this to list format:

     This class exists to solve the irritating problem raised by cgi.parse_qs, which returns a list for every key..

   • Given that the QueryDict values are transformed into lists, they will need to be accessed then like this (same idea with request.GET):

     • request.POST['user_id'][0]

     • request.POST['user_id'][-1]

     • request.POST.get('user_id')[0]

     • request.POST.get('user_id)[-1]

       But, these are horrible ways to access the data. So. Django overridden the __getitem__ and __get__ in MultiValueDict. This is the simplified version:

       def __getitem__(self, key):
           """
           Accesses the list value automatically 
           using the `-1` list index.
           """
           list_ = super().__getitem__(key)
           return list_[-1]
       
       def get(self, key, default=None):
           """
           Just calls the `__getitem__` above.
           """
           return self[key]
       

       With these, you could now have a more intuitive accessors:

       • request.POST['user_id']
       • request.POST.get('user_id')

2. Django Forms

   • In Django, you could declare forms like this (includes ModelForm):

     class ArticleForm(...):
         title = ...
     

   • These forms inherit from BaseForm, and have these overridden magic methods (simplified version):

     def __iter__(self):
        for name in self.fields:
            yield self[name]
     
     def __getitem__(self, name):
         return self.fields[name]
     

     resulting to these convenient patterns:

     # Instead of `for field in form.fields`.
     # This is a common pattern in Django templates.
     for field in form
         ...
     
     # Instead of `title = form.fields['title']`
     title = form['title']
     

In summary, magic methods (or their overrides) increase code readability and developer experience/convenience.

The use of __getitem__ includes implementing control flow measures that for some weird reason cannot be performed lower in the execution stack:

class HeavenlyList(list):
    """don't let caller get 666th element"""
    
    def __getitem__(self, key):
        """return element"""
        if isinstance(key, slice):
            return [
                super().__getitem__(i)
                for i in range(key.start, key.stop, key.step)
                if i != 666
            ]
        return super().__getitem__(key) if key != 666 else None

A similar, but more interesting reason is to allow slice-based access to elements in container/sequence types that ordinarily don’t allow it:

class SliceDict(dict):
    """handles slices"""
    
    def __setitem__(self, key, value):
        """map key to value"""
        if not isinstance(key, int)
            raise TypeError("key must be an integer")
        super().__setitem__(key, value)
    
    def __getitem__(self, key):
        """return value(s)"""
        if not isinstance(key, slice):
            return super().__getitem__(key)
        return [
            super().__getitem__(i)
            for i in range(key.start, key.stop, key.step)
        ]

Another interesting use is overriding str.__getitem__ to accept str objects as well as ints and slices, such that the str input is a regular expression, and the return value is the match object iterator returned by re.finditer:

from re import finditer

class REString(str):
    """handles regular expressions"""
    
    re_flags = 0
    
    def __getitem__(self, key):
        """return some/all of string or re.finditer"""
        if isinstance(key, str):
            return finditer(key, self, flags=self.re_flags)
        return super().__getitem__(key)

A real-world problem where overriding dict.__getitem__ in particular proves useful is when a program requires information that is distributed over the internet and available over HTTP. Because these information are remote, the process can employ some level of laziness– only retrieving data for items it doesn’t have or that have changed. The specific example is having a dictionary instance lazily retrieve and store Python Enhancement Proposals. There are many of these documents, sometimes they are revised, and they all reside on hosts known by the domain name peps.python.org. Therefore the idea is to make a HTTP GET request for the PEP number passed into __getitem__, fetching it if the dictionary doesn’t already contain it or the PEPs HTTP ETAG changed.

from http import HTTPStatus, client


class PEPDict(dict):
    """lazy PEP container"""
    
    conn = client.HTTPSConnection("peps.python.org")
    
    def __getitem__(self, pep):
        """return pep pep"""
        
        # if lazy for too long
        if self.conn.sock is None:
            self.conn.connect()
        
        # build etag check in request header
        requestheaders = dict()
        if pep in self:
            requestheaders = {
                "if-none-match": super().__getitem__(pep)[0]
            }
        
        # make request and fetch response
        self.conn.request(
            "GET",
            "/%s/" % str(pep).zfill(4),
            headers=requestheaders
        )
        response = self.conn.getresponse()
        
        # (re)set the pep
        if response.status = HTTPStatus.OK:
            self.__setitem__(
                pep, (
                    response.getheader("etag"),
                    response.read()
                )
            )
        
        # raise if status is not ok or not modified
        if response.status != HTTPStatus.NOT_MODIFIED:
            raise Exception("something weird happened")
        
        return super().__getitem__(pep)[1]
        

A good resource for understanding further what is the use of it is to review its associated special/dunder methods in the emulating container types section of Python’s data model document.

OK I’ll just leave this here. OP questions the very basics of software engineering.

This is about defining class interface. Consistency, readability or whatever else is secondary.

First of all this is about how different parts of the project can talk to your object.

Imagine function which calls [] on some object. Now you are tasked to do exactly what this function does with some new type object that you have. But your object is not a list or dict, or tuple.

Now you don’t need to implement anything but define a __getitem__ for the class of your object.

Interfaces create building blocks out of bunch of internal implementations. Define them wisely.

Further examples of more complex cases

The following example shows exactly what you get when calling []/__getitem__ with various inputs, which should help clarify how it works:

class C(object):
    def __getitem__(self, k):
        return k

# Single argument is passed directly.
assert C()[0] == 0

# Multiple indices generate a tuple.
assert C()[0, 1] == (0, 1)

# Slice notation generates a slice object.
assert C()[1:2:3] == slice(1, 2, 3)

# Empty slice entries become None.
assert C()[:2:] == slice(None, 2, None)

# Ellipsis notation generates the Ellipsis class object.
# Ellipsis is a singleton, so we can compare with `is`.
assert C()[...] is Ellipsis

# Everything mixed up.
assert C()[1, 2:3:4, ..., 6, :7:, ..., 8] == 
       (1, slice(2,3,4), Ellipsis, 6, slice(None,7,None), Ellipsis, 8)

What you do with the argument of __getitem__ is then arbitrary. Of course, anything besides array-like indexing would likely make for an insane API. But nothing prevents you from going wild!

I also covered Ellipsis at: What does the Ellipsis object do?

Tested in Python 3.5.2 and 2.7.12.