Python dynamic inheritance: How to choose base class upon instance creation?
Question:
Introduction
I have encountered an interesting case in my programming job that requires me to implement a mechanism of dynamic class inheritance in python. What I mean when using the term “dynamic inheritance” is a class that doesn’t inherit from any base class in particular, but rather chooses to inherit from one of several base classes at instantiation, depending on some parameter.
My question is thus the following: in the case I will present, what would be the best, most standard and “pythonic” way of implementing the needed extra functionality via dynamic inheritance.
To summarize the case in point in a simple manner, I will give an example using two classes that represent two different image formats: 'jpg' and 'png' images. I will then try to add the ability to support a third format: the 'gz' image. I realize my question isn’t that simple, but I hope you are ready to bear with me for a few more lines.
The two images example case
This script contains two classes: ImageJPG and ImagePNG, both inheriting
from the Image base class. To create an instance of an image object, the user is asked to call the image_factory function with a file path as the only parameter.
This function then guesses the file format (jpg or png) from the path and
returns an instance of the corresponding class.
Both concrete image classes (ImageJPGand ImagePNG) are able to decode
files via their data property. Both do this in a different way. However,
both ask the Image base class for a file object in order to do this.
import os
#------------------------------------------------------------------------------#
def image_factory(path):
'''Guesses the file format from the file extension
and returns a corresponding image instance.'''
format = os.path.splitext(path)[1][1:]
if format == 'jpg': return ImageJPG(path)
if format == 'png': return ImagePNG(path)
else: raise Exception('The format "' + format + '" is not supported.')
#------------------------------------------------------------------------------#
class Image(object):
'''Fake 1D image object consisting of twelve pixels.'''
def __init__(self, path):
self.path = path
def get_pixel(self, x):
assert x < 12
return self.data[x]
@property
def file_obj(self): return open(self.path, 'r')
#------------------------------------------------------------------------------#
class ImageJPG(Image):
'''Fake JPG image class that parses a file in a given way.'''
@property
def format(self): return 'Joint Photographic Experts Group'
@property
def data(self):
with self.file_obj as f:
f.seek(-50)
return f.read(12)
#------------------------------------------------------------------------------#
class ImagePNG(Image):
'''Fake PNG image class that parses a file in a different way.'''
@property
def format(self): return 'Portable Network Graphics'
@property
def data(self):
with self.file_obj as f:
f.seek(10)
return f.read(12)
################################################################################
i = image_factory('images/lena.png')
print i.format
print i.get_pixel(5)
The compressed image example case
Building on the first image example case, one would like to
add the following functionality:
An extra file format should be supported, the gz format. Instead of
being a new image file format, it is simply a compression layer that,
once decompressed, reveals either a jpg image or a png image.
The image_factory function keeps its working mechanism and will
simply try to create an instance of the concrete image class ImageZIP
when it is given a gz file. Exactly in the same way it would
create an instance of ImageJPG when given a jpg file.
The ImageZIP class just wants to redefine the file_obj property.
In no case does it want to redefine the data property. The crux
of the problem is that, depending on what file format is hiding
inside the zip archive, the ImageZIP classes needs to inherit
either from ImageJPG or from ImagePNG dynamically. The correct class to
inherit from can only be determined upon class creation when the path
parameter is parsed.
Hence, here is the same script with the extra ImageZIP class
and a single added line to the image_factory function.
Obviously, the ImageZIP class is non-functional in this example.
This code requires Python 2.7.
import os, gzip
#------------------------------------------------------------------------------#
def image_factory(path):
'''Guesses the file format from the file extension
and returns a corresponding image instance.'''
format = os.path.splitext(path)[1][1:]
if format == 'jpg': return ImageJPG(path)
if format == 'png': return ImagePNG(path)
if format == 'gz': return ImageZIP(path)
else: raise Exception('The format "' + format + '" is not supported.')
#------------------------------------------------------------------------------#
class Image(object):
'''Fake 1D image object consisting of twelve pixels.'''
def __init__(self, path):
self.path = path
def get_pixel(self, x):
assert x < 12
return self.data[x]
@property
def file_obj(self): return open(self.path, 'r')
#------------------------------------------------------------------------------#
class ImageJPG(Image):
'''Fake JPG image class that parses a file in a given way.'''
@property
def format(self): return 'Joint Photographic Experts Group'
@property
def data(self):
with self.file_obj as f:
f.seek(-50)
return f.read(12)
#------------------------------------------------------------------------------#
class ImagePNG(Image):
'''Fake PNG image class that parses a file in a different way.'''
@property
def format(self): return 'Portable Network Graphics'
@property
def data(self):
with self.file_obj as f:
f.seek(10)
return f.read(12)
#------------------------------------------------------------------------------#
class ImageZIP(### ImageJPG OR ImagePNG ? ###):
'''Class representing a compressed file. Sometimes inherits from
ImageJPG and at other times inherits from ImagePNG'''
@property
def format(self): return 'Compressed ' + super(ImageZIP, self).format
@property
def file_obj(self): return gzip.open(self.path, 'r')
################################################################################
i = image_factory('images/lena.png.gz')
print i.format
print i.get_pixel(5)
A possible solution
I have found a way of getting the wanted behavior by intercepting the __new__ call in the ImageZIP class and using the type function. But it feels clumsy and I suspect there might be a better way using some Python techniques or design patterns I don’t yet know about.
import re
class ImageZIP(object):
'''Class representing a compressed file. Sometimes inherits from
ImageJPG and at other times inherits from ImagePNG'''
def __new__(cls, path):
if cls is ImageZIP:
format = re.findall('(...).gz', path)[-1]
if format == 'jpg': return type("CompressedJPG", (ImageZIP,ImageJPG), {})(path)
if format == 'png': return type("CompressedPNG", (ImageZIP,ImagePNG), {})(path)
else:
return object.__new__(cls)
@property
def format(self): return 'Compressed ' + super(ImageZIP, self).format
@property
def file_obj(self): return gzip.open(self.path, 'r')
Conclusion
Bear in mind if you want to propose a solution that the goal is not to change the behavior of the image_factory function. That function should remain untouched. The goal, ideally, is to build a dynamic ImageZIP class.
I just don’t really know what the best way to do this is. But this is a perfect occasion for me to learn more about some of Python’s “black magic”. Maybe my answer lies with strategies like modifying the self.__cls__ attribute after creation or maybe using the __metaclass__ class attribute? Or maybe something to do with the special abc abstract base classes could help here? Or other unexplored Python territory?
Answers:
I would favor composition over inheritance here. I think your current inheritance hierarchy seems wrong. Some things, like opening the file with or gzip have little to do with the actual image format and can be easily handled in one place while you want to separate the details of working with a specific format own classes. I think using composition you can delegate implementation specific details and have a simple common Image class without requiring metaclasses or multiple inheritance.
import gzip
import struct
class ImageFormat(object):
def __init__(self, fileobj):
self._fileobj = fileobj
@property
def name(self):
raise NotImplementedError
@property
def magic_bytes(self):
raise NotImplementedError
@property
def magic_bytes_format(self):
raise NotImplementedError
def check_format(self):
peek = self._fileobj.read(len(self.magic_bytes_format))
self._fileobj.seek(0)
bytes = struct.unpack_from(self.magic_bytes_format, peek)
if (bytes == self.magic_bytes):
return True
return False
def get_pixel(self, n):
# ...
pass
class JpegFormat(ImageFormat):
name = "JPEG"
magic_bytes = (255, 216, 255, 224, 0, 16, 'J', 'F', 'I', 'F')
magic_bytes_format = "BBBBBBcccc"
class PngFormat(ImageFormat):
name = "PNG"
magic_bytes = (137, 80, 78, 71, 13, 10, 26, 10)
magic_bytes_format = "BBBBBBBB"
class Image(object):
supported_formats = (JpegFormat, PngFormat)
def __init__(self, path):
self.path = path
self._file = self._open()
self._format = self._identify_format()
@property
def format(self):
return self._format.name
def get_pixel(self, n):
return self._format.get_pixel(n)
def _open(self):
opener = open
if self.path.endswith(".gz"):
opener = gzip.open
return opener(self.path, "rb")
def _identify_format(self):
for format in self.supported_formats:
f = format(self._file)
if f.check_format():
return f
else:
raise ValueError("Unsupported file format!")
if __name__=="__main__":
jpeg = Image("images/a.jpg")
png = Image("images/b.png.gz")
I only tested this on a few local png and jpeg files but hopefully it illustrates another way of thinking about this problem.
You should use composition in this case, not inheritance. Take a look at the decorator design pattern. The ImageZIP class should decorate other image classes with the desired functionality.
With decorators, you get a very dynamic behavior depending on the composition that you create:
ImageZIP(ImageJPG(path))
It’s more flexible also, you can have other decorators:
ImageDecrypt(password, ImageZIP(ImageJPG(path)))
Each decorator just encapsulates the functionality it adds and delegates to the composed class as needed.
If you ever need “black magic”, first try to think about a solution that doesn’t require it. You’re likely to find something that works better and results in needs clearer code.
It may be better for the image class constructors to take an already opened file instead of a path.
Then, you’re not limited to files on the disk, but you can use file-like objects from urllib, gzip, and the like.
Also, since you can tell JPG from PNG by looking at the contents of the file, and for gzip file you need this detection anyway, I recommend not looking at the file extension at all.
class Image(object):
def __init__(self, fileobj):
self.fileobj = fileobj
def image_factory(path):
return(image_from_file(open(path, 'rb')))
def image_from_file(fileobj):
if looks_like_png(fileobj):
return ImagePNG(fileobj)
elif looks_like_jpg(fileobj):
return ImageJPG(fileobj)
elif looks_like_gzip(fileobj):
return image_from_file(gzip.GzipFile(fileobj=fileobj))
else:
raise Exception('The format "' + format + '" is not supported.')
def looks_like_png(fileobj):
fileobj.seek(0)
return fileobj.read(4) == 'x89PNG' # or, better, use a library
# etc.
For black magic, go to What is a metaclass in Python?, but think twice before using that, especially at work.
What about defining the ImageZIP class on function-level ?
This will enable your dynamic inheritance.
def image_factory(path):
# ...
if format == ".gz":
image = unpack_gz(path)
format = os.path.splitext(image)[1][1:]
if format == "jpg":
return MakeImageZip(ImageJPG, image)
elif format == "png":
return MakeImageZip(ImagePNG, image)
else: raise Exception('The format "' + format + '" is not supported.')
def MakeImageZIP(base, path):
'''`base` either ImageJPG or ImagePNG.'''
class ImageZIP(base):
# ...
return ImageZIP(path)
Edit: Without need to change image_factory
def ImageZIP(path):
path = unpack_gz(path)
format = os.path.splitext(image)[1][1:]
if format == "jpg": base = ImageJPG
elif format == "png": base = ImagePNG
else: raise_unsupported_format_error()
class ImageZIP(base): # would it be better to use ImageZip_.__name__ = "ImageZIP" ?
# ...
return ImageZIP(path)
Introduction
I have encountered an interesting case in my programming job that requires me to implement a mechanism of dynamic class inheritance in python. What I mean when using the term “dynamic inheritance” is a class that doesn’t inherit from any base class in particular, but rather chooses to inherit from one of several base classes at instantiation, depending on some parameter.
My question is thus the following: in the case I will present, what would be the best, most standard and “pythonic” way of implementing the needed extra functionality via dynamic inheritance.
To summarize the case in point in a simple manner, I will give an example using two classes that represent two different image formats: 'jpg' and 'png' images. I will then try to add the ability to support a third format: the 'gz' image. I realize my question isn’t that simple, but I hope you are ready to bear with me for a few more lines.
The two images example case
This script contains two classes: ImageJPG and ImagePNG, both inheriting
from the Image base class. To create an instance of an image object, the user is asked to call the image_factory function with a file path as the only parameter.
This function then guesses the file format (jpg or png) from the path and
returns an instance of the corresponding class.
Both concrete image classes (ImageJPGand ImagePNG) are able to decode
files via their data property. Both do this in a different way. However,
both ask the Image base class for a file object in order to do this.
import os
#------------------------------------------------------------------------------#
def image_factory(path):
'''Guesses the file format from the file extension
and returns a corresponding image instance.'''
format = os.path.splitext(path)[1][1:]
if format == 'jpg': return ImageJPG(path)
if format == 'png': return ImagePNG(path)
else: raise Exception('The format "' + format + '" is not supported.')
#------------------------------------------------------------------------------#
class Image(object):
'''Fake 1D image object consisting of twelve pixels.'''
def __init__(self, path):
self.path = path
def get_pixel(self, x):
assert x < 12
return self.data[x]
@property
def file_obj(self): return open(self.path, 'r')
#------------------------------------------------------------------------------#
class ImageJPG(Image):
'''Fake JPG image class that parses a file in a given way.'''
@property
def format(self): return 'Joint Photographic Experts Group'
@property
def data(self):
with self.file_obj as f:
f.seek(-50)
return f.read(12)
#------------------------------------------------------------------------------#
class ImagePNG(Image):
'''Fake PNG image class that parses a file in a different way.'''
@property
def format(self): return 'Portable Network Graphics'
@property
def data(self):
with self.file_obj as f:
f.seek(10)
return f.read(12)
################################################################################
i = image_factory('images/lena.png')
print i.format
print i.get_pixel(5)
The compressed image example case
Building on the first image example case, one would like to
add the following functionality:
An extra file format should be supported, the gz format. Instead of
being a new image file format, it is simply a compression layer that,
once decompressed, reveals either a jpg image or a png image.
The image_factory function keeps its working mechanism and will
simply try to create an instance of the concrete image class ImageZIP
when it is given a gz file. Exactly in the same way it would
create an instance of ImageJPG when given a jpg file.
The ImageZIP class just wants to redefine the file_obj property.
In no case does it want to redefine the data property. The crux
of the problem is that, depending on what file format is hiding
inside the zip archive, the ImageZIP classes needs to inherit
either from ImageJPG or from ImagePNG dynamically. The correct class to
inherit from can only be determined upon class creation when the path
parameter is parsed.
Hence, here is the same script with the extra ImageZIP class
and a single added line to the image_factory function.
Obviously, the ImageZIP class is non-functional in this example.
This code requires Python 2.7.
import os, gzip
#------------------------------------------------------------------------------#
def image_factory(path):
'''Guesses the file format from the file extension
and returns a corresponding image instance.'''
format = os.path.splitext(path)[1][1:]
if format == 'jpg': return ImageJPG(path)
if format == 'png': return ImagePNG(path)
if format == 'gz': return ImageZIP(path)
else: raise Exception('The format "' + format + '" is not supported.')
#------------------------------------------------------------------------------#
class Image(object):
'''Fake 1D image object consisting of twelve pixels.'''
def __init__(self, path):
self.path = path
def get_pixel(self, x):
assert x < 12
return self.data[x]
@property
def file_obj(self): return open(self.path, 'r')
#------------------------------------------------------------------------------#
class ImageJPG(Image):
'''Fake JPG image class that parses a file in a given way.'''
@property
def format(self): return 'Joint Photographic Experts Group'
@property
def data(self):
with self.file_obj as f:
f.seek(-50)
return f.read(12)
#------------------------------------------------------------------------------#
class ImagePNG(Image):
'''Fake PNG image class that parses a file in a different way.'''
@property
def format(self): return 'Portable Network Graphics'
@property
def data(self):
with self.file_obj as f:
f.seek(10)
return f.read(12)
#------------------------------------------------------------------------------#
class ImageZIP(### ImageJPG OR ImagePNG ? ###):
'''Class representing a compressed file. Sometimes inherits from
ImageJPG and at other times inherits from ImagePNG'''
@property
def format(self): return 'Compressed ' + super(ImageZIP, self).format
@property
def file_obj(self): return gzip.open(self.path, 'r')
################################################################################
i = image_factory('images/lena.png.gz')
print i.format
print i.get_pixel(5)
A possible solution
I have found a way of getting the wanted behavior by intercepting the __new__ call in the ImageZIP class and using the type function. But it feels clumsy and I suspect there might be a better way using some Python techniques or design patterns I don’t yet know about.
import re
class ImageZIP(object):
'''Class representing a compressed file. Sometimes inherits from
ImageJPG and at other times inherits from ImagePNG'''
def __new__(cls, path):
if cls is ImageZIP:
format = re.findall('(...).gz', path)[-1]
if format == 'jpg': return type("CompressedJPG", (ImageZIP,ImageJPG), {})(path)
if format == 'png': return type("CompressedPNG", (ImageZIP,ImagePNG), {})(path)
else:
return object.__new__(cls)
@property
def format(self): return 'Compressed ' + super(ImageZIP, self).format
@property
def file_obj(self): return gzip.open(self.path, 'r')
Conclusion
Bear in mind if you want to propose a solution that the goal is not to change the behavior of the image_factory function. That function should remain untouched. The goal, ideally, is to build a dynamic ImageZIP class.
I just don’t really know what the best way to do this is. But this is a perfect occasion for me to learn more about some of Python’s “black magic”. Maybe my answer lies with strategies like modifying the self.__cls__ attribute after creation or maybe using the __metaclass__ class attribute? Or maybe something to do with the special abc abstract base classes could help here? Or other unexplored Python territory?
I would favor composition over inheritance here. I think your current inheritance hierarchy seems wrong. Some things, like opening the file with or gzip have little to do with the actual image format and can be easily handled in one place while you want to separate the details of working with a specific format own classes. I think using composition you can delegate implementation specific details and have a simple common Image class without requiring metaclasses or multiple inheritance.
import gzip
import struct
class ImageFormat(object):
def __init__(self, fileobj):
self._fileobj = fileobj
@property
def name(self):
raise NotImplementedError
@property
def magic_bytes(self):
raise NotImplementedError
@property
def magic_bytes_format(self):
raise NotImplementedError
def check_format(self):
peek = self._fileobj.read(len(self.magic_bytes_format))
self._fileobj.seek(0)
bytes = struct.unpack_from(self.magic_bytes_format, peek)
if (bytes == self.magic_bytes):
return True
return False
def get_pixel(self, n):
# ...
pass
class JpegFormat(ImageFormat):
name = "JPEG"
magic_bytes = (255, 216, 255, 224, 0, 16, 'J', 'F', 'I', 'F')
magic_bytes_format = "BBBBBBcccc"
class PngFormat(ImageFormat):
name = "PNG"
magic_bytes = (137, 80, 78, 71, 13, 10, 26, 10)
magic_bytes_format = "BBBBBBBB"
class Image(object):
supported_formats = (JpegFormat, PngFormat)
def __init__(self, path):
self.path = path
self._file = self._open()
self._format = self._identify_format()
@property
def format(self):
return self._format.name
def get_pixel(self, n):
return self._format.get_pixel(n)
def _open(self):
opener = open
if self.path.endswith(".gz"):
opener = gzip.open
return opener(self.path, "rb")
def _identify_format(self):
for format in self.supported_formats:
f = format(self._file)
if f.check_format():
return f
else:
raise ValueError("Unsupported file format!")
if __name__=="__main__":
jpeg = Image("images/a.jpg")
png = Image("images/b.png.gz")
I only tested this on a few local png and jpeg files but hopefully it illustrates another way of thinking about this problem.
You should use composition in this case, not inheritance. Take a look at the decorator design pattern. The ImageZIP class should decorate other image classes with the desired functionality.
With decorators, you get a very dynamic behavior depending on the composition that you create:
ImageZIP(ImageJPG(path))
It’s more flexible also, you can have other decorators:
ImageDecrypt(password, ImageZIP(ImageJPG(path)))
Each decorator just encapsulates the functionality it adds and delegates to the composed class as needed.
If you ever need “black magic”, first try to think about a solution that doesn’t require it. You’re likely to find something that works better and results in needs clearer code.
It may be better for the image class constructors to take an already opened file instead of a path.
Then, you’re not limited to files on the disk, but you can use file-like objects from urllib, gzip, and the like.
Also, since you can tell JPG from PNG by looking at the contents of the file, and for gzip file you need this detection anyway, I recommend not looking at the file extension at all.
class Image(object):
def __init__(self, fileobj):
self.fileobj = fileobj
def image_factory(path):
return(image_from_file(open(path, 'rb')))
def image_from_file(fileobj):
if looks_like_png(fileobj):
return ImagePNG(fileobj)
elif looks_like_jpg(fileobj):
return ImageJPG(fileobj)
elif looks_like_gzip(fileobj):
return image_from_file(gzip.GzipFile(fileobj=fileobj))
else:
raise Exception('The format "' + format + '" is not supported.')
def looks_like_png(fileobj):
fileobj.seek(0)
return fileobj.read(4) == 'x89PNG' # or, better, use a library
# etc.
For black magic, go to What is a metaclass in Python?, but think twice before using that, especially at work.
What about defining the ImageZIP class on function-level ?
This will enable your dynamic inheritance.
def image_factory(path):
# ...
if format == ".gz":
image = unpack_gz(path)
format = os.path.splitext(image)[1][1:]
if format == "jpg":
return MakeImageZip(ImageJPG, image)
elif format == "png":
return MakeImageZip(ImagePNG, image)
else: raise Exception('The format "' + format + '" is not supported.')
def MakeImageZIP(base, path):
'''`base` either ImageJPG or ImagePNG.'''
class ImageZIP(base):
# ...
return ImageZIP(path)
Edit: Without need to change image_factory
def ImageZIP(path):
path = unpack_gz(path)
format = os.path.splitext(image)[1][1:]
if format == "jpg": base = ImageJPG
elif format == "png": base = ImagePNG
else: raise_unsupported_format_error()
class ImageZIP(base): # would it be better to use ImageZip_.__name__ = "ImageZIP" ?
# ...
return ImageZIP(path)