Python – mechanism to identify compressed file type and uncompress
Question:
A compressed file can be classified into below logical groups
a. The operating system which you are working on (*ix, Win) etc.
b. Different types of compression algorithm (i.e .zip,.Z,.bz2,.rar,.gzip). Atleast from a standard list of mostly used compressed files.
c. Then we have tar ball mechanism – where I suppose there are no compression. But it acts more like a concatenation.
Now, if we start addressing the above set of compressed files,
a. Option (a) would be taken care by python since it is platform independent language.
b. Option (b) and (c) seems to have a problem.
What do I need
How do I identify the file type (compression type) and then UN-compress them?
Like:
fileType = getFileType(fileName)
switch(fileType):
case .rar: unrar....
case .zip: unzip....
etc
So the fundamental question is how do we identify the compression algorithm based on the file (assuming the extension is not provided or incorrect)? Is there any specific way to do it in python?
Answers:
“a” is completely false.
“b” can be easily interpreted badly, as “.zip” doesn’t mean the file is actually a zip file. It could be a JPEG with zip extension (for confusing purposes, if you want).
You actually need to check if the data inside the file matches the data expected to have by it’s extension.
Also have a look at magic byte.
This is a complex question that depends on a number of factors: the most important being how portable your solution needs to be.
The basics behind finding the file type given a file is to find an identifying header in the file, usually something called a “magic sequence” or signature header, which identifies that a file is of a certain type. Its name or extension is usually not used if it can be avoided. For some files, Python has this built in. For example, to deal with .tar files, you can use the tarfile module, which has a convenient is_tarfile method. There is a similar module named zipfile. These modules will also let you extract files in pure Python.
For example:
f = file('myfile','r')
if zipfile.is_zipfile(f):
zip = zipfile.ZipFile(f)
zip.extractall('/dest/dir')
elif tarfile.is_tarfile(f):
...
If your solution is Linux or OSX only, there is also the file command which will do a lot of the work for you. You can also use the built-in tools to uncompress the files. If you are just doing a simple script, this method is simpler and will give you better performance.
This page has a list of “magic” file signatures. Grab the ones you need and put them in a dict like below. Then we need a function that matches the dict keys with the start of the file. I’ve written a suggestion, though it can be optimized by preprocessing the magic_dict into e.g. one giant compiled regexp.
magic_dict = {
"x1fx8bx08": "gz",
"x42x5ax68": "bz2",
"x50x4bx03x04": "zip"
}
max_len = max(len(x) for x in magic_dict)
def file_type(filename):
with open(filename) as f:
file_start = f.read(max_len)
for magic, filetype in magic_dict.items():
if file_start.startswith(magic):
return filetype
return "no match"
This solution should be cross-plattform and is of course not dependent on file name extension, but it may give false positives for files with random content that just happen to start with some specific magic bytes.
Based on lazyr’s answer and my comment, here is what I mean:
class CompressedFile (object):
magic = None
file_type = None
mime_type = None
proper_extension = None
def __init__(self, f):
# f is an open file or file like object
self.f = f
self.accessor = self.open()
@classmethod
def is_magic(self, data):
return data.startswith(self.magic)
def open(self):
return None
import zipfile
class ZIPFile (CompressedFile):
magic = 'x50x4bx03x04'
file_type = 'zip'
mime_type = 'compressed/zip'
def open(self):
return zipfile.ZipFile(self.f)
import bz2
class BZ2File (CompressedFile):
magic = 'x42x5ax68'
file_type = 'bz2'
mime_type = 'compressed/bz2'
def open(self):
return bz2.BZ2File(self.f)
import gzip
class GZFile (CompressedFile):
magic = 'x1fx8bx08'
file_type = 'gz'
mime_type = 'compressed/gz'
def open(self):
return gzip.GzipFile(self.f)
# factory function to create a suitable instance for accessing files
def get_compressed_file(filename):
with file(filename, 'rb') as f:
start_of_file = f.read(1024)
f.seek(0)
for cls in (ZIPFile, BZ2File, GZFile):
if cls.is_magic(start_of_file):
return cls(f)
return None
filename='test.zip'
cf = get_compressed_file(filename)
if cf is not None:
print filename, 'is a', cf.mime_type, 'file'
print cf.accessor
Can now access the compressed data using cf.accessor. All the modules provide similar methods like ‘read()’, ‘write()’, etc. to do this.
If the exercise is to identify it just to label files, you have lots of answers. If you want to uncompress the archive, why don’t you just try and catch the execptions/errors? For example:
>>> tarfile.is_tarfile('lala.txt')
False
>>> zipfile.is_zipfile('lala.txt')
False
>>> with bz2.BZ2File('startup.bat','r') as f:
... f.read()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
IOError: invalid data stream
2019 update:
I was looking for a solution to detect if a .csv file was gzipped or not. The answer @Lauritz gave was throwing errors for me, i imagine it’s just because the way files are read has changed in the past 7 years.
This library worked perfectly for me!
https://pypi.org/project/filetype/
The accepted solution looks great, but it doesn’t work with python-3, here are the modifications that made it work — using binary I/O instead of strings:
magic_dict = {
b"x1fx8bx08": "gz",
b"x42x5ax68": "bz2",
b"x50x4bx03x04": "zip"
}
''' SKIP '''
with open(filename, "rb") as f:
''' The rest is the same '''
A compressed file can be classified into below logical groups
a. The operating system which you are working on (*ix, Win) etc.
b. Different types of compression algorithm (i.e .zip,.Z,.bz2,.rar,.gzip). Atleast from a standard list of mostly used compressed files.
c. Then we have tar ball mechanism – where I suppose there are no compression. But it acts more like a concatenation.
Now, if we start addressing the above set of compressed files,
a. Option (a) would be taken care by python since it is platform independent language.
b. Option (b) and (c) seems to have a problem.
What do I need
How do I identify the file type (compression type) and then UN-compress them?
Like:
fileType = getFileType(fileName)
switch(fileType):
case .rar: unrar....
case .zip: unzip....
etc
So the fundamental question is how do we identify the compression algorithm based on the file (assuming the extension is not provided or incorrect)? Is there any specific way to do it in python?
“a” is completely false.
“b” can be easily interpreted badly, as “.zip” doesn’t mean the file is actually a zip file. It could be a JPEG with zip extension (for confusing purposes, if you want).
You actually need to check if the data inside the file matches the data expected to have by it’s extension.
Also have a look at magic byte.
This is a complex question that depends on a number of factors: the most important being how portable your solution needs to be.
The basics behind finding the file type given a file is to find an identifying header in the file, usually something called a “magic sequence” or signature header, which identifies that a file is of a certain type. Its name or extension is usually not used if it can be avoided. For some files, Python has this built in. For example, to deal with .tar files, you can use the tarfile module, which has a convenient is_tarfile method. There is a similar module named zipfile. These modules will also let you extract files in pure Python.
For example:
f = file('myfile','r')
if zipfile.is_zipfile(f):
zip = zipfile.ZipFile(f)
zip.extractall('/dest/dir')
elif tarfile.is_tarfile(f):
...
If your solution is Linux or OSX only, there is also the file command which will do a lot of the work for you. You can also use the built-in tools to uncompress the files. If you are just doing a simple script, this method is simpler and will give you better performance.
This page has a list of “magic” file signatures. Grab the ones you need and put them in a dict like below. Then we need a function that matches the dict keys with the start of the file. I’ve written a suggestion, though it can be optimized by preprocessing the magic_dict into e.g. one giant compiled regexp.
magic_dict = {
"x1fx8bx08": "gz",
"x42x5ax68": "bz2",
"x50x4bx03x04": "zip"
}
max_len = max(len(x) for x in magic_dict)
def file_type(filename):
with open(filename) as f:
file_start = f.read(max_len)
for magic, filetype in magic_dict.items():
if file_start.startswith(magic):
return filetype
return "no match"
This solution should be cross-plattform and is of course not dependent on file name extension, but it may give false positives for files with random content that just happen to start with some specific magic bytes.
Based on lazyr’s answer and my comment, here is what I mean:
class CompressedFile (object):
magic = None
file_type = None
mime_type = None
proper_extension = None
def __init__(self, f):
# f is an open file or file like object
self.f = f
self.accessor = self.open()
@classmethod
def is_magic(self, data):
return data.startswith(self.magic)
def open(self):
return None
import zipfile
class ZIPFile (CompressedFile):
magic = 'x50x4bx03x04'
file_type = 'zip'
mime_type = 'compressed/zip'
def open(self):
return zipfile.ZipFile(self.f)
import bz2
class BZ2File (CompressedFile):
magic = 'x42x5ax68'
file_type = 'bz2'
mime_type = 'compressed/bz2'
def open(self):
return bz2.BZ2File(self.f)
import gzip
class GZFile (CompressedFile):
magic = 'x1fx8bx08'
file_type = 'gz'
mime_type = 'compressed/gz'
def open(self):
return gzip.GzipFile(self.f)
# factory function to create a suitable instance for accessing files
def get_compressed_file(filename):
with file(filename, 'rb') as f:
start_of_file = f.read(1024)
f.seek(0)
for cls in (ZIPFile, BZ2File, GZFile):
if cls.is_magic(start_of_file):
return cls(f)
return None
filename='test.zip'
cf = get_compressed_file(filename)
if cf is not None:
print filename, 'is a', cf.mime_type, 'file'
print cf.accessor
Can now access the compressed data using cf.accessor. All the modules provide similar methods like ‘read()’, ‘write()’, etc. to do this.
If the exercise is to identify it just to label files, you have lots of answers. If you want to uncompress the archive, why don’t you just try and catch the execptions/errors? For example:
>>> tarfile.is_tarfile('lala.txt')
False
>>> zipfile.is_zipfile('lala.txt')
False
>>> with bz2.BZ2File('startup.bat','r') as f:
... f.read()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
IOError: invalid data stream
2019 update:
I was looking for a solution to detect if a .csv file was gzipped or not. The answer @Lauritz gave was throwing errors for me, i imagine it’s just because the way files are read has changed in the past 7 years.
This library worked perfectly for me!
https://pypi.org/project/filetype/
The accepted solution looks great, but it doesn’t work with python-3, here are the modifications that made it work — using binary I/O instead of strings:
magic_dict = {
b"x1fx8bx08": "gz",
b"x42x5ax68": "bz2",
b"x50x4bx03x04": "zip"
}
''' SKIP '''
with open(filename, "rb") as f:
''' The rest is the same '''