Searching/reading binary data in Python
Question:
I’m reading in a binary file (a jpg in this case), and need to find some values in that file. For those interested, the binary file is a jpg and I’m attempting to pick out its dimensions by looking for the binary structure as detailed here.
I need to find FFC0 in the binary data, skip ahead some number of bytes, and then read 4 bytes (this should give me the image dimensions).
What’s a good way of searching for the value in the binary data? Is there an equivalent of ‘find’, or something like re?
Answers:
You could actually load the file into a string and search that string for the byte sequence 0xffc0 using the str.find() method. It works for any byte sequence.
The code to do this depends on a couple things. If you open the file in binary mode and you’re using Python 3 (both of which are probably best practice for this scenario), you’ll need to search for a byte string (as opposed to a character string), which means you have to prefix the string with b.
with open(filename, 'rb') as f:
s = f.read()
s.find(b'xffxc0')
If you open the file in text mode in Python 3, you’d have to search for a character string:
with open(filename, 'r') as f:
s = f.read()
s.find('xffxc0')
though there’s no particular reason to do this. It doesn’t get you any advantage over the previous way, and if you’re on a platform that treats binary files and text files differently (e.g. Windows), there is a chance this will cause problems.
Python 2 doesn’t make the distinction between byte strings and character strings, so if you’re using that version, it doesn’t matter whether you include or exclude the b in b'xffxc0'. And if your platform treats binary files and text files identically (e.g. Mac or Linux), it doesn’t matter whether you use 'r' or 'rb' as the file mode either. But I’d still recommend using something like the first code sample above just for forward compatibility – in case you ever do switch to Python 3, it’s one less thing to fix.
The re module does work with both string and binary data (str in Python 2 and bytes in Python 3), so you can use it as well as str.find for your task.
Well, obviously there is PIL The Image module has size as an attribute. If you are wanting to get the size exactly how you suggest and without loading the file you are going to have to go through it line by line. Not the nicest way to do it but it would work.
The bitstring module was designed for pretty much this purpose. For your case the following code (which I haven’t tested) should help illustrate:
from bitstring import ConstBitStream
# Can initialise from files, bytes, etc.
s = ConstBitStream(filename='your_file')
# Search to Start of Frame 0 code on byte boundary
found = s.find('0xffc0', bytealigned=True)
if found:
print("Found start code at byte offset %d." % found[0])
s0f0, length, bitdepth, height, width = s.readlist('hex:16, uint:16,
uint:8, 2*uint:16')
print("Width %d, Height %d" % (width, height))
Instead of reading the entire file into memory, searching it and then writing a new file out to disk you can use the mmap module for this. mmap will not store the entire file in memory and it allows for in-place modification.
#!/usr/bin/python
import mmap
with open("hugefile", "rw+b") as f:
mm = mmap.mmap(f.fileno(), 0)
print mm.find('x00x09x03x03')
For Python >=3.2:
import re
f = open("filename.jpg", "rb")
byte = f.read()
f.close()
matchObj = re.match( b'xffxd8.*xffxc0...(..)(..).*xffxd9', byte, re.MULTILINE|re.DOTALL)
if matchObj:
# https://stackoverflow.com/q/444591
print (int.from_bytes(matchObj.group(1), 'big')) # height
print (int.from_bytes(matchObj.group(2), 'big')) # width
The find() method should be used only if you need to know the position of sub, if not, you can use the in operator, for example:
with open("foo.bin", 'rb') as f:
if b'x00' in f.read():
print('The file is binary!')
else:
print('The file is not binary!')
In Python 3.x you can search a byte string by another byte string like this:
>>> byte_array = b'this is a byte arrayrnrnXYZx80x04x95 x00x00x00x00x00'
>>> byte_array.find('rnrn'.encode())
20
>>>
I’m reading in a binary file (a jpg in this case), and need to find some values in that file. For those interested, the binary file is a jpg and I’m attempting to pick out its dimensions by looking for the binary structure as detailed here.
I need to find FFC0 in the binary data, skip ahead some number of bytes, and then read 4 bytes (this should give me the image dimensions).
What’s a good way of searching for the value in the binary data? Is there an equivalent of ‘find’, or something like re?
You could actually load the file into a string and search that string for the byte sequence 0xffc0 using the str.find() method. It works for any byte sequence.
The code to do this depends on a couple things. If you open the file in binary mode and you’re using Python 3 (both of which are probably best practice for this scenario), you’ll need to search for a byte string (as opposed to a character string), which means you have to prefix the string with b.
with open(filename, 'rb') as f:
s = f.read()
s.find(b'xffxc0')
If you open the file in text mode in Python 3, you’d have to search for a character string:
with open(filename, 'r') as f:
s = f.read()
s.find('xffxc0')
though there’s no particular reason to do this. It doesn’t get you any advantage over the previous way, and if you’re on a platform that treats binary files and text files differently (e.g. Windows), there is a chance this will cause problems.
Python 2 doesn’t make the distinction between byte strings and character strings, so if you’re using that version, it doesn’t matter whether you include or exclude the b in b'xffxc0'. And if your platform treats binary files and text files identically (e.g. Mac or Linux), it doesn’t matter whether you use 'r' or 'rb' as the file mode either. But I’d still recommend using something like the first code sample above just for forward compatibility – in case you ever do switch to Python 3, it’s one less thing to fix.
The re module does work with both string and binary data (str in Python 2 and bytes in Python 3), so you can use it as well as str.find for your task.
Well, obviously there is PIL The Image module has size as an attribute. If you are wanting to get the size exactly how you suggest and without loading the file you are going to have to go through it line by line. Not the nicest way to do it but it would work.
The bitstring module was designed for pretty much this purpose. For your case the following code (which I haven’t tested) should help illustrate:
from bitstring import ConstBitStream
# Can initialise from files, bytes, etc.
s = ConstBitStream(filename='your_file')
# Search to Start of Frame 0 code on byte boundary
found = s.find('0xffc0', bytealigned=True)
if found:
print("Found start code at byte offset %d." % found[0])
s0f0, length, bitdepth, height, width = s.readlist('hex:16, uint:16,
uint:8, 2*uint:16')
print("Width %d, Height %d" % (width, height))
Instead of reading the entire file into memory, searching it and then writing a new file out to disk you can use the mmap module for this. mmap will not store the entire file in memory and it allows for in-place modification.
#!/usr/bin/python
import mmap
with open("hugefile", "rw+b") as f:
mm = mmap.mmap(f.fileno(), 0)
print mm.find('x00x09x03x03')
For Python >=3.2:
import re
f = open("filename.jpg", "rb")
byte = f.read()
f.close()
matchObj = re.match( b'xffxd8.*xffxc0...(..)(..).*xffxd9', byte, re.MULTILINE|re.DOTALL)
if matchObj:
# https://stackoverflow.com/q/444591
print (int.from_bytes(matchObj.group(1), 'big')) # height
print (int.from_bytes(matchObj.group(2), 'big')) # width
The find() method should be used only if you need to know the position of sub, if not, you can use the in operator, for example:
with open("foo.bin", 'rb') as f:
if b'x00' in f.read():
print('The file is binary!')
else:
print('The file is not binary!')
In Python 3.x you can search a byte string by another byte string like this:
>>> byte_array = b'this is a byte arrayrnrnXYZx80x04x95 x00x00x00x00x00'
>>> byte_array.find('rnrn'.encode())
20
>>>