Convert python long/int to fixed size byte array
Question:
I’m trying to implement RC4 and DH key exchange in python. Problem is that I have no idea about how to convert the python long/int from the key exchange to the byte array I need for the RC4 implementation. Is there a simple way to convert a long to the required length byte array?
Update: forgot to mention that the numbers I’m dealing with are 768 bit unsigned integers.
Answers:
You can try using struct:
import struct
struct.pack('L',longvalue)
long/int to the byte array looks like exact purpose of struct.pack
. For long integers that exceed 4(8) bytes, you can come up with something like the next:
>>> limit = 256*256*256*256 - 1
>>> i = 1234567890987654321
>>> parts = []
>>> while i:
parts.append(i & limit)
i >>= 32
>>> struct.pack('>' + 'L'*len(parts), *parts )
'xb1lx1cxb1x11"x10xf4'
>>> struct.unpack('>LL', 'xb1lx1cxb1x11"x10xf4')
(2976652465L, 287445236)
>>> (287445236L << 32) + 2976652465L
1234567890987654321L
Basically what you need to do is convert the int/long into its base 256 representation — i.e. a number whose “digits” range from 0-255. Here’s a fairly efficient way to do something like that:
def base256_encode(n, minwidth=0): # int/long to byte array
if n > 0:
arr = []
while n:
n, rem = divmod(n, 256)
arr.append(rem)
b = bytearray(reversed(arr))
elif n == 0:
b = bytearray(b'x00')
else:
raise ValueError
if minwidth > 0 and len(b) < minwidth: # zero padding needed?
b = (minwidth-len(b)) * 'x00' + b
return b
You many not need thereversed()
call depending on the endian-ness desired (doing so would require the padding to be done differently as well). Also note that as written it doesn’t handle negative numbers.
You might also want to take a look at the similar but highly optimized long_to_bytes()
function in thenumber.py
module which is part of the open source Python Cryptography Toolkit. It actually converts the number into a string, not a byte array, but that’s a minor issue.
I haven’t done any benchmarks, but this recipe “works for me”.
The short version: use '%x' % val
, then unhexlify
the result. The devil is in the details, though, as unhexlify
requires an even number of hex digits, which %x
doesn’t guarantee. See the docstring, and the liberal inline comments for details.
from binascii import unhexlify
def long_to_bytes (val, endianness='big'):
"""
Use :ref:`string formatting` and :func:`~binascii.unhexlify` to
convert ``val``, a :func:`long`, to a byte :func:`str`.
:param long val: The value to pack
:param str endianness: The endianness of the result. ``'big'`` for
big-endian, ``'little'`` for little-endian.
If you want byte- and word-ordering to differ, you're on your own.
Using :ref:`string formatting` lets us use Python's C innards.
"""
# one (1) hex digit per four (4) bits
width = val.bit_length()
# unhexlify wants an even multiple of eight (8) bits, but we don't
# want more digits than we need (hence the ternary-ish 'or')
width += 8 - ((width % 8) or 8)
# format width specifier: four (4) bits per hex digit
fmt = '%%0%dx' % (width // 4)
# prepend zero (0) to the width, to zero-pad the output
s = unhexlify(fmt % val)
if endianness == 'little':
# see http://stackoverflow.com/a/931095/309233
s = s[::-1]
return s
…and my nosetest unit tests 😉
class TestHelpers (object):
def test_long_to_bytes_big_endian_small_even (self):
s = long_to_bytes(0x42)
assert s == 'x42'
s = long_to_bytes(0xFF)
assert s == 'xff'
def test_long_to_bytes_big_endian_small_odd (self):
s = long_to_bytes(0x1FF)
assert s == 'x01xff'
s = long_to_bytes(0x201FF)
assert s == 'x02x01xff'
def test_long_to_bytes_big_endian_large_even (self):
s = long_to_bytes(0xab23456c8901234567)
assert s == 'xabx23x45x6cx89x01x23x45x67'
def test_long_to_bytes_big_endian_large_odd (self):
s = long_to_bytes(0x12345678901234567)
assert s == 'x01x23x45x67x89x01x23x45x67'
def test_long_to_bytes_little_endian_small_even (self):
s = long_to_bytes(0x42, 'little')
assert s == 'x42'
s = long_to_bytes(0xFF, 'little')
assert s == 'xff'
def test_long_to_bytes_little_endian_small_odd (self):
s = long_to_bytes(0x1FF, 'little')
assert s == 'xffx01'
s = long_to_bytes(0x201FF, 'little')
assert s == 'xffx01x02'
def test_long_to_bytes_little_endian_large_even (self):
s = long_to_bytes(0xab23456c8901234567, 'little')
assert s == 'x67x45x23x01x89x6cx45x23xab'
def test_long_to_bytes_little_endian_large_odd (self):
s = long_to_bytes(0x12345678901234567, 'little')
assert s == 'x67x45x23x01x89x67x45x23x01'
Little-endian, reverse the result or the range if you want Big-endian:
def int_to_bytes(val, num_bytes):
return [(val & (0xff << pos*8)) >> pos*8 for pos in range(num_bytes)]
Big-endian:
def int_to_bytes(val, num_bytes):
return [(val & (0xff << pos*8)) >> pos*8 for pos in reversed(range(num_bytes))]
One-liner:
bytearray.fromhex('{:0192x}'.format(big_int))
The 192 is 768 / 4, because OP wanted 768-bit numbers and there are 4 bits in a hex digit. If you need a bigger bytearray
use a format string with a higher number. Example:
>>> big_int = 911085911092802609795174074963333909087482261102921406113936886764014693975052768158290106460018649707059449553895568111944093294751504971131180816868149233377773327312327573120920667381269572962606994373889233844814776702037586419
>>> bytearray.fromhex('{:0192x}'.format(big_int))
bytearray(b'x96;h^xdbJx8f3obLx9cxc2xb0-x9exa4Sj-xf6ixc1x9ex97x94x85Mx1dx93x10\x81xc2x89xcdxe0axc0Dx81vxdfxedxa9xc1x83pxdbUxf1xd0xfeR)xcex07xdepMx88xccx7fv\x1cx8dix87Nx00x8dxa8xbd[<xdfxafx13z:Hxedxc2)xa4x1ex0fxa7x92xa7xc6x16x86xf1xf3')
>>> lepi_int = 0x963b685edb4a8f336f624c9cc2b02d9ea4536a2df669c19e9794854d1d93105c81c289cde061c0448176dfeda9c18370db55f1d0fe5229ce07de704d88cc7f765c1c8d69874e008da8bd5b3cdfaf137a3a48edc229a41e0fa792a7c61686f1f
>>> bytearray.fromhex('{:0192x}'.format(lepi_int))
bytearray(b'tcxb6x85xedxb4xa8xf36xf6$xc9xcc+x02xd9xeaE6xa2xdffx9cx19xe9yHTxd1xd91x05xc8x1c(x9cxdex06x1cx04Hx17mxfexdax9cx187rxb5_x1dx0fxe5"x9cxe0}xe7x04xd8x8cxc7xf7exc1xc8xd6x98txe0x08xdax8bxd5xb3xcdxfaxf17xa3xa4x8exdc"x9aAxe0xfay*|ahox1f')
[My answer had used hex()
before. I corrected it with format()
in order to handle ints with odd-sized byte expressions. This fixes previous complaints about ValueError
.]
With Python 3.2 and later, you can use int.to_bytes
and int.from_bytes
: https://docs.python.org/3/library/stdtypes.html#int.to_bytes
Everyone has overcomplicated this answer:
some_int = <256 bit integer>
some_bytes = some_int.to_bytes(32, sys.byteorder)
my_bytearray = bytearray(some_bytes)
You just need to know the number of bytes that you are trying to convert. In my use cases, normally I only use this large of numbers for crypto, and at that point I have to worry about modulus and what-not, so I don’t think this is a big problem to be required to know the max number of bytes to return.
Since you are doing it as 768-bit math, then instead of 32 as the argument it would be 96.
Python 2.7 does not implement the int.to- very slow_bytes() method.
I tried 3 methods:
- hex unpack/pack : very slow
- byte shifting 8 bits at a time: significantly faster.
- using a “C” module and packing into the lower (7 ia64 or 3 i32) bytes.
This was about twice as fast as 2/ . It is the fastest option, but still too slow.
All these methods are very inefficient for two reasons:
- Python 2.7 does not support this useful operation.
- c does not support extended precision arithmetic using the carry/borrow/overflow flags available on most platforms.
i = 0x12345678
s = struct.pack('<I',i)
b = struct.unpack('BBBB',s)
I’m trying to implement RC4 and DH key exchange in python. Problem is that I have no idea about how to convert the python long/int from the key exchange to the byte array I need for the RC4 implementation. Is there a simple way to convert a long to the required length byte array?
Update: forgot to mention that the numbers I’m dealing with are 768 bit unsigned integers.
You can try using struct:
import struct
struct.pack('L',longvalue)
long/int to the byte array looks like exact purpose of struct.pack
. For long integers that exceed 4(8) bytes, you can come up with something like the next:
>>> limit = 256*256*256*256 - 1
>>> i = 1234567890987654321
>>> parts = []
>>> while i:
parts.append(i & limit)
i >>= 32
>>> struct.pack('>' + 'L'*len(parts), *parts )
'xb1lx1cxb1x11"x10xf4'
>>> struct.unpack('>LL', 'xb1lx1cxb1x11"x10xf4')
(2976652465L, 287445236)
>>> (287445236L << 32) + 2976652465L
1234567890987654321L
Basically what you need to do is convert the int/long into its base 256 representation — i.e. a number whose “digits” range from 0-255. Here’s a fairly efficient way to do something like that:
def base256_encode(n, minwidth=0): # int/long to byte array
if n > 0:
arr = []
while n:
n, rem = divmod(n, 256)
arr.append(rem)
b = bytearray(reversed(arr))
elif n == 0:
b = bytearray(b'x00')
else:
raise ValueError
if minwidth > 0 and len(b) < minwidth: # zero padding needed?
b = (minwidth-len(b)) * 'x00' + b
return b
You many not need thereversed()
call depending on the endian-ness desired (doing so would require the padding to be done differently as well). Also note that as written it doesn’t handle negative numbers.
You might also want to take a look at the similar but highly optimized long_to_bytes()
function in thenumber.py
module which is part of the open source Python Cryptography Toolkit. It actually converts the number into a string, not a byte array, but that’s a minor issue.
I haven’t done any benchmarks, but this recipe “works for me”.
The short version: use '%x' % val
, then unhexlify
the result. The devil is in the details, though, as unhexlify
requires an even number of hex digits, which %x
doesn’t guarantee. See the docstring, and the liberal inline comments for details.
from binascii import unhexlify
def long_to_bytes (val, endianness='big'):
"""
Use :ref:`string formatting` and :func:`~binascii.unhexlify` to
convert ``val``, a :func:`long`, to a byte :func:`str`.
:param long val: The value to pack
:param str endianness: The endianness of the result. ``'big'`` for
big-endian, ``'little'`` for little-endian.
If you want byte- and word-ordering to differ, you're on your own.
Using :ref:`string formatting` lets us use Python's C innards.
"""
# one (1) hex digit per four (4) bits
width = val.bit_length()
# unhexlify wants an even multiple of eight (8) bits, but we don't
# want more digits than we need (hence the ternary-ish 'or')
width += 8 - ((width % 8) or 8)
# format width specifier: four (4) bits per hex digit
fmt = '%%0%dx' % (width // 4)
# prepend zero (0) to the width, to zero-pad the output
s = unhexlify(fmt % val)
if endianness == 'little':
# see http://stackoverflow.com/a/931095/309233
s = s[::-1]
return s
…and my nosetest unit tests 😉
class TestHelpers (object):
def test_long_to_bytes_big_endian_small_even (self):
s = long_to_bytes(0x42)
assert s == 'x42'
s = long_to_bytes(0xFF)
assert s == 'xff'
def test_long_to_bytes_big_endian_small_odd (self):
s = long_to_bytes(0x1FF)
assert s == 'x01xff'
s = long_to_bytes(0x201FF)
assert s == 'x02x01xff'
def test_long_to_bytes_big_endian_large_even (self):
s = long_to_bytes(0xab23456c8901234567)
assert s == 'xabx23x45x6cx89x01x23x45x67'
def test_long_to_bytes_big_endian_large_odd (self):
s = long_to_bytes(0x12345678901234567)
assert s == 'x01x23x45x67x89x01x23x45x67'
def test_long_to_bytes_little_endian_small_even (self):
s = long_to_bytes(0x42, 'little')
assert s == 'x42'
s = long_to_bytes(0xFF, 'little')
assert s == 'xff'
def test_long_to_bytes_little_endian_small_odd (self):
s = long_to_bytes(0x1FF, 'little')
assert s == 'xffx01'
s = long_to_bytes(0x201FF, 'little')
assert s == 'xffx01x02'
def test_long_to_bytes_little_endian_large_even (self):
s = long_to_bytes(0xab23456c8901234567, 'little')
assert s == 'x67x45x23x01x89x6cx45x23xab'
def test_long_to_bytes_little_endian_large_odd (self):
s = long_to_bytes(0x12345678901234567, 'little')
assert s == 'x67x45x23x01x89x67x45x23x01'
Little-endian, reverse the result or the range if you want Big-endian:
def int_to_bytes(val, num_bytes):
return [(val & (0xff << pos*8)) >> pos*8 for pos in range(num_bytes)]
Big-endian:
def int_to_bytes(val, num_bytes):
return [(val & (0xff << pos*8)) >> pos*8 for pos in reversed(range(num_bytes))]
One-liner:
bytearray.fromhex('{:0192x}'.format(big_int))
The 192 is 768 / 4, because OP wanted 768-bit numbers and there are 4 bits in a hex digit. If you need a bigger bytearray
use a format string with a higher number. Example:
>>> big_int = 911085911092802609795174074963333909087482261102921406113936886764014693975052768158290106460018649707059449553895568111944093294751504971131180816868149233377773327312327573120920667381269572962606994373889233844814776702037586419
>>> bytearray.fromhex('{:0192x}'.format(big_int))
bytearray(b'x96;h^xdbJx8f3obLx9cxc2xb0-x9exa4Sj-xf6ixc1x9ex97x94x85Mx1dx93x10\x81xc2x89xcdxe0axc0Dx81vxdfxedxa9xc1x83pxdbUxf1xd0xfeR)xcex07xdepMx88xccx7fv\x1cx8dix87Nx00x8dxa8xbd[<xdfxafx13z:Hxedxc2)xa4x1ex0fxa7x92xa7xc6x16x86xf1xf3')
>>> lepi_int = 0x963b685edb4a8f336f624c9cc2b02d9ea4536a2df669c19e9794854d1d93105c81c289cde061c0448176dfeda9c18370db55f1d0fe5229ce07de704d88cc7f765c1c8d69874e008da8bd5b3cdfaf137a3a48edc229a41e0fa792a7c61686f1f
>>> bytearray.fromhex('{:0192x}'.format(lepi_int))
bytearray(b'tcxb6x85xedxb4xa8xf36xf6$xc9xcc+x02xd9xeaE6xa2xdffx9cx19xe9yHTxd1xd91x05xc8x1c(x9cxdex06x1cx04Hx17mxfexdax9cx187rxb5_x1dx0fxe5"x9cxe0}xe7x04xd8x8cxc7xf7exc1xc8xd6x98txe0x08xdax8bxd5xb3xcdxfaxf17xa3xa4x8exdc"x9aAxe0xfay*|ahox1f')
[My answer had used hex()
before. I corrected it with format()
in order to handle ints with odd-sized byte expressions. This fixes previous complaints about ValueError
.]
With Python 3.2 and later, you can use int.to_bytes
and int.from_bytes
: https://docs.python.org/3/library/stdtypes.html#int.to_bytes
Everyone has overcomplicated this answer:
some_int = <256 bit integer>
some_bytes = some_int.to_bytes(32, sys.byteorder)
my_bytearray = bytearray(some_bytes)
You just need to know the number of bytes that you are trying to convert. In my use cases, normally I only use this large of numbers for crypto, and at that point I have to worry about modulus and what-not, so I don’t think this is a big problem to be required to know the max number of bytes to return.
Since you are doing it as 768-bit math, then instead of 32 as the argument it would be 96.
Python 2.7 does not implement the int.to- very slow_bytes() method.
I tried 3 methods:
- hex unpack/pack : very slow
- byte shifting 8 bits at a time: significantly faster.
- using a “C” module and packing into the lower (7 ia64 or 3 i32) bytes.
This was about twice as fast as 2/ . It is the fastest option, but still too slow.
All these methods are very inefficient for two reasons:
- Python 2.7 does not support this useful operation.
- c does not support extended precision arithmetic using the carry/borrow/overflow flags available on most platforms.
i = 0x12345678
s = struct.pack('<I',i)
b = struct.unpack('BBBB',s)