Converting a Python Float to a String without losing precision
Question:
I am maintaining a Python script that uses xlrd to retrieve values from Excel spreadsheets, and then do various things with them. Some of the cells in the spreadsheet are high-precision numbers, and they must remain as such. When retrieving the values of one of these cells, xlrd gives me a float such as 0.38288746115497402.
However, I need to get this value into a string later on in the code. Doing either str(value) or unicode(value) will return something like “0.382887461155”. The requirements say that this is not acceptable; the precision needs to be preserved.
I’ve tried a couple things so far to no success. The first was using a string formatting thingy:
data = "%.40s" % (value)
data2 = "%.40r" % (value)
But both produce the same rounded number, “0.382887461155”.
Upon searching around for people with similar problems on SO and elsewhere on the internet, a common suggestion was to use the Decimal class. But I can’t change the way the data is given to me (unless somebody knows of a secret way to make xlrd return Decimals). And when I try to do this:
data = Decimal(value)
I get a TypeError: Cannot convert float to Decimal. First convert the float to a string. But obviously I can’t convert it to a string, or else I will lose the precision.
So yeah, I’m open to any suggestions — even really gross/hacky ones if necessary. I’m not terribly experienced with Python (more of a Java/C# guy myself) so feel free to correct me if I’ve got some kind of fundamental misunderstanding here.
EDIT: Just thought I would add that I am using Python 2.6.4. I don’t think there are any formal requirements stopping me from changing versions; it just has to not mess up any of the other code.
Answers:
You can use repr() to convert to a string without losing precision, then convert to a Decimal:
>>> from decimal import Decimal
>>> f = 0.38288746115497402
>>> d = Decimal(repr(f))
>>> print d
0.38288746115497402
EDIT: Cleared my previous answer b/c it didn’t work properly.
I’m on Python 2.6.5 and this works for me:
a = 0.38288746115497402
print repr(a)
type(repr(a)) #Says it's a string
Note: This just converts to a string. You’ll need to convert to Decimal yourself later if needed.
EDIT: I am wrong. I shall leave this answer here so the rest of the thread makes sense, but it’s not true. Please see John Machin’s answer above. Thanks guys =).
If the above answers work that’s great — it will save you a lot of nasty hacking. However, at least on my system, they won’t. You can check this with e.g.
import sys
print( "%.30f" % sys.float_info.epsilon )
That number is the smallest float that your system can distinguish from zero. Anything smaller than that may be randomly added or subtracted from any float when you perform an operation. This means that, at least on my Python setup, the precision is lost inside the guts of xlrd, and there seems to be nothing you can do without modifying it. Which is odd; I’d have expected this case to have occurred before, but apparently not!
It may be possible to modify your local xlrd installation to change the float cast. Open up site-packagesxlrdsheet.py and go down to line 1099:
...
elif rc == XL_INTEGER:
rowx, colx, cell_attr, d = local_unpack('<HH3sH', data)
self_put_number_cell(rowx, colx, float(d), self.fixed_BIFF2_xfindex(cell_attr, rowx, colx))
...
Notice the float cast — you could try changing that to a decimal.Decimal and see what happens.
I’m the author of xlrd. There is so much confusion in other answers and comments to rebut in comments so I’m doing it in an answer.
@katriealex: “””precision being lost in the guts of xlrd””” — entirely unfounded and untrue. xlrd reproduces exactly the 64-bit float that’s stored in the XLS file.
@katriealex: “””It may be possible to modify your local xlrd installation to change the float cast””” — I don’t know why you would want to do this; you don’t lose any precision by floating a 16-bit integer!!! In any case that code is used only when reading Excel 2.X files (which had an INTEGER-type cell record). The OP gives no indication that he is reading such ancient files.
@jloubert: You must be mistaken. "%.40r" % a_float is just a baroque way of getting the same answer as repr(a_float).
@EVERYBODY: You don’t need to convert a float to a decimal to preserve the precision. The whole point of the repr() function is that the following is guaranteed:
float(repr(a_float)) == a_float
Python 2.X (X <= 6) repr gives a constant 17 decimal digits of precision, as that is guaranteed to reproduce the original value. Later Pythons (2.7, 3.1) give the minimal number of decimal digits that will reproduce the original value.
Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] on win32
>>> f = 0.38288746115497402
>>> repr(f)
'0.38288746115497402'
>>> float(repr(f)) == f
True
Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32
>>> f = 0.38288746115497402
>>> repr(f)
'0.382887461154974'
>>> float(repr(f)) == f
True
So the bottom line is that if you want a string that preserves all the precision of a float object, use preserved = repr(the_float_object) … recover the value later by float(preserved). It’s that simple. No need for the decimal module.
As has already been said, a float isn’t precise at all – so preserving precision can be somewhat misleading.
Here’s a way to get every last bit of information out of a float object:
>>> from decimal import Decimal
>>> str(Decimal.from_float(0.1))
'0.1000000000000000055511151231257827021181583404541015625'
Another way would be like so.
>>> 0.1.hex()
'0x1.999999999999ap-4'
Both strings represent the exact contents of the float. Allmost anything else interprets the float as python thinks it was probably intended (which most of the time is correct).
I am maintaining a Python script that uses xlrd to retrieve values from Excel spreadsheets, and then do various things with them. Some of the cells in the spreadsheet are high-precision numbers, and they must remain as such. When retrieving the values of one of these cells, xlrd gives me a float such as 0.38288746115497402.
However, I need to get this value into a string later on in the code. Doing either str(value) or unicode(value) will return something like “0.382887461155”. The requirements say that this is not acceptable; the precision needs to be preserved.
I’ve tried a couple things so far to no success. The first was using a string formatting thingy:
data = "%.40s" % (value)
data2 = "%.40r" % (value)
But both produce the same rounded number, “0.382887461155”.
Upon searching around for people with similar problems on SO and elsewhere on the internet, a common suggestion was to use the Decimal class. But I can’t change the way the data is given to me (unless somebody knows of a secret way to make xlrd return Decimals). And when I try to do this:
data = Decimal(value)
I get a TypeError: Cannot convert float to Decimal. First convert the float to a string. But obviously I can’t convert it to a string, or else I will lose the precision.
So yeah, I’m open to any suggestions — even really gross/hacky ones if necessary. I’m not terribly experienced with Python (more of a Java/C# guy myself) so feel free to correct me if I’ve got some kind of fundamental misunderstanding here.
EDIT: Just thought I would add that I am using Python 2.6.4. I don’t think there are any formal requirements stopping me from changing versions; it just has to not mess up any of the other code.
You can use repr() to convert to a string without losing precision, then convert to a Decimal:
>>> from decimal import Decimal
>>> f = 0.38288746115497402
>>> d = Decimal(repr(f))
>>> print d
0.38288746115497402
EDIT: Cleared my previous answer b/c it didn’t work properly.
I’m on Python 2.6.5 and this works for me:
a = 0.38288746115497402
print repr(a)
type(repr(a)) #Says it's a string
Note: This just converts to a string. You’ll need to convert to Decimal yourself later if needed.
EDIT: I am wrong. I shall leave this answer here so the rest of the thread makes sense, but it’s not true. Please see John Machin’s answer above. Thanks guys =).
If the above answers work that’s great — it will save you a lot of nasty hacking. However, at least on my system, they won’t. You can check this with e.g.
import sys
print( "%.30f" % sys.float_info.epsilon )
That number is the smallest float that your system can distinguish from zero. Anything smaller than that may be randomly added or subtracted from any float when you perform an operation. This means that, at least on my Python setup, the precision is lost inside the guts of xlrd, and there seems to be nothing you can do without modifying it. Which is odd; I’d have expected this case to have occurred before, but apparently not!
It may be possible to modify your local xlrd installation to change the float cast. Open up site-packagesxlrdsheet.py and go down to line 1099:
...
elif rc == XL_INTEGER:
rowx, colx, cell_attr, d = local_unpack('<HH3sH', data)
self_put_number_cell(rowx, colx, float(d), self.fixed_BIFF2_xfindex(cell_attr, rowx, colx))
...
Notice the float cast — you could try changing that to a decimal.Decimal and see what happens.
I’m the author of xlrd. There is so much confusion in other answers and comments to rebut in comments so I’m doing it in an answer.
@katriealex: “””precision being lost in the guts of xlrd””” — entirely unfounded and untrue. xlrd reproduces exactly the 64-bit float that’s stored in the XLS file.
@katriealex: “””It may be possible to modify your local xlrd installation to change the float cast””” — I don’t know why you would want to do this; you don’t lose any precision by floating a 16-bit integer!!! In any case that code is used only when reading Excel 2.X files (which had an INTEGER-type cell record). The OP gives no indication that he is reading such ancient files.
@jloubert: You must be mistaken. "%.40r" % a_float is just a baroque way of getting the same answer as repr(a_float).
@EVERYBODY: You don’t need to convert a float to a decimal to preserve the precision. The whole point of the repr() function is that the following is guaranteed:
float(repr(a_float)) == a_float
Python 2.X (X <= 6) repr gives a constant 17 decimal digits of precision, as that is guaranteed to reproduce the original value. Later Pythons (2.7, 3.1) give the minimal number of decimal digits that will reproduce the original value.
Python 2.6.4 (r264:75708, Oct 26 2009, 08:23:19) [MSC v.1500 32 bit (Intel)] on win32
>>> f = 0.38288746115497402
>>> repr(f)
'0.38288746115497402'
>>> float(repr(f)) == f
True
Python 2.7 (r27:82525, Jul 4 2010, 09:01:59) [MSC v.1500 32 bit (Intel)] on win32
>>> f = 0.38288746115497402
>>> repr(f)
'0.382887461154974'
>>> float(repr(f)) == f
True
So the bottom line is that if you want a string that preserves all the precision of a float object, use preserved = repr(the_float_object) … recover the value later by float(preserved). It’s that simple. No need for the decimal module.
As has already been said, a float isn’t precise at all – so preserving precision can be somewhat misleading.
Here’s a way to get every last bit of information out of a float object:
>>> from decimal import Decimal
>>> str(Decimal.from_float(0.1))
'0.1000000000000000055511151231257827021181583404541015625'
Another way would be like so.
>>> 0.1.hex()
'0x1.999999999999ap-4'
Both strings represent the exact contents of the float. Allmost anything else interprets the float as python thinks it was probably intended (which most of the time is correct).