Python float mysteriously off by anywhere between 0.1 to 0.3

Question

I’m writing a function to convert a weirdly formatted Degrees Minutes Seconds to Degrees Decimal.

My code is:

def fromDMS(coordinate):
    lat_dms = coordinate[0:10]
    lon_dms = coordinate[11:21]

    lat_sign = lat_dms[0]
    lat_deg = float(lat_dms[1:3])
    lat_min = float(lat_dms[3:5])
    lat_sec = float(lat_dms[5:])

    lon_sign = lon_dms[0]
    lon_deg = float(lon_dms[1:4])
    lon_min = float(lat_dms[4:6])
    lon_sec = float(lat_dms[6:])

    lat_deg = (lat_deg + (lat_min/60) + (lat_sec/(60 * 2)))
    if lat_sign == "-": lat_deg = lat_deg * -1
    lon_deg = (lon_deg + (lon_min/60) + (lon_sec/(60 * 2)))
    if lon_deg == "-": lon_deg = lon_deg * -1

    return lat_deg, lon_deg

The format in question is this string

-365535.000+1745401.000

where "-365535.000" (-36 degrees, 55 minutes, 35 seconds) is the latitude and "+1745401.000" (174 degrees, 55 minutes, and 1 second) is the longitude. Using an online calculator, these values should result in "-36.926389" and "174.916944", but end up as "37.20833333333333" and "174.92499999999998". I’ve heard that float’s can be a little weird sometimes, but not to this extent.

Asked By: Bevan Ellis

||

Source

Answer 1

lon_sign = lon_dms[0]
lon_deg = float(lon_dms[1:4])
lon_min = float(lat_dms[4:6])
lon_sec = float(lat_dms[6:])

On the third and fourth lines, you’re converting from lat_dms instead of lon_dms.

Answered By: John Gordon

Answer 2

According to https://www.fcc.gov/media/radio/dms-decimal, the decimal degrees should be -36.926389 and 174.900278.

One problem with this code is that it divides the seconds by 60 * 2 (i.e., 120) instead of 60 ** 2 (i.e., 3600). Making this change causes the latitude to be -36.92638888888889.

The second problem is that the assignments to lon_min and lon_sec use lat_dms instead of lon_dms. Making this change causes the longitude to be 174.90027777777777.

Answered By: Matthew James Kraai

Python float mysteriously off by anywhere between 0.1 to 0.3

Question:

Answers: