ValueError: could not convert string to float: id
Question:
I’m running the following python script:
#!/usr/bin/python
import os,sys
from scipy import stats
import numpy as np
f = open('data2.txt', 'r').readlines()
for i in range(0, len(f)-1):
l1 = f[i].split()
list1 = [float(x) for x in l1]
However I got the errors like:
ValueError: could not convert string to float: id
I’m confused by this.
When I try this for only one line in interactive section, instead of for loop using script:
from scipy import stats
import numpy as np
f = open('data2.txt','r').readlines()
l1 = f[1].split()
list1 = [float(x) for x in l1]
list1
# [5.3209183842, 4.6422726719, 4.3788135547]
it works well. Can anyone explain a little bit about this?
Answers:
This error is pretty verbose:
ValueError: could not convert string to float: id
Somewhere in your text file, a line has the word id in it, which can’t really be converted to a number.
Your test code works because the word id isn’t present in line 2.
If you want to catch that line, try this code. I cleaned your code up a tad:
#!/usr/bin/python
import os, sys
from scipy import stats
import numpy as np
for index, line in enumerate(open('data2.txt', 'r').readlines()):
w = line.split(' ')
l1 = w[1:8]
l2 = w[8:15]
try:
list1 = map(float, l1)
list2 = map(float, l2)
except ValueError:
print 'Line {i} is corrupt!'.format(i = index)'
break
result = stats.ttest_ind(list1, list2)
print result[1]
Obviously some of your lines don’t have valid float data, specifically some line have text id which can’t be converted to float.
When you try it in interactive prompt you are trying only first line, so best way is to print the line where you are getting this error and you will know the wrong line e.g.
#!/usr/bin/python
import os,sys
from scipy import stats
import numpy as np
f=open('data2.txt', 'r').readlines()
N=len(f)-1
for i in range(0,N):
w=f[i].split()
l1=w[1:8]
l2=w[8:15]
try:
list1=[float(x) for x in l1]
list2=[float(x) for x in l2]
except ValueError,e:
print "error",e,"on line",i
result=stats.ttest_ind(list1,list2)
print result[1]
Your data may not be what you expect — it seems you’re expecting, but not getting, floats.
A simple solution to figuring out where this occurs would be to add a try/except to the for-loop:
for i in range(0,N):
w=f[i].split()
l1=w[1:8]
l2=w[8:15]
try:
list1=[float(x) for x in l1]
list2=[float(x) for x in l2]
except ValueError, e:
# report the error in some way that is helpful -- maybe print out i
result=stats.ttest_ind(list1,list2)
print result[1]
My error was very simple: the text file containing the data had some space (so not visible) character on the last line.
As an output of grep, I had 45 instead of just 45.
Perhaps your numbers aren’t actually numbers, but letters masquerading as numbers?
In my case, the font I was using meant that “l” and “1” looked very similar. I had a string like ‘l1919’ which I thought was ‘11919’ and that messed things up.
I solved the similar situation with basic technique using pandas. First load the csv or text file using pandas.It’s pretty simple
data=pd.read_excel('link to the file')
Then set the index of data to the respected column that needs to be changed. For example, if your data has ID as one attribute or column, then set index to ID.
data = data.set_index("ID")
Then delete all the rows with “id” as the value instead of number using following command.
data = data.drop("id", axis=0).
Hope, this will help you.
For a Pandas dataframe with a column of numbers with commas, use this:
df["Numbers"] = [float(str(i).replace(",", "")) for i in df["Numbers"]]
So values like 4,200.42 would be converted to 4200.42 as a float.
Bonus 1: This is fast.
Bonus 2: More space efficient if saving that dataframe in something like Apache Parquet format.
Shortest way:
df["id"] = df['id'].str.replace(',', '').astype(float) – if ‘,’ is the problem
df["id"] = df['id'].str.replace(' ', '').astype(float) – if blank space is the problem
Update empty string values with 0.0 values:
if you know the possible non-float values then update it.
df.loc[df['score'] == '', 'score'] = 0.0
df['score']=df['score'].astype(float)
A good option to handle these types of erroneous values in the data is to remove it at the read_csv step by specifying na_values. This will identify strings to recognize as NA/NaN.
By default the following values are interpreted as NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘None’, ‘n/a’, ‘nan’, ‘null’. So in your case, since it’s complaining about the string ‘id’ in the data. you could do the following:
df = pd.read_csv('file.csv', na_values = ['id'])
This will specify values the columns with ‘id’ in them as null and resolve the value error when running analysis on the column of interest
This error (or a very similar error) commonly appears when changing the dtype of a pandas column from object to float using astype() or apply(). The cause is there are non-numeric strings that cannot be converted into floats. One solution is to use pd.to_numeric() instead, with errors='coerce' passed. This replaces non-numeric values such as the literal string 'id' to NaN.
df = pd.DataFrame({'col': ['id', '1.5', '2.4']})
df['col'] = df['col'].astype(float) # <---- ValueError: could not convert string to float: 'id'
df['col'] = df['col'].apply(lambda x: float(x)) # <---- ValueError
df['col'] = pd.to_numeric(df['col'], errors='coerce') # <---- OK
# ^^^^^^^^^^^^^^^ <--- converts non-numbers to NaN
0 NaN
1 1.5
2 2.4
Name: col, dtype: float64
pd.to_numeric() works only on individual columns, so if you need to change the dtype of multiple columns in one go (similar to how .astype(float) may be used), then passing it to apply() should do the job.
df = pd.DataFrame({'col1': ['id', '1.5', '2.4'], 'col2': ['10.2', '21.3', '20.6']})
df[['col1', 'col2']] = df.apply(pd.to_numeric, errors='coerce')
col1 col2
0 NaN 10.2
1 1.5 21.3
2 2.4 20.6
Sometimes there are thousands separator commas, which throws a similar error:
ValueError: could not convert string to float: '2,000.4'
in which case, first removing them before the pd.to_numeric() call solves the issue.
df = pd.DataFrame({'col': ['id', '1.5', '2,000.4']})
df['col'] = df['col'].replace(regex=',', value='')
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^ <--- remove commas
df['col'] = pd.to_numeric(df['col'], errors='coerce')
0 NaN
1 1.5
2 2000.4
Name: col, dtype: float64
I’m running the following python script:
#!/usr/bin/python
import os,sys
from scipy import stats
import numpy as np
f = open('data2.txt', 'r').readlines()
for i in range(0, len(f)-1):
l1 = f[i].split()
list1 = [float(x) for x in l1]
However I got the errors like:
ValueError: could not convert string to float: id
I’m confused by this.
When I try this for only one line in interactive section, instead of for loop using script:
from scipy import stats
import numpy as np
f = open('data2.txt','r').readlines()
l1 = f[1].split()
list1 = [float(x) for x in l1]
list1
# [5.3209183842, 4.6422726719, 4.3788135547]
it works well. Can anyone explain a little bit about this?
This error is pretty verbose:
ValueError: could not convert string to float: id
Somewhere in your text file, a line has the word id in it, which can’t really be converted to a number.
Your test code works because the word id isn’t present in line 2.
If you want to catch that line, try this code. I cleaned your code up a tad:
#!/usr/bin/python
import os, sys
from scipy import stats
import numpy as np
for index, line in enumerate(open('data2.txt', 'r').readlines()):
w = line.split(' ')
l1 = w[1:8]
l2 = w[8:15]
try:
list1 = map(float, l1)
list2 = map(float, l2)
except ValueError:
print 'Line {i} is corrupt!'.format(i = index)'
break
result = stats.ttest_ind(list1, list2)
print result[1]
Obviously some of your lines don’t have valid float data, specifically some line have text id which can’t be converted to float.
When you try it in interactive prompt you are trying only first line, so best way is to print the line where you are getting this error and you will know the wrong line e.g.
#!/usr/bin/python
import os,sys
from scipy import stats
import numpy as np
f=open('data2.txt', 'r').readlines()
N=len(f)-1
for i in range(0,N):
w=f[i].split()
l1=w[1:8]
l2=w[8:15]
try:
list1=[float(x) for x in l1]
list2=[float(x) for x in l2]
except ValueError,e:
print "error",e,"on line",i
result=stats.ttest_ind(list1,list2)
print result[1]
Your data may not be what you expect — it seems you’re expecting, but not getting, floats.
A simple solution to figuring out where this occurs would be to add a try/except to the for-loop:
for i in range(0,N):
w=f[i].split()
l1=w[1:8]
l2=w[8:15]
try:
list1=[float(x) for x in l1]
list2=[float(x) for x in l2]
except ValueError, e:
# report the error in some way that is helpful -- maybe print out i
result=stats.ttest_ind(list1,list2)
print result[1]
My error was very simple: the text file containing the data had some space (so not visible) character on the last line.
As an output of grep, I had 45 instead of just 45.
Perhaps your numbers aren’t actually numbers, but letters masquerading as numbers?
In my case, the font I was using meant that “l” and “1” looked very similar. I had a string like ‘l1919’ which I thought was ‘11919’ and that messed things up.
I solved the similar situation with basic technique using pandas. First load the csv or text file using pandas.It’s pretty simple
data=pd.read_excel('link to the file')
Then set the index of data to the respected column that needs to be changed. For example, if your data has ID as one attribute or column, then set index to ID.
data = data.set_index("ID")
Then delete all the rows with “id” as the value instead of number using following command.
data = data.drop("id", axis=0).
Hope, this will help you.
For a Pandas dataframe with a column of numbers with commas, use this:
df["Numbers"] = [float(str(i).replace(",", "")) for i in df["Numbers"]]
So values like 4,200.42 would be converted to 4200.42 as a float.
Bonus 1: This is fast.
Bonus 2: More space efficient if saving that dataframe in something like Apache Parquet format.
Shortest way:
df["id"] = df['id'].str.replace(',', '').astype(float) – if ‘,’ is the problem
df["id"] = df['id'].str.replace(' ', '').astype(float) – if blank space is the problem
Update empty string values with 0.0 values:
if you know the possible non-float values then update it.
df.loc[df['score'] == '', 'score'] = 0.0
df['score']=df['score'].astype(float)
A good option to handle these types of erroneous values in the data is to remove it at the read_csv step by specifying na_values. This will identify strings to recognize as NA/NaN.
By default the following values are interpreted as NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘None’, ‘n/a’, ‘nan’, ‘null’. So in your case, since it’s complaining about the string ‘id’ in the data. you could do the following:
df = pd.read_csv('file.csv', na_values = ['id'])
This will specify values the columns with ‘id’ in them as null and resolve the value error when running analysis on the column of interest
This error (or a very similar error) commonly appears when changing the dtype of a pandas column from object to float using astype() or apply(). The cause is there are non-numeric strings that cannot be converted into floats. One solution is to use pd.to_numeric() instead, with errors='coerce' passed. This replaces non-numeric values such as the literal string 'id' to NaN.
df = pd.DataFrame({'col': ['id', '1.5', '2.4']})
df['col'] = df['col'].astype(float) # <---- ValueError: could not convert string to float: 'id'
df['col'] = df['col'].apply(lambda x: float(x)) # <---- ValueError
df['col'] = pd.to_numeric(df['col'], errors='coerce') # <---- OK
# ^^^^^^^^^^^^^^^ <--- converts non-numbers to NaN
0 NaN
1 1.5
2 2.4
Name: col, dtype: float64
pd.to_numeric() works only on individual columns, so if you need to change the dtype of multiple columns in one go (similar to how .astype(float) may be used), then passing it to apply() should do the job.
df = pd.DataFrame({'col1': ['id', '1.5', '2.4'], 'col2': ['10.2', '21.3', '20.6']})
df[['col1', 'col2']] = df.apply(pd.to_numeric, errors='coerce')
col1 col2
0 NaN 10.2
1 1.5 21.3
2 2.4 20.6
Sometimes there are thousands separator commas, which throws a similar error:
ValueError: could not convert string to float: '2,000.4'
in which case, first removing them before the pd.to_numeric() call solves the issue.
df = pd.DataFrame({'col': ['id', '1.5', '2,000.4']})
df['col'] = df['col'].replace(regex=',', value='')
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^ <--- remove commas
df['col'] = pd.to_numeric(df['col'], errors='coerce')
0 NaN
1 1.5
2 2000.4
Name: col, dtype: float64