I have a list that is generated by a function. when I execute
but when I want to create a
DataFrame with this command:
df = sqlContext.createDataFrame(preds_labels, ["prediction", "label"])
I get an error message:
not supported type: type ‘numpy.float64’
If I create the list manually, I have no problem. Do you have an idea?
pyspark uses its own type system and unfortunately it doesn’t deal with numpy well. It works with python types though. So you could manually convert the
df = sqlContext.createDataFrame( [(float(tup), float(tup) for tup in preds_labels], ["prediction", "label"] )
Note pyspark will then take them as
To anyone arriving here with the error:
typeerror not supported type class 'numpy.str_'
This is true for string as well. So if you created your list strings using numpy , try to change it to pure python.
Create list of single item repeated N times