TypeError: col should be Column with apache spark

Question

I have this method where I am gathering positive values

def pos_values(df, metrics):
    num_pos_values = df.where(df.ttu > 1).count()

    df.withColumn("loader_ttu_pos_value", num_pos_values)

    df.write.json(metrics)

However I get TypeError: col should be Column whenever I go to test it. I tried to cast it but that doesn’t seem to be an option.

Asked By: Mike3355

||

Source

Answer 1

The reason you’re getting this error is because df.withColumn expects a Column object as second argument, whereas you’re giving num_pos_values which is an integer.

If you want to assign a literal value to a column (you’ll have the same value for each row), you can use the lit function of pyspark.sql.functions.

Something like this works:

df = spark.createDataFrame([("2022", "January"), ("2021", "December")], ["Year", "Month"])

df.show()
+----+--------+
|Year|   Month|
+----+--------+
|2022| January|
|2021|December|
+----+--------+

from pyspark.sql.functions import lit

df.withColumn("testColumn", lit(5)).show()
+----+--------+----------+
|Year|   Month|testColumn|
+----+--------+----------+
|2022| January|         5|
|2021|December|         5|
+----+--------+----------+

Answered By: Koedlt

TypeError: col should be Column with apache spark

Question:

Answers: