ImportError: cannot import name 'VarcharType' from 'pyspark.sql.types'

Question

I am getting this error on Google Colab. Is it because I’m using the wrong version?

ImportError: cannot import name 'VarcharType' from 'pyspark.sql.types' (/content/spark-3.1.2-bin-hadoop2.7/python/pyspark/sql/types.py)

Asked By: AutumnRain

||

Source

Answer 1

That is because VarcharType does not exists yet in spark 3.1.2.

The source code of pyspark.sql.types in __all__ declares the only available types to import:

__all__ = [
    "DataType", "NullType", "StringType", "BinaryType", "BooleanType", "DateType",
    "TimestampType", "DecimalType", "DoubleType", "FloatType", "ByteType", "IntegerType",
    "LongType", "ShortType", "ArrayType", "MapType", "StructField", "StructType"
]

Therefore, StringType is the alternative:

from pyspark.sql.types import StringType

Answered By: Alexander Volok

ImportError: cannot import name 'VarcharType' from 'pyspark.sql.types'

Question:

Answers: