PYSpark data Frame schema is showing String for every column

Question

I am reading CSV file from below code snippet

df_pyspark = spark.read.csv("sample_data.csv") df_pyspark

and when i try to print data Frame its output is like shown as below:

DataFrame[_c0: string, _c1: string, _c2: string, _c3: string, _c4: string, _c5: string]

For each column dataType is showing ‘String’ even though column contains different dataType’s as below:

df_pyspark.show()

|_c0|       _c1|      _c2|                 _c3|        _c4|       _c5|
+---+----------+---------+--------------------+-----------+----------+
| id|first_name|last_name|               email|     gender|     phone|
|  1|    Bidget| Mirfield|bmirfield0@scient...|     Female|5628618353|
|  2|   Gonzalo|    Vango|    [email protected]|       Male|9556535457|
|  3|      Rock| Pampling|rpampling2@guardi...|   Bigender|4472741337|
|  4|   Dorella|  Edelman|dedelman3@histats...|     Female|4303062344|
|  5|     Faber|  Thwaite|fthwaite4@google....|Genderqueer|1348658809|
|  6|     Debee| Philcott|dphilcott5@cafepr...|     Female|7906881842|`

I want to print the exact DataType of every column?

Asked By: Darshan patil

||

Source

Answer 1

Use inferSchema parameter during read of CSV file it’ll Show the exact/correct datatype according to the values in columns

    df_pyspark = spark.read.csv("sample_data.csv", header=True, inferSchema=True)

    +---+----------+---------+--------------------+-----------+----------+
    | id|first_name|last_name|               email|     gender|     phone|
    +---+----------+---------+--------------------+-----------+----------+
    |  1|    Bidget| Mirfield|bmirfield0@scient...|     Female|5628618353|
    |  2|   Gonzalo|    Vango|    [email protected]|       Male|9556535457|
    |  3|      Rock| Pampling|rpampling2@guardi...|   Bigender|4472741337|
    |  4|   Dorella|  Edelman|dedelman3@histats...|     Female|4303062344|
    |  5|     Faber|  Thwaite|fthwaite4@google....|Genderqueer|1348658809|
    +---+----------+---------+--------------------+-----------+----------+
    only showing top 5 rows

    df_pyspark.printSchema()

    root
     |-- id: integer (nullable = true)
     |-- first_name: string (nullable = true)
     |-- last_name: string (nullable = true)
     |-- email: string (nullable = true)
     |-- gender: string (nullable = true)
     |-- phone: long (nullable = true)

Answered By: Adinath Nikam

PYSpark data Frame schema is showing String for every column

Question:

Answers: