Python/pyspark data frame rearrange columns

Question

I have a data frame in python/pyspark with columns id time city zip and so on……

Now I added a new column name to this data frame.

Now I have to arrange the columns in such a way that the name column comes after id

I have done like below

change_cols = ['id', 'name']

cols = ([col for col in change_cols if col in df] 
        + [col for col in df if col not in change_cols])

df = df[cols]

I am getting this error

pyspark.sql.utils.AnalysisException: u"Reference 'id' is ambiguous, could be: id#609, id#1224.;"

Why is this error occuring. How can I rectify this.

Asked By: User12345

||

Source

Answer 1

You can use select to change the order of the columns:

df.select("id","name","time","city")

Answered By: Alex

Answer 2

If you’re working with a large number of columns:

df.select(sorted(df.columns))

Answered By: melchoir55

Answer 3

If you just want to reorder some of them, while keeping the rest and not bothering about their order :

def get_cols_to_front(df, columns_to_front) :
    original = df.columns
    # Filter to present columns
    columns_to_front = [c for c in columns_to_front if c in original]
    # Keep the rest of the columns and sort it for consistency
    columns_other = list(set(original) - set(columns_to_front))
    columns_other.sort()
    # Apply the order
    df = df.select(*columns_to_front, *columns_other)

    return df

Answered By: ZettaP

Python/pyspark data frame rearrange columns

Question:

Answers: