Running get_dummies on several DataFrame columns?

Question

How can one idiomatically run a function like get_dummies, which expects a single column and returns several, on multiple DataFrame columns?

Asked By: Emre

||

Source

Answer 1

Since pandas version 0.15.0, pd.get_dummies can handle a DataFrame directly (before that, it could only handle a single Series, and see below for the workaround):

In [1]: df = DataFrame({'A': ['a', 'b', 'a'], 'B': ['c', 'c', 'b'],
   ...:                 'C': [1, 2, 3]})

In [2]: df
Out[2]:
   A  B  C
0  a  c  1
1  b  c  2
2  a  b  3

In [3]: pd.get_dummies(df)
Out[3]:
   C  A_a  A_b  B_b  B_c
0  1    1    0    0    1
1  2    0    1    0    1
2  3    1    0    1    0

Workaround for pandas < 0.15.0

You can do it for each column seperate and then concat the results:

In [111]: df
Out[111]: 
   A  B
0  a  x
1  a  y
2  b  z
3  b  x
4  c  x
5  a  y
6  b  y
7  c  z

In [112]: pd.concat([pd.get_dummies(df[col]) for col in df], axis=1, keys=df.columns)
Out[112]: 
   A        B      
   a  b  c  x  y  z
0  1  0  0  1  0  0
1  1  0  0  0  1  0
2  0  1  0  0  0  1
3  0  1  0  1  0  0
4  0  0  1  1  0  0
5  1  0  0  0  1  0
6  0  1  0  0  1  0
7  0  0  1  0  0  1

If you don’t want the multi-index column, then remove the keys=.. from the concat function call.

Answered By: joris

Answer 2

Somebody may have something more clever, but here are two approaches. Assuming you have a dataframe named df with columns ‘Name’ and ‘Year’ you want dummies for.

First, simply iterating over the columns isn’t too bad:

In [93]: for column in ['Name', 'Year']:
    ...:     dummies = pd.get_dummies(df[column])
    ...:     df[dummies.columns] = dummies

Another idea would be to use the patsy package, which is designed to construct data matrices from R-type formulas.

In [94]: patsy.dmatrix(' ~ C(Name) + C(Year)', df, return_type="dataframe")

Answered By: chrisb

Answer 3

Unless I don’t understand the question, it is supported natively in get_dummies by passing the columns argument.

Answered By: sapo_cosmico

Answer 4

With pandas 0.19, you can do that in a single line :

pd.get_dummies(data=df, columns=['A', 'B'])

Columns specifies where to do the One Hot Encoding.

>>> df
   A  B  C
0  a  c  1
1  b  c  2
2  a  b  3

>>> pd.get_dummies(data=df, columns=['A', 'B'])
   C  A_a  A_b  B_b  B_c
0  1  1.0  0.0  0.0  1.0
1  2  0.0  1.0  0.0  1.0
2  3  1.0  0.0  1.0  0.0

Answered By: mxdbld

Answer 5

The simple trick I am currently using is a for-loop.
First separate categorical data from Data Frame by using select_dtypes(include="object"),
then by using for loop apply get_dummies to each column iteratively
as I have shown in code below:

train_cate=train_data.select_dtypes(include="object")
test_cate=test_data.select_dtypes(include="object")
# vectorize catagorical data
for col in train_cate:
    cate1=pd.get_dummies(train_cate[col])
    train_cate[cate1.columns]=cate1
    cate2=pd.get_dummies(test_cate[col])
    test_cate[cate2.columns]=cate2

Answered By: Reyan Ishtiaq

Running get_dummies on several DataFrame columns?

Question:

Answers: