Convert whole dataframe from lower case to upper case with Pandas
Question:
I have a dataframe like the one displayed below:
# Create an example dataframe about a fictional army
raw_data = {'regiment': ['Nighthawks', 'Nighthawks', 'Nighthawks', 'Nighthawks'],
'company': ['1st', '1st', '2nd', '2nd'],
'deaths': ['kkk', 52, '25', 616],
'battles': [5, '42', 2, 2],
'size': ['l', 'll', 'l', 'm']}
df = pd.DataFrame(raw_data, columns = ['regiment', 'company', 'deaths', 'battles', 'size'])
My goal is to transform every single string inside of the dataframe to upper case so that it looks like this:
Notice: all data types are objects and must not be changed; the output must contain all objects. I want to avoid to convert every single column one by one… I would like to do it generally over the whole dataframe possibly.
What I tried so far is to do this but without success
df.str.upper()
Answers:
astype() will cast each series to the dtype object (string) and then call the str() method on the converted series to get the string literally and call the function upper() on it. Note that after this, the dtype of all columns changes to object.
In [17]: df
Out[17]:
regiment company deaths battles size
0 Nighthawks 1st kkk 5 l
1 Nighthawks 1st 52 42 ll
2 Nighthawks 2nd 25 2 l
3 Nighthawks 2nd 616 2 m
In [18]: df.apply(lambda x: x.astype(str).str.upper())
Out[18]:
regiment company deaths battles size
0 NIGHTHAWKS 1ST KKK 5 L
1 NIGHTHAWKS 1ST 52 42 LL
2 NIGHTHAWKS 2ND 25 2 L
3 NIGHTHAWKS 2ND 616 2 M
You can later convert the ‘battles’ column to numeric again, using to_numeric():
In [42]: df2 = df.apply(lambda x: x.astype(str).str.upper())
In [43]: df2['battles'] = pd.to_numeric(df2['battles'])
In [44]: df2
Out[44]:
regiment company deaths battles size
0 NIGHTHAWKS 1ST KKK 5 L
1 NIGHTHAWKS 1ST 52 42 LL
2 NIGHTHAWKS 2ND 25 2 L
3 NIGHTHAWKS 2ND 616 2 M
In [45]: df2.dtypes
Out[45]:
regiment object
company object
deaths object
battles int64
size object
dtype: object
Since str only works for series, you can apply it to each column individually then concatenate:
In [6]: pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
Out[6]:
regiment company deaths battles size
0 NIGHTHAWKS 1ST KKK 5 L
1 NIGHTHAWKS 1ST 52 42 LL
2 NIGHTHAWKS 2ND 25 2 L
3 NIGHTHAWKS 2ND 616 2 M
Edit: performance comparison
In [10]: %timeit df.apply(lambda x: x.astype(str).str.upper())
100 loops, best of 3: 3.32 ms per loop
In [11]: %timeit pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
100 loops, best of 3: 3.32 ms per loop
Both answers perform equally on a small dataframe.
In [15]: df = pd.concat(10000 * [df])
In [16]: %timeit pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
10 loops, best of 3: 104 ms per loop
In [17]: %timeit df.apply(lambda x: x.astype(str).str.upper())
10 loops, best of 3: 130 ms per loop
On a large dataframe my answer is slightly faster.
This can be solved by the following applymap method:
df = df.applymap(lambda s: s.lower() if type(s) == str else s)
If you want to conserve the dtype use isinstance(obj,type)
df.apply(lambda x: x.str.upper().str.strip() if isinstance(x, object) else x)
try this
df2 = df2.apply(lambda x: x.str.upper() if x.dtype == "object" else x)
Loops are very slow instead of using apply function to each and cell in a row, try to get columns names in a list and then loop over list of columns to convert each column text to lowercase.
Code below is the vector operation which is faster than apply function.
for columns in dataset.columns:
dataset[columns] = dataset[columns].str.lower()
if you want conserv dtype or change only one type.. try for and if:
for x in dados.columns:
if dados[x].dtype == 'object':
print('object - allow upper')
dados[x] = dados[x].str.upper()
else:
print('other? - not allow upper')
dados[x] = dados[x].str.upper()
You can apply it for every cols…
oh_df.columns = map(str.lower, oh_df.columns)
I have a dataframe like the one displayed below:
# Create an example dataframe about a fictional army
raw_data = {'regiment': ['Nighthawks', 'Nighthawks', 'Nighthawks', 'Nighthawks'],
'company': ['1st', '1st', '2nd', '2nd'],
'deaths': ['kkk', 52, '25', 616],
'battles': [5, '42', 2, 2],
'size': ['l', 'll', 'l', 'm']}
df = pd.DataFrame(raw_data, columns = ['regiment', 'company', 'deaths', 'battles', 'size'])
My goal is to transform every single string inside of the dataframe to upper case so that it looks like this:
Notice: all data types are objects and must not be changed; the output must contain all objects. I want to avoid to convert every single column one by one… I would like to do it generally over the whole dataframe possibly.
What I tried so far is to do this but without success
df.str.upper()
astype() will cast each series to the dtype object (string) and then call the str() method on the converted series to get the string literally and call the function upper() on it. Note that after this, the dtype of all columns changes to object.
In [17]: df
Out[17]:
regiment company deaths battles size
0 Nighthawks 1st kkk 5 l
1 Nighthawks 1st 52 42 ll
2 Nighthawks 2nd 25 2 l
3 Nighthawks 2nd 616 2 m
In [18]: df.apply(lambda x: x.astype(str).str.upper())
Out[18]:
regiment company deaths battles size
0 NIGHTHAWKS 1ST KKK 5 L
1 NIGHTHAWKS 1ST 52 42 LL
2 NIGHTHAWKS 2ND 25 2 L
3 NIGHTHAWKS 2ND 616 2 M
You can later convert the ‘battles’ column to numeric again, using to_numeric():
In [42]: df2 = df.apply(lambda x: x.astype(str).str.upper())
In [43]: df2['battles'] = pd.to_numeric(df2['battles'])
In [44]: df2
Out[44]:
regiment company deaths battles size
0 NIGHTHAWKS 1ST KKK 5 L
1 NIGHTHAWKS 1ST 52 42 LL
2 NIGHTHAWKS 2ND 25 2 L
3 NIGHTHAWKS 2ND 616 2 M
In [45]: df2.dtypes
Out[45]:
regiment object
company object
deaths object
battles int64
size object
dtype: object
Since str only works for series, you can apply it to each column individually then concatenate:
In [6]: pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
Out[6]:
regiment company deaths battles size
0 NIGHTHAWKS 1ST KKK 5 L
1 NIGHTHAWKS 1ST 52 42 LL
2 NIGHTHAWKS 2ND 25 2 L
3 NIGHTHAWKS 2ND 616 2 M
Edit: performance comparison
In [10]: %timeit df.apply(lambda x: x.astype(str).str.upper())
100 loops, best of 3: 3.32 ms per loop
In [11]: %timeit pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
100 loops, best of 3: 3.32 ms per loop
Both answers perform equally on a small dataframe.
In [15]: df = pd.concat(10000 * [df])
In [16]: %timeit pd.concat([df[col].astype(str).str.upper() for col in df.columns], axis=1)
10 loops, best of 3: 104 ms per loop
In [17]: %timeit df.apply(lambda x: x.astype(str).str.upper())
10 loops, best of 3: 130 ms per loop
On a large dataframe my answer is slightly faster.
This can be solved by the following applymap method:
df = df.applymap(lambda s: s.lower() if type(s) == str else s)
If you want to conserve the dtype use isinstance(obj,type)
df.apply(lambda x: x.str.upper().str.strip() if isinstance(x, object) else x)
try this
df2 = df2.apply(lambda x: x.str.upper() if x.dtype == "object" else x)
Loops are very slow instead of using apply function to each and cell in a row, try to get columns names in a list and then loop over list of columns to convert each column text to lowercase.
Code below is the vector operation which is faster than apply function.
for columns in dataset.columns:
dataset[columns] = dataset[columns].str.lower()
if you want conserv dtype or change only one type.. try for and if:
for x in dados.columns:
if dados[x].dtype == 'object':
print('object - allow upper')
dados[x] = dados[x].str.upper()
else:
print('other? - not allow upper')
dados[x] = dados[x].str.upper()
You can apply it for every cols…
oh_df.columns = map(str.lower, oh_df.columns)