Trim leading zero's using python pandas without changing the datatype of any columns
Question:
I have a csv file of around 42000 lines and around 80 columns, from which I need to remove leading Zero’s, hence I am using Pandas to_csv and saving it back to text file by which leading Zero’s are removed.
Any column may contain null values in any row, but those columns are getting converted to Float datatype and getting decimal values as output, I want to avoid that scenario.
For example, below is a sample of my original file.
0000055|O|Price Rite Marketplace|361|1600 Memorial Dr|Chicopee|MA|010203933|Chicopee|25|013|USA|05|1|H|C|42.2001|-72.5731|A|250138113012012|||10
0000071|O|Big Es Supermarket|189|11 Union St|Easthampton|MA|010271417|Easthampton|25|015|USA|05|5|A|I|42.2697|-72.6717|A|250158224021037||
0000084|O|Big Y Supermarket|14|441 N Main St|East Longmeadow|MA|010281804|East Longmeadow|25|013|USA|05|5|G|C|42.0788|-72.5280|A|250138134012011|||15
0000101|O|Stop & Shop|95|440 Russell St|Hadley|MA|010359566|Hadley|25|015|USA|05|5|K|C|42.3644|-72.5382|A|250158214004004|||14
0000139|O|Key Food Marketplace|2508|13 Cabot St|Holyoke|MA|010406055|Holyoke|25|013|USA|05|5|A|C|42.1980|-72.6042|A|250138115002019|||06
0000149|O|Stop & Shop|9|28 Lincoln St|Holyoke|MA|010403472|Holyoke|25|013|USA|05|5|K|C|42.2150|-72.6172|A|250138118005012|||13
I used the below method to convert to
import pandas as pd
df = pd.read_csv(r"/home/ter/stest/cminxte1.txt", sep="|")
df.to_csv(r"/home/ter/stest/cminxte.txt", sep='|', index=False)
The output file looks like
55|O|Price Rite Marketplace|361|1600 Memorial Dr|Chicopee|MA|10203933|Chicopee|25|13|USA|5|1|H|C|42.2001|-72.5731|A|250138113012012|||10.0
71|O|Big Es Supermarket|189|11 Union St|Easthampton|MA|10271417|Easthampton|25|15|USA|5|5|A|I|42.2697|-72.6717|A|250158224021037||
84|O|Big Y Supermarket|14|441 N Main St|East Longmeadow|MA|10281804|East Longmeadow|25|13|USA|5|5|G|C|42.0788|-72.528|A|250138134012011|||15.0
101|O|Stop & Shop|95|440 Russell St|Hadley|MA|10359566|Hadley|25|15|USA|5|5|K|C|42.3644|-72.5382|A|250158214004004|||14.0
139|O|Key Food Marketplace|2508|13 Cabot St|Holyoke|MA|10406055|Holyoke|25|13|USA|5|5|A|C|42.198|-72.6042|A|250138115002019|||6.0
149|O|Stop & Shop|9|28 Lincoln St|Holyoke|MA|10403472|Holyoke|25|13|USA|5|5|K|C|42.215|-72.6172|A|250138118005012|||13.0
It has removed all the leading Zero’s in all the columns as expected, however, at the last column, it is converting to float with decimal values as that column has got null values. Any idea on how can this be rectified?
Answers:
First convert all values to strings and in next step remove trailing zeros:
df = pd.read_csv(r"/home/ter/stest/cminxte1.txt", sep="|", dtype=str)
df = df.apply(lambda x: x.str.lstrip('0'))
df.to_csv(r"/home/ter/stest/cminxte.txt", sep='|', index=False)
I have a csv file of around 42000 lines and around 80 columns, from which I need to remove leading Zero’s, hence I am using Pandas to_csv and saving it back to text file by which leading Zero’s are removed.
Any column may contain null values in any row, but those columns are getting converted to Float datatype and getting decimal values as output, I want to avoid that scenario.
For example, below is a sample of my original file.
0000055|O|Price Rite Marketplace|361|1600 Memorial Dr|Chicopee|MA|010203933|Chicopee|25|013|USA|05|1|H|C|42.2001|-72.5731|A|250138113012012|||10
0000071|O|Big Es Supermarket|189|11 Union St|Easthampton|MA|010271417|Easthampton|25|015|USA|05|5|A|I|42.2697|-72.6717|A|250158224021037||
0000084|O|Big Y Supermarket|14|441 N Main St|East Longmeadow|MA|010281804|East Longmeadow|25|013|USA|05|5|G|C|42.0788|-72.5280|A|250138134012011|||15
0000101|O|Stop & Shop|95|440 Russell St|Hadley|MA|010359566|Hadley|25|015|USA|05|5|K|C|42.3644|-72.5382|A|250158214004004|||14
0000139|O|Key Food Marketplace|2508|13 Cabot St|Holyoke|MA|010406055|Holyoke|25|013|USA|05|5|A|C|42.1980|-72.6042|A|250138115002019|||06
0000149|O|Stop & Shop|9|28 Lincoln St|Holyoke|MA|010403472|Holyoke|25|013|USA|05|5|K|C|42.2150|-72.6172|A|250138118005012|||13
I used the below method to convert to
import pandas as pd
df = pd.read_csv(r"/home/ter/stest/cminxte1.txt", sep="|")
df.to_csv(r"/home/ter/stest/cminxte.txt", sep='|', index=False)
The output file looks like
55|O|Price Rite Marketplace|361|1600 Memorial Dr|Chicopee|MA|10203933|Chicopee|25|13|USA|5|1|H|C|42.2001|-72.5731|A|250138113012012|||10.0
71|O|Big Es Supermarket|189|11 Union St|Easthampton|MA|10271417|Easthampton|25|15|USA|5|5|A|I|42.2697|-72.6717|A|250158224021037||
84|O|Big Y Supermarket|14|441 N Main St|East Longmeadow|MA|10281804|East Longmeadow|25|13|USA|5|5|G|C|42.0788|-72.528|A|250138134012011|||15.0
101|O|Stop & Shop|95|440 Russell St|Hadley|MA|10359566|Hadley|25|15|USA|5|5|K|C|42.3644|-72.5382|A|250158214004004|||14.0
139|O|Key Food Marketplace|2508|13 Cabot St|Holyoke|MA|10406055|Holyoke|25|13|USA|5|5|A|C|42.198|-72.6042|A|250138115002019|||6.0
149|O|Stop & Shop|9|28 Lincoln St|Holyoke|MA|10403472|Holyoke|25|13|USA|5|5|K|C|42.215|-72.6172|A|250138118005012|||13.0
It has removed all the leading Zero’s in all the columns as expected, however, at the last column, it is converting to float with decimal values as that column has got null values. Any idea on how can this be rectified?
First convert all values to strings and in next step remove trailing zeros:
df = pd.read_csv(r"/home/ter/stest/cminxte1.txt", sep="|", dtype=str)
df = df.apply(lambda x: x.str.lstrip('0'))
df.to_csv(r"/home/ter/stest/cminxte.txt", sep='|', index=False)