pd.read_csv: delimiter = 't' and header=None not compatible
Question:
I have this line of code:
df = pd.read_csv('some_file.txt',engine ='python',
delimiter = 't', header=None, encoding="utf-16")
I’m using those txt files quiet often in my lab, one of our machines gives them as output.
If I only use the delimiter I get a nice table, but with the first element as header for everything. If I only use header = None I get rid of the header, but have a bunch of t everywhere. If I try to use both commands, I get this error:
ParserError: Expected 1 fields in line 3, saw 23
When removing enigne = ‘python’ I get a similar error.
(also tried seperator and a bunch of other things)
Help would be very much appreciated!
Edit: As requested that’s how the file looks like:
##BLOCKS= 1
Plate: Plate1 1.3 PlateFormat Endpoint Absorbance Raw FALSE 1 1 562 1 12 96 1 8
Temperature(¡C) 1 2 3 4 5 6 7 8 9 10 11 12
26.5 0.8368 0.5211 0.321 0.2707 0.2124 0.1768 0.1694 0.1635 0.1659 0.1029 0.1032 0.104
0.7142 0.4866 0.2968 0.252 0.2111 0.1737 0.1633 0.162 0.1599 0.1009 0.1007 0.1025
0.3499 0.2119 0.2799 0.2097 0.3114 0.3393 0.2544 0.2965 0.2392 0.3063 0.3093 0.2655
0.305 0.2068 0.2573 0.2008 0.287 0.2765 0.2373 0.2703 0.2357 0.2865 0.2926 0.263
0.2922 0.3456 0.1964 0.2667 0.3022 0.2596 0.2256 0.2387 0.2498 0.2936 0.2396 0.3411
0.3018 0.349 0.2069 0.272 0.2926 0.2444 0.2141 0.2348 0.2486 0.2678 0.2346 0.2944
0.2965 0.3505 0.2427 0.3322 0.1873 0.2286 0.3758 0.208 0.3023 0.3573 0.3141 0.2658
0.2956 0.3155 0.2514 0.2929 0.1985 0.2379 0.1898 0.2101 0.3211 0.3558 0.3121 0.2567
~End
Original Filename: 20220725_Benedikt_DEF; Date Last Saved: 7/25/2022 2:31:30 PM
That’s how it looks like when I read it without pandas:
[‘##BLOCKS= 1n’, ‘Plate:tPlate1t1.3tPlateFormattEndpointtAbsorbancetRawtFALSEt1tttttt1t562 t1t12t96t1t8ttn’, ‘tTemperature(¡C)t1t2t3t4t5t6t7t8t9t10t11t12ttn’, ‘t26.5t0.8368t0.5211t0.321t0.2707t0.2124t0.1768t0.1694t0.1635t0.1659t0.1029t0.1032t0.104ttn’, ‘tt0.7142t0.4866t0.2968t0.252t0.2111t0.1737t0.1633t0.162t0.1599t0.1009t0.1007t0.1025ttn’, ‘tt0.3499t0.2119t0.2799t0.2097t0.3114t0.3393t0.2544t0.2965t0.2392t0.3063t0.3093t0.2655ttn’, ‘tt0.305t0.2068t0.2573t0.2008t0.287t0.2765t0.2373t0.2703t0.2357t0.2865t0.2926t0.263ttn’, ‘tt0.2922t0.3456t0.1964t0.2667t0.3022t0.2596t0.2256t0.2387t0.2498t0.2936t0.2396t0.3411ttn’, ‘tt0.3018t0.349t0.2069t0.272t0.2926t0.2444t0.2141t0.2348t0.2486t0.2678t0.2346t0.2944ttn’, ‘tt0.2965t0.3505t0.2427t0.3322t0.1873t0.2286t0.3758t0.208t0.3023t0.3573t0.3141t0.2658ttn’, ‘tt0.2956t0.3155t0.2514t0.2929t0.1985t0.2379t0.1898t0.2101t0.3211t0.3558t0.3121t0.2567ttn’, ‘n’, ‘~Endn’, ‘Original Filename: some_file; Date Last Saved: 7/25/2022 2:31:30 PMn’]
When I use just use the pd.read_csv(file, encoding =”utf-16′)
I get this:
It’ basically a file that is stating the wavelength absorbance from a sample plate with 8 rows and 12 columns (96 samples).
Answers:
Assuming all the files have the same structure and you only want the data; skip the first four rows, don’t use the last three rows, whitespace delimiter, no header, python engine.
>>> df = pd.read_csv(csv,skiprows=4,skipfooter=3,header=None,delim_whitespace=True,engine='python')
>>> df
0 1 2 3 4 5 6 7 8 9 10 11
0 0.7142 0.4866 0.2968 0.2520 0.2111 0.1737 0.1633 0.1620 0.1599 0.1009 0.1007 0.1025
1 0.3499 0.2119 0.2799 0.2097 0.3114 0.3393 0.2544 0.2965 0.2392 0.3063 0.3093 0.2655
2 0.3050 0.2068 0.2573 0.2008 0.2870 0.2765 0.2373 0.2703 0.2357 0.2865 0.2926 0.2630
3 0.2922 0.3456 0.1964 0.2667 0.3022 0.2596 0.2256 0.2387 0.2498 0.2936 0.2396 0.3411
4 0.3018 0.3490 0.2069 0.2720 0.2926 0.2444 0.2141 0.2348 0.2486 0.2678 0.2346 0.2944
5 0.2965 0.3505 0.2427 0.3322 0.1873 0.2286 0.3758 0.2080 0.3023 0.3573 0.3141 0.2658
6 0.2956 0.3155 0.2514 0.2929 0.1985 0.2379 0.1898 0.2101 0.3211 0.3558 0.3121 0.2567
>>>
I have this line of code:
df = pd.read_csv('some_file.txt',engine ='python',
delimiter = 't', header=None, encoding="utf-16")
I’m using those txt files quiet often in my lab, one of our machines gives them as output.
If I only use the delimiter I get a nice table, but with the first element as header for everything. If I only use header = None I get rid of the header, but have a bunch of t everywhere. If I try to use both commands, I get this error:
ParserError: Expected 1 fields in line 3, saw 23
When removing enigne = ‘python’ I get a similar error.
(also tried seperator and a bunch of other things)
Help would be very much appreciated!
Edit: As requested that’s how the file looks like:
##BLOCKS= 1
Plate: Plate1 1.3 PlateFormat Endpoint Absorbance Raw FALSE 1 1 562 1 12 96 1 8
Temperature(¡C) 1 2 3 4 5 6 7 8 9 10 11 12
26.5 0.8368 0.5211 0.321 0.2707 0.2124 0.1768 0.1694 0.1635 0.1659 0.1029 0.1032 0.104
0.7142 0.4866 0.2968 0.252 0.2111 0.1737 0.1633 0.162 0.1599 0.1009 0.1007 0.1025
0.3499 0.2119 0.2799 0.2097 0.3114 0.3393 0.2544 0.2965 0.2392 0.3063 0.3093 0.2655
0.305 0.2068 0.2573 0.2008 0.287 0.2765 0.2373 0.2703 0.2357 0.2865 0.2926 0.263
0.2922 0.3456 0.1964 0.2667 0.3022 0.2596 0.2256 0.2387 0.2498 0.2936 0.2396 0.3411
0.3018 0.349 0.2069 0.272 0.2926 0.2444 0.2141 0.2348 0.2486 0.2678 0.2346 0.2944
0.2965 0.3505 0.2427 0.3322 0.1873 0.2286 0.3758 0.208 0.3023 0.3573 0.3141 0.2658
0.2956 0.3155 0.2514 0.2929 0.1985 0.2379 0.1898 0.2101 0.3211 0.3558 0.3121 0.2567
~End
Original Filename: 20220725_Benedikt_DEF; Date Last Saved: 7/25/2022 2:31:30 PM
That’s how it looks like when I read it without pandas:
[‘##BLOCKS= 1n’, ‘Plate:tPlate1t1.3tPlateFormattEndpointtAbsorbancetRawtFALSEt1tttttt1t562 t1t12t96t1t8ttn’, ‘tTemperature(¡C)t1t2t3t4t5t6t7t8t9t10t11t12ttn’, ‘t26.5t0.8368t0.5211t0.321t0.2707t0.2124t0.1768t0.1694t0.1635t0.1659t0.1029t0.1032t0.104ttn’, ‘tt0.7142t0.4866t0.2968t0.252t0.2111t0.1737t0.1633t0.162t0.1599t0.1009t0.1007t0.1025ttn’, ‘tt0.3499t0.2119t0.2799t0.2097t0.3114t0.3393t0.2544t0.2965t0.2392t0.3063t0.3093t0.2655ttn’, ‘tt0.305t0.2068t0.2573t0.2008t0.287t0.2765t0.2373t0.2703t0.2357t0.2865t0.2926t0.263ttn’, ‘tt0.2922t0.3456t0.1964t0.2667t0.3022t0.2596t0.2256t0.2387t0.2498t0.2936t0.2396t0.3411ttn’, ‘tt0.3018t0.349t0.2069t0.272t0.2926t0.2444t0.2141t0.2348t0.2486t0.2678t0.2346t0.2944ttn’, ‘tt0.2965t0.3505t0.2427t0.3322t0.1873t0.2286t0.3758t0.208t0.3023t0.3573t0.3141t0.2658ttn’, ‘tt0.2956t0.3155t0.2514t0.2929t0.1985t0.2379t0.1898t0.2101t0.3211t0.3558t0.3121t0.2567ttn’, ‘n’, ‘~Endn’, ‘Original Filename: some_file; Date Last Saved: 7/25/2022 2:31:30 PMn’]
When I use just use the pd.read_csv(file, encoding =”utf-16′)
I get this:
It’ basically a file that is stating the wavelength absorbance from a sample plate with 8 rows and 12 columns (96 samples).
Assuming all the files have the same structure and you only want the data; skip the first four rows, don’t use the last three rows, whitespace delimiter, no header, python engine.
>>> df = pd.read_csv(csv,skiprows=4,skipfooter=3,header=None,delim_whitespace=True,engine='python')
>>> df
0 1 2 3 4 5 6 7 8 9 10 11
0 0.7142 0.4866 0.2968 0.2520 0.2111 0.1737 0.1633 0.1620 0.1599 0.1009 0.1007 0.1025
1 0.3499 0.2119 0.2799 0.2097 0.3114 0.3393 0.2544 0.2965 0.2392 0.3063 0.3093 0.2655
2 0.3050 0.2068 0.2573 0.2008 0.2870 0.2765 0.2373 0.2703 0.2357 0.2865 0.2926 0.2630
3 0.2922 0.3456 0.1964 0.2667 0.3022 0.2596 0.2256 0.2387 0.2498 0.2936 0.2396 0.3411
4 0.3018 0.3490 0.2069 0.2720 0.2926 0.2444 0.2141 0.2348 0.2486 0.2678 0.2346 0.2944
5 0.2965 0.3505 0.2427 0.3322 0.1873 0.2286 0.3758 0.2080 0.3023 0.3573 0.3141 0.2658
6 0.2956 0.3155 0.2514 0.2929 0.1985 0.2379 0.1898 0.2101 0.3211 0.3558 0.3121 0.2567
>>>