Pandas DataFrame doesn't have the entire data and uses three dots instead
Question:
Got this .txt file
Dia Precio marginal en el sistema español (EUR/MWh) Precio marginal en el sistema portugués (EUR/MWh) Energía total de compra sistema español (MWh) Energía total de venta sistema español (MWh) Energía total de compra sistema portugués (MWh) Energía total de venta sistema portugués (MWh) Energía total del mercado Ibérico (MWh) Energía total con bilaterales del mercado Ibér... Importación de España desde Portugal (MWh) Exportación de España a Portugal (MWh)
1 9.57 9.57 21440.2 21044.6 2214 2609.6 23654.2 29281.5 395.6 0
2 0 0 21568.7 20602.7 1304 2270 22872.7 28254.7 966 0
3 0 0 21727.9 20306.8 1030.5 2451.6 22758.4 27934.1 1421.1 0
4 0 0 21703.8 20103.8 1119.4 2719.4 22823.2 27603.4 1600 0
5 0 0 21050.3 19450.3 1113.9 2713.9 22164.2 26645.1 1600 0
6 0 0 20830.5 19230.5 1114.8 2714.8 21945.3 26364.9 1600 0
7 0 0 20725.9 19125.9 1124.2 2724.2 21850.1 26364.8 1600 0
8 0 0 20790.8 19190.8 1145.5 2745.5 21936.3 26827 1600 0
9 0 0 21348.5 19748.5 1162 2762 22510.5 27289 1600 0
10 0 0 21813.3 20213.3 1183.1 2783.1 22996.4 28189.4 1600 0
11 0 0 22175.3 20575.3 1215.1 2815.1 23390.4 29058 1600 0
12 0 0 22653.6 21053.6 1205.9 2805.9 23859.5 29630 1600 0
13 0 0 22637.9 21286.4 1282.5 2634 23920.4 29812.3 1351.5 0
14 0 0 22686.5 21622.1 1569.6 2634 24256.1 29999.5 1064.4 0
15 0 0 23045 22153.7 1742.7 2634 24787.7 30471.6 891.3 0
16 0 0 22678.6 21986.4 1941.8 2634 24620.4 30206.7 692.2 0
17 0 0 22046.9 21575.7 2162.8 2634 24209.7 29765.1 471.2 0
18 0 0 20583.8 20486.5 2536.7 2634 23120.5 28778.4 97.3 0
19 10 10 20436.8 21197.9 3490.1 2729 23926.9 30031.2 0 761.1
20 15 25.31 20328.1 21428.1 4211.7 3111.7 24539.8 30849.8 0 1100
21 27.32 27.32 20709 21808.9 4334.4 3234.5 25043.4 31449.9 0 1099.9
22 30 30 21466.4 22448.9 4223.5 3241 25689.9 32160.6 0 982.5
23 30.8 30.8 21651.9 22371.4 3980.5 3261 25632.4 31784.1 0 719.5
24 29.1 29.1 20191.1 20725.1 3775 3241 23966.1 29685.7 0 534
While creating a Pandas DataFrame from it, I was getting
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xf1 in position
40: invalid continuation byte
In order to solve that problem, had to specify the encoding to latin-1.
df = pd.read_csv(r'C:file_locationfile_name.txt', encoding='latin-1')
Then, when printing df,
Dia Precio marginal en el sistema español (EUR/MWh) Precio marginal en el sistema portugués (EUR/MWh) Energía total de compra sistema español (MWh) Energía total de venta sistema español (MWh) Energía total de compra sistema portugués (MWh) Energía total de venta sistema portugués (MWh) Energía total del mercado Ibérico (MWh) Energía total con bilaterales del mercado Ibér... Importación de España desde Portugal (MWh) Exportación de España a Portugal (MWh)
0 1 ...
1 2 ...
2 3 ...
3 4 ...
4 5 ...
5 6 ...
6 7 ...
7 8 ...
8 9 ...
9 10 ...
10 11 ...
11 12 ...
12 13 ...
13 14 ...
14 15 ...
15 16 ...
16 17 ...
17 18 ...
18 19 ...
19 20 ...
20 21 ...
21 22 ...
22 23 ...
23 24 ...
The problem with this print result is that instead of showing the values for all the columns (like the file has), it appears the three dots – ….
I tried changing the file to .csv instead of .txt, removing the header, using read_table and writing to a file but kept receiving the same result.
Answers:
You have a lot of columns so your data might be truncated
You can try
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', -1)
to eliminate any kind of column truncation
try this…
import pandas as pd
df = pd.read_csv(r'C:file_locationfile_name.txt', encoding='latin-1')
print(df.to_string())
Note:
Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use "None" to not limit the column width.
pd.set_option('display.max_colwidth', None)
Got this .txt file
Dia Precio marginal en el sistema español (EUR/MWh) Precio marginal en el sistema portugués (EUR/MWh) Energía total de compra sistema español (MWh) Energía total de venta sistema español (MWh) Energía total de compra sistema portugués (MWh) Energía total de venta sistema portugués (MWh) Energía total del mercado Ibérico (MWh) Energía total con bilaterales del mercado Ibér... Importación de España desde Portugal (MWh) Exportación de España a Portugal (MWh)
1 9.57 9.57 21440.2 21044.6 2214 2609.6 23654.2 29281.5 395.6 0
2 0 0 21568.7 20602.7 1304 2270 22872.7 28254.7 966 0
3 0 0 21727.9 20306.8 1030.5 2451.6 22758.4 27934.1 1421.1 0
4 0 0 21703.8 20103.8 1119.4 2719.4 22823.2 27603.4 1600 0
5 0 0 21050.3 19450.3 1113.9 2713.9 22164.2 26645.1 1600 0
6 0 0 20830.5 19230.5 1114.8 2714.8 21945.3 26364.9 1600 0
7 0 0 20725.9 19125.9 1124.2 2724.2 21850.1 26364.8 1600 0
8 0 0 20790.8 19190.8 1145.5 2745.5 21936.3 26827 1600 0
9 0 0 21348.5 19748.5 1162 2762 22510.5 27289 1600 0
10 0 0 21813.3 20213.3 1183.1 2783.1 22996.4 28189.4 1600 0
11 0 0 22175.3 20575.3 1215.1 2815.1 23390.4 29058 1600 0
12 0 0 22653.6 21053.6 1205.9 2805.9 23859.5 29630 1600 0
13 0 0 22637.9 21286.4 1282.5 2634 23920.4 29812.3 1351.5 0
14 0 0 22686.5 21622.1 1569.6 2634 24256.1 29999.5 1064.4 0
15 0 0 23045 22153.7 1742.7 2634 24787.7 30471.6 891.3 0
16 0 0 22678.6 21986.4 1941.8 2634 24620.4 30206.7 692.2 0
17 0 0 22046.9 21575.7 2162.8 2634 24209.7 29765.1 471.2 0
18 0 0 20583.8 20486.5 2536.7 2634 23120.5 28778.4 97.3 0
19 10 10 20436.8 21197.9 3490.1 2729 23926.9 30031.2 0 761.1
20 15 25.31 20328.1 21428.1 4211.7 3111.7 24539.8 30849.8 0 1100
21 27.32 27.32 20709 21808.9 4334.4 3234.5 25043.4 31449.9 0 1099.9
22 30 30 21466.4 22448.9 4223.5 3241 25689.9 32160.6 0 982.5
23 30.8 30.8 21651.9 22371.4 3980.5 3261 25632.4 31784.1 0 719.5
24 29.1 29.1 20191.1 20725.1 3775 3241 23966.1 29685.7 0 534
While creating a Pandas DataFrame from it, I was getting
UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xf1 in position
40: invalid continuation byte
In order to solve that problem, had to specify the encoding to latin-1.
df = pd.read_csv(r'C:file_locationfile_name.txt', encoding='latin-1')
Then, when printing df,
Dia Precio marginal en el sistema español (EUR/MWh) Precio marginal en el sistema portugués (EUR/MWh) Energía total de compra sistema español (MWh) Energía total de venta sistema español (MWh) Energía total de compra sistema portugués (MWh) Energía total de venta sistema portugués (MWh) Energía total del mercado Ibérico (MWh) Energía total con bilaterales del mercado Ibér... Importación de España desde Portugal (MWh) Exportación de España a Portugal (MWh)
0 1 ...
1 2 ...
2 3 ...
3 4 ...
4 5 ...
5 6 ...
6 7 ...
7 8 ...
8 9 ...
9 10 ...
10 11 ...
11 12 ...
12 13 ...
13 14 ...
14 15 ...
15 16 ...
16 17 ...
17 18 ...
18 19 ...
19 20 ...
20 21 ...
21 22 ...
22 23 ...
23 24 ...
The problem with this print result is that instead of showing the values for all the columns (like the file has), it appears the three dots – ….
I tried changing the file to .csv instead of .txt, removing the header, using read_table and writing to a file but kept receiving the same result.
You have a lot of columns so your data might be truncated
You can try
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('display.max_colwidth', -1)
to eliminate any kind of column truncation
try this…
import pandas as pd
df = pd.read_csv(r'C:file_locationfile_name.txt', encoding='latin-1')
print(df.to_string())
Note:
Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use "None" to not limit the column width.
pd.set_option('display.max_colwidth', None)