Pandas Merge – CSV Column data is offset/misaligned

Question

I have a merged two dataframes where the values are not starting from the top row when looking at the CSV file created.

import pandas as pd

df_ifOperStatus = pd.read_csv('WG_ifOperStatus.csv')
df_IfDescr = pd.read_csv('TEST_ifDescr.csv')

merge = df_IfDescr.merge(df_ifOperStatus, left_on='Desc', right_on='if', how='left')
merge.to_csv('MergeTEST.csv', index=False)

When viewing the combined dataset the column data is not together. The merge combined two CSV with two columns each, giving more four columns in the merged file.

Instead i get four columns where dataset two begins on row 100 for example.

Desc         ONT            if  ifOperStatus 
17972705281  ADTN2019402e       
17972715521  ADTN2000604b       
17972725761  ADTN201942fc       
17972736001  ADTN20124a09       
17972746241  ADTN20002d95       
17972756481  ADTN2100a0b6       
                            1   up
                            2   down
                            3   down
                            4   down
                            5   down
                            6   down

I added axis=1 when using concat and it fixed the issue, but I need merge for this use case.

I also need Column 1 and 3 to match to and filter out anything in 3 that is not in 1. But that may be another post.

I’m sure I’ve missed something simple but this is new to me and I haven’t found a post or YouTube video describing this issue yet.

Asked By: Nuclear7740

||

Source

Answer 1

You want to merge by index. Therefore you should:

merge = df_IfDescr.merge(df_ifOperStatus, left_index=True, right_index=True, how='left')

If you want to leave values where ‘if’ equals ‘Desc’:

merge = merge.reset_index()
merge = merge[merge['if'] == merge['Desc']]

This will work only if both columns are of the same type, i.e. both Integer or both Object

Answered By: gtomer

Pandas Merge – CSV Column data is offset/misaligned

Question:

Answers: