Join or Merge Pandas Dataframes filling null values


I have two dataframes – Deployment and HPA as shown. Both dataframes mostly share the values in columns Deployment-Label and HPA-Label which need to merge/join by filling-up null values in all other columns in both dataframes (wherever the values are missing in either of columns Deployment-Label and HPA-Label.)

Tried using pd.merge, but the it is dropping the non-common rows in both df.

# Merge the two dataframes, using LABEL common col, after renaming
df3 = pd.merge(df1, df2, on = 'LABEL')

# Write it to a new CSV file
Deployment-Label Deployment Table – Other Columns
accountdataservice 1
accountservice-grpc 2
fmsdataservice 3
fuelbenchmarkservice-grpc 4
fueldeviationservice-grpc 5
httpclientservice 6
packageservice-grpc 7
provisioningdataservice 8
traefik-ingress-ilb 9
translationservice-grpc 10
HPA-Label HPA Table – Other Columns
accountdataservice A
accountservice-grpc B
fmsdataservice C
fuelbenchmarkservice-grpc D
fueldeviationservice-grpc E
hangfire F
httpclientservice G
packageservice-grpc H
portalservicerest I
provisioningdataservice J
schedularservice-grpc K
translationservice-grpc L

Expected Outcome:

enter image description here

Asked By: Rajesh Swarnkar



You can pass how="outer" and sort=True as keyword arguments to pd.merge :

df3 = pd.merge(df1, df2, on = "LABEL", how="outer", sort=True)

#or given your MRE, use df3 = pd.merge(df1, df2, left_on="Deployment-Label",
#                                     right_on = "HPA-Label", how="outer", sort=True)

​Output :


             Deployment-Label  Deployment Table - Other Columns                  HPA-Label HPA Table - Other Columns
0          accountdataservice                               1.0         accountdataservice                         A
1         accountservice-grpc                               2.0        accountservice-grpc                         B
2              fmsdataservice                               3.0             fmsdataservice                         C
3   fuelbenchmarkservice-grpc                               4.0  fuelbenchmarkservice-grpc                         D
4   fueldeviationservice-grpc                               5.0  fueldeviationservice-grpc                         E
5                         NaN                               NaN                   hangfire                         F
6           httpclientservice                               6.0          httpclientservice                         G
7         packageservice-grpc                               7.0        packageservice-grpc                         H
8                         NaN                               NaN          portalservicerest                         I
9     provisioningdataservice                               8.0    provisioningdataservice                         J
10                        NaN                               NaN      schedularservice-grpc                         K
11        traefik-ingress-ilb                               9.0                        NaN                       NaN
12    translationservice-grpc                              10.0    translationservice-grpc                         L
Answered By: Timeless
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.