concatenating multiple columns include NaN in dataframe

Question

i want to concatenate/join many columns include Nan value to one new column.
how to avoid/pass the NaN in join result?
below just to show my try i used both .agg and .apply.

import pandas as pd
import numpy as np
df = pd.DataFrame({'foo':['a',np.nan,'c'], 'bar':[1, 2, 3], 'new':['apple', 'banana', 'pear']})
subcat_names=["foo","new"]

df["result"] = df[subcat_names].astype(str).agg(','.join, axis=1)

df=df.fillna("")

df["result_2"] =df[subcat_names].apply(lambda x : '{},{}'.format(x[0],x[1]), axis=1)

print(df)
    
  foo  bar     new      result result_2
0   a    1   apple     a,apple  a,apple
1        2  banana  nan,banana  ,banana
2   c    3    pear      c,pear   c,pear

at result the nan, is unwanted
at result_2 , is unwanted

thanks

Asked By: Ninja

||

Source

Answer 1

I think that the second option is almost correct, you just have to implement your lambda in a bit more involved way. The following is pseudocode and it’s not tested:

def process(row):
    filtered = list()

    for item in row:
        if np.isnan(item).any():
            continue

        filtered.append(item)

    return ",".join(filtered)

df["result_2"] =df[subcat_names].apply(process, axis=1)

Most likely you could rely on not_na pandas function to collect valid values out of current row

Answered By: CaptainTrunky

Answer 2

You can try pd.notnull()

subcat_names = ["foo", "new"]
df["result"] = df[subcat_names].apply(lambda x: ",".join(x[pd.notnull(x)]), axis=1)
print(df)

Output:

   foo  bar     new   result
0    a    1   apple  a,apple
1         2  banana   banana
2    c    3    pear   c,pear

Answered By: Jamiu Shaibu

concatenating multiple columns include NaN in dataframe

Question:

Answers: