How to merge a Series and DataFrame

Here’s one way:

df.join(pd.DataFrame(s).T).fillna(method='ffill')

To break down what happens here…

pd.DataFrame(s).T creates a one-row DataFrame from s which looks like this:

   s1  s2
0   5   6

Next, join concatenates this new frame with df:

   a  b  s1  s2
0  1  3   5   6
1  2  4 NaN NaN

Lastly, the NaN values at index 1 are filled with the previous values in the column using fillna with the forward-fill (ffill) argument:

   a  b  s1  s2
0  1  3   5   6
1  2  4   5   6

---

To avoid using fillna, it’s possible to use pd.concat to repeat the rows of the DataFrame constructed from s. In this case, the general solution is:

df.join(pd.concat([pd.DataFrame(s).T] * len(df), ignore_index=True))

---

Here’s another solution to address the indexing challenge posed in the edited question:

df.join(pd.DataFrame(s.repeat(len(df)).values.reshape((len(df), -1), order='F'), 
        columns=s.index, 
        index=df.index))

s is transformed into a DataFrame by repeating the values and reshaping (specifying ‘Fortran’ order), and also passing in the appropriate column names and index. This new DataFrame is then joined to df.

Update
From v0.24.0 onwards, you can merge on DataFrame and Series as long as the Series is named.

df.merge(s.rename('new'), left_index=True, right_index=True)
# If series is already named,
# df.merge(s, left_index=True, right_index=True)

---

Nowadays, you can simply convert the Series to a DataFrame with to_frame(). So (if joining on index):

df.merge(s.to_frame(), left_index=True, right_index=True)

If I could suggest setting up your dataframes like this (auto-indexing):

df = pd.DataFrame({'a':[np.nan, 1, 2], 'b':[4, 5, 6]})

then you can set up your s1 and s2 values thus (using shape() to return the number of rows from df):

s = pd.DataFrame({'s1':[5]*df.shape[0], 's2':[6]*df.shape[0]})

then the result you want is easy:

display (df.merge(s, left_index=True, right_index=True))

Alternatively, just add the new values to your dataframe df:

df = pd.DataFrame({'a':[nan, 1, 2], 'b':[4, 5, 6]})
df['s1']=5
df['s2']=6
display(df)

Both return:

     a  b  s1  s2
0  NaN  4   5   6
1  1.0  5   5   6
2  2.0  6   5   6

If you have another list of data (instead of just a single value to apply), and you know it is in the same sequence as df, eg:

s1=['a','b','c']

then you can attach this in the same way:

df['s1']=s1

returns:

     a  b s1
0  NaN  4  a
1  1.0  5  b
2  2.0  6  c

You can easily set a pandas.DataFrame column to a constant. This constant can be an int such as in your example. If the column you specify isn’t in the df, then pandas will create a new column with the name you specify. So after your dataframe is constructed, (from your question):

df = pd.DataFrame({'a':[np.nan, 2, 3], 'b':[4, 5, 6]}, index=[3, 5, 6])

You can just run:

df['s1'], df['s2'] = 5, 6

You could write a loop or comprehension to make it do this for all the elements in a list of tuples, or keys and values in a dictionary depending on how you have your real data stored.

If df is a pandas.DataFrame then df['new_col']= Series list_object of length len(df) will add the or Series list_object as a column named 'new_col'. df['new_col']= scalar (such as 5 or 6 in your case) also works and is equivalent to df['new_col']= [scalar]*len(df)

So a two-line code serves the purpose:

df = pd.DataFrame({'a':[1, 2], 'b':[3, 4]})
s = pd.Series({'s1':5, 's2':6})
for x in s.index:    
    df[x] = s[x]

Output: 
   a  b  s1  s2
0  1  3   5   6
1  2  4   5   6

Nowadays, much simpler and concise solution can achieve the same task. Leveraging the capability of DataFrame.apply() to turn a Series into columns of its belonging DataFrame, we can use:

df.join(df.apply(lambda x: s, axis=1))

Result:

     a  b  s1  s2
3  NaN  4   5   6
5  2.0  5   5   6
6  3.0  6   5   6

Here, we used DataFrame.apply() with a simple lambda function as the applied function on axis=1. The applied lambda function simply just returns the Series s:

df.apply(lambda x: s, axis=1)

Result:

   s1  s2
3   5   6
5   5   6
6   5   6

The result has already inherited the row index of the original DataFrame df. Consequently, we can simply join df with this interim result by DataFrame.join() to get the desired final result (since they have the same row index).

This capability of DataFrame.apply() to turn a Series into columns of its belonging DataFrame is well documented in the official document as follows:

By default (result_type=None), the final return type is inferred from
the return type of the applied function.

---

The default behaviour (result_type=None) depends on the return value of the
applied function: list-like results will be returned as a Series of
those. However if the apply function returns a Series these are
expanded to columns.

The official document also includes example of such usage:

Returning a Series inside the function is similar to passing
result_type=’expand’. The resulting column names will be the Series
index.

df.apply(lambda x: pd.Series([1, 2], index=['foo', 'bar']), axis=1)   

   foo  bar
0    1    2
1    1    2
2    1    2