Create Pandas date column from fix starting date and offset days as integer colum

I have the following one-column Pandas data frame:

num_days
--------
9236
9601
9636
10454

Here the integers are number of days counted from a constant predefined date:

START_DATE = pd.to_datetime('1980-01-01')

Now I want to have a column with dates (calculated as START_DATE + the respective num_days) like this:

num_days, date
--------------
9236      '2005-04-15'
9601      '2006-04-15'
9636      '2006-05-20'
10454     '2008-08-15'

I have tried this:

df['date'] = pd.to_datetime('1980-01-01') + timedelta(df.num_days)

but no success:

TypeError: unsupported type for timedelta days component: Series