Saving a Pandas DataFrame to a Django Model

Question

I have stock price data that is stored in a pandas DataFrame as shown below (actually it was in a panel, but I converted it to a DataFrame)

        date  ticker  close       tsr
0 2013-03-28  abc     22.81  1.000439
1 2013-03-28  def     94.21  1.006947
2 2013-03-28  ghi     95.84  1.014180
3 2013-03-28  jkl     31.80  1.000000
4 2013-03-28  mno     32.10  1.003125
...many more rows

I want to save this in a Django model, which looks like this (matches the column names):

class HistoricalPrices(models.Model):
    ticker = models.CharField(max_length=10)
    date = models.DateField()
    tsr = models.DecimalField()
    close = models.DecimalField()

The best I’ve come up so far is using this to save it, where df is my DataFrame:

entries = []
for e in df.T.to_dict().values():
    entries.append(HistoricalPrices(**e))
HistoricalPrices.objects.bulk_create(entries)

Is there a better way to save this?

I’ve looked at django-pandas, but looks like it just reads from the DB.

Asked By: Kritz

||

Source

Answer 1

It would be most efficient to use to_sql() with appropriate connection parameters for the engine, and run this inside your Django app rather than iterating through the DataFrame and saving one model instance at a time:

from sqlalchemy import create_engine
from django.conf import settings

user = settings.DATABASES['default']['USER']
password = settings.DATABASES['default']['PASSWORD']
database_name = settings.DATABASES['default']['NAME']

database_url = 'postgresql://{user}:{password}@localhost:5432/{database_name}'.format(
    user=user,
    password=password,
    database_name=database_name,
)

engine = create_engine(database_url, echo=False)
df.to_sql(HistoricalPrices, con=engine)

Answered By: Stefan

Answer 2

Easier way , you can try this :

json_list = json.loads(json.dumps(list(df.T.to_dict().values())))

for dic in json_list:
     HistoricalPrices.objects.get_or_create(**dic)

Answered By: Amir.S

Saving a Pandas DataFrame to a Django Model

Question:

Answers: