How to write a Pandas DataFrame to Django model

Question:

I have been using pandas in python and I usually write a dataframe to my db table as below. I am now migrating to Django, how can I write the same dataframe to a table through a model called MyModel? Assistance really appreciated.

# Original pandas code
    engine = create_engine('postgresql://myuser:mypassword@localhost:5432/mydb', echo=False)
    mydataframe.to_sql('mytable', engine, if_exists='append', index=True)
Asked By: Avagut

||
Source

Answers:

Use your own pandas code along side a Django model that is mapped to the same SQL table

I am not aware of any explicit support to write a pandas dataframe to a Django model. However, in a Django app, you can still use your own code to read or write to the database, in addition to using the ORM (e.g. through your Django model)

And given that you most likely have data in the database previously written by pandas’ to_sql, you can keep using the same database and the same pandas code and simply create a Django model that can access that table

e.g. if your pandas code was writing to SQL table mytable, simply create a model like this:

class MyModel(Model):
    class Meta:
        db_table = 'mytable' # This tells Django where the SQL table is
        managed = False # Use this if table already exists
                        # and doesn't need to be managed by Django

    field_1 = ...
    field_2 = ...

Now you can use this model from Django simultaneously with your existing pandas code (possibly in a single Django app)

Django database settings

To get the same DB credentials into the pandas SQL functions simply read the fields from Django settings, e.g.:

from django.conf import settings

user = settings.DATABASES['default']['USER']
password = settings.DATABASES['default']['PASSWORD']
database_name = settings.DATABASES['default']['NAME']
# host = settings.DATABASES['default']['HOST']
# port = settings.DATABASES['default']['PORT']

database_url = 'postgresql://{user}:{password}@localhost:5432/{database_name}'.format(
    user=user,
    password=password,
    database_name=database_name,
)

engine = create_engine(database_url, echo=False)

The alternative is not recommended as it’s inefficient

I don’t really see a way beside reading the dataframe row by row and then creating a model instance, and saving it, which is really slow. You might get away with some batch insert operation, but why bother since pandas’ to_sql already does that for us. And reading Django querysets into a pandas dataframe is just inefficient when pandas can do that faster for us too.

# Doing it like this is slow
for index, row in df.iterrows():
     model = MyModel()
     model.field_1 = row['field_1']
     model.save()
Answered By: bakkal

I’m just going through the same exercise at the moment. The approach I’ve taken is to create a list of new objects from the DataFrame and then bulk create them:

bulk_create(objs, batch_size=None)

This method inserts the provided list of objects into the database in an efficient manner (generally only 1 query, no matter how many objects there are)

An example might look like this:

# Not able to iterate directly over the DataFrame
df_records = df.to_dict('records')

model_instances = [MyModel(
    field_1=record['field_1'],
    field_2=record['field_2'],
) for record in df_records]

MyModel.objects.bulk_create(model_instances)
Answered By: Jon Hannah

My solution using pickle and optionally zlib for compression

import pickle
#optional
#import zlib

class SuperModel(models.Model):
    DFToStore = models.BinaryField(default=None, null=True, blank=True)

    def save(self, *args, **kwargs):
        if not isinstance(self.DFToStore, (bytes)):
            self.DFToStore = pickle.dumps(self.DFToStore)
            #optional with compression
            #self.DFToStore = zlib.compress(pickle.dumps(self.DFToStore))
        super(SuperModel, self).save(*args, **kwargs)

    def get_DFToStore(self):
        if isinstance(self.DFToStore, (bytes)):
            return pickle.loads(self.DFToStore)
            #optional with compression
            #return pickle.loads(zlib.decompress(self.DFToStore))
        if not isinstance(self.DFToStore, (bytes)):
            return self.DFToStore
Answered By: Laird Foret