psycopg2 – Inserting list of dictionaries into PosgreSQL database. Too many executions?
Question:
I am inserting a list of dictionaries to a PostgreSQL database. The list will be growing quickly and the number of dict values (columns) is around 30. The simplified data:
projects = [
{'name': 'project alpha', 'code': 12, 'active': True},
{'name': 'project beta', 'code': 25, 'active': True},
{'name': 'project charlie', 'code': 46, 'active': False}
]
Inserting the data into the PostgreSQL database with the following code does work (as in this answer), but I am worried about executing too many queries.
for project in projects:
columns = project.keys()
values = project.values()
query = """INSERT INTO projects (%s) VALUES %s;"""
# print(cursor.mogrify(query, (AsIs(','.join(project.keys())), tuple(project.values()))))
cursor.execute(query, (AsIs(','.join(columns)), tuple(values)))
conn.commit()
Is there a better practice? Thank you so much in advance for your help!
Answers:
You can use bulk loading to make it faster.
https://www.citusdata.com/blog/2017/11/08/faster-bulk-loading-in-postgresql-with-copy/
Use execute_values() to insert many rows in a single query.
import psycopg2
from psycopg2.extras import execute_values
# ...
projects = [
{'name': 'project alpha', 'code': 12, 'active': True},
{'name': 'project beta', 'code': 25, 'active': True},
{'name': 'project charlie', 'code': 46, 'active': False}
]
columns = projects[0].keys()
query = "INSERT INTO projects ({}) VALUES %s".format(','.join(columns))
# convert projects values to list of lists
values = [[value for value in project.values()] for project in projects]
execute_values(cursor, query, values)
conn.commit()
Another performant option that does not require so much data munging for a list of dictionaries is execute_batch (new in psycopg2 version 2.7).
For example:
import psycopg2
from psycopg2.extras import execute_batch
values = [{'name': 'project alpha', 'code': 12, 'active': True}, ...]
query = "INSERT INTO projects VALUES (%(name)s, %(code)s, %(active)s)"
execute_batch(cursor, query, values)
conn.commit()
https://www.psycopg.org/docs/extras.html#psycopg2.extras.execute_batch
I am inserting a list of dictionaries to a PostgreSQL database. The list will be growing quickly and the number of dict values (columns) is around 30. The simplified data:
projects = [
{'name': 'project alpha', 'code': 12, 'active': True},
{'name': 'project beta', 'code': 25, 'active': True},
{'name': 'project charlie', 'code': 46, 'active': False}
]
Inserting the data into the PostgreSQL database with the following code does work (as in this answer), but I am worried about executing too many queries.
for project in projects:
columns = project.keys()
values = project.values()
query = """INSERT INTO projects (%s) VALUES %s;"""
# print(cursor.mogrify(query, (AsIs(','.join(project.keys())), tuple(project.values()))))
cursor.execute(query, (AsIs(','.join(columns)), tuple(values)))
conn.commit()
Is there a better practice? Thank you so much in advance for your help!
You can use bulk loading to make it faster.
https://www.citusdata.com/blog/2017/11/08/faster-bulk-loading-in-postgresql-with-copy/
Use execute_values() to insert many rows in a single query.
import psycopg2
from psycopg2.extras import execute_values
# ...
projects = [
{'name': 'project alpha', 'code': 12, 'active': True},
{'name': 'project beta', 'code': 25, 'active': True},
{'name': 'project charlie', 'code': 46, 'active': False}
]
columns = projects[0].keys()
query = "INSERT INTO projects ({}) VALUES %s".format(','.join(columns))
# convert projects values to list of lists
values = [[value for value in project.values()] for project in projects]
execute_values(cursor, query, values)
conn.commit()
Another performant option that does not require so much data munging for a list of dictionaries is execute_batch (new in psycopg2 version 2.7).
For example:
import psycopg2
from psycopg2.extras import execute_batch
values = [{'name': 'project alpha', 'code': 12, 'active': True}, ...]
query = "INSERT INTO projects VALUES (%(name)s, %(code)s, %(active)s)"
execute_batch(cursor, query, values)
conn.commit()
https://www.psycopg.org/docs/extras.html#psycopg2.extras.execute_batch