Insert Python NumPy array into PostgreSQL database

Question:

How do I insert a large array of coordinates (x,y) into a postgresSQL table? I don’t want to use a for loop. It is a raster with 3601×3601 pixels.

import numpy as np
import psycopg2


# Data example:
east = np.linspace(-180.0,180.0,num=10)
north = np.linspace(-90.0,90.0,num=10)
coor = np.vstack([east, north])

conn = psycopg2.connect("dbname='postgres' user='dbuser' host='localhost' password='dbpass'")
cur = conn.cursor()
cur.execute("DROP TABLE IF EXISTS foobar;")
cur.execute("CREATE TABLE foobar (coordinate   point);")

# Working for an coordinate example:
cur.execute("INSERT INTO foobar VALUES (('12.56,56.43'));")

# Working for 1st coordinate in coor:
tmp = ','.join(str(e) for e in coor[:,0])
cur.execute('INSERT INTO foobar VALUES (point(' + tmp + '));')

# NOT WORKING!!!
# Insert all points in one go:
# cur.execute('INSERT INTO foobar VALUES (coor);')

conn.commit()
Asked By: TobiasDK

||
Source

Answers:

You can use cur.executemany or psycopg2.extras.execute_values to insert many records at once. Here is your sample adapted to use execute_values:

import psycopg2.extras
values = map(lambda a: ['{},{}'.format(a[0],a[1])], np.column_stack((east, north)))
psycopg2.extras.execute_values(cur, "INSERT INTO foobar (coordinate) VALUES %s", values)
Answered By: mehdix

With the function execute_values() you can insert multiple rows using a single SQL statement. You should prepare the data for the function in this format:

[['(-180.0, -90.0)'],
 ['(-140.0, -70.0)'],
 ['(-100.0, -50.0)'],
 ['(-60.0, -30.0)'],
 ['(-20.0, -10.0)'],
 ['(20.0, 10.0)'],
 ['(60.0, 30.0)'],
 ['(100.0, 50.0)'],
 ['(140.0, 70.0)'],
 ['(180.0, 90.0)']]

Code:

from psycopg2.extras import execute_values

# Data example:
east = np.linspace(-180.0,180.0,num=10)
north = np.linspace(-90.0,90.0,num=10)

# get array of pairs [east, north]
coor = np.dstack([east, north])

# convert to array of tuples (east, north) as strings
values = [[str(tuple(i))] for i in coor[0]]

execute_values(cur, 'INSERT INTO foobar VALUES %s', values)

conn.commit()

See also this answer.

Answered By: klin