Merge tables from two different databases – sqlite3/Python

Question

I have two different SQLite databases XXX and YYY.
XXX contains table A and YYY contains B respectively.
A and B have same structure(columns).
How to append the rows of B in A in Python – SQLite API.
After appending A contains rows of A and rows of B.

Asked By: Vinod

||

Source

Answer 1

You first get a connection to the database using sqlite3.connect, then create a cursor so you can execute sql. Once you have a cursor, you can execute arbitrary sql commands.

Example:

import sqlite3

# Get connections to the databases
db_a = sqlite3.connect('database_a.db')
db_b = sqlite3.connect('database_b.db')

# Get the contents of a table
b_cursor = db_b.cursor()
b_cursor.execute('SELECT * FROM mytable')
output = b_cursor.fetchall()   # Returns the results as a list.

# Insert those contents into another table.
a_cursor = db_a.cursor()
for row in output:
    a_cursor.execute('INSERT INTO myothertable VALUES (?, ?, ...etc..., ?, ?)', row)

# Cleanup
db_a.commit()
a_cursor.close()
b_cursor.close()

Caveat: I haven’t actually tested this, so it might have a few bugs in it, but the basic idea is sound, I think.

Answered By: Michael0x2a

Answer 2

This is a generalized function and should be customized to your particular environment. To do this, you may structure the "dynamically determine SQL expression requirements" section with the static SQL parameters (rather than PRAGMA table_info). This should improve performance.

import sqlite3

def merge_tables(cursor_new: sqlite3.Cursor, cursor_old: sqlite3.Cursor, table_name: str, del_old_table: bool = False) -> None:
    '''
    This function merges the content of a specific table from an old cursor into a new cursor. 
    
    :param cursor_new: [sqlite3.Cursor] the primary cursor
    :param cursor_old: [sqlite3.Cursor] the secondary cursor
    :param table_name: [str] the name of the table
    :return: None
    '''

    # dynamically determine SQL expression requirements
    column_names = cursor_new.execute(f"PRAGMA table_info({table_name})").fetchall()
    column_names = tuple([x[1] for x in column_names][1:])  # remove the primary keyword
    values_placeholders = ', '.join(['?' for x in column_names])  # format appropriately
    
    # SQL select columns from table
    data = cursor_old.execute(f"SELECT {', '.join(column_names)} FROM {table_name}").fetchall()

    # insert the data into the primary cursor
    cursor_new.executemany(f"INSERT INTO {table_name} {column_names} VALUES ({values_placeholders})", data)
    if (cursor_new.connection.commit() == None):
        # With Ephemeral RAM connections & testing, deleting the table may be ill-advised
        if del_old_table:
            cursor_old.execute(f"DELETE FROM {table_name}")  # cursor_old.execute(f'DROP TABLE {table_name}')
            cursor_old.connection.commit()
        print(f"Table {table_name} merged from {cursor_old.connection} to {cursor_new.connection}") # Consider logging.info()
    return None

Answered By: David Moruzzi

Merge tables from two different databases – sqlite3/Python

Question:

Answers: