Adding name of the tables with use of logging

Question

I would like to use logging in my code in order to run my code and also get names of the tables for which I was not able to perform operation. I’m getting this error message:

Data is not available for table DIM_LOGS. The requested is beyond the object creation time.

My example:

import pandas as pd
import logging 

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

sql = "SELECT * FROM TABLE"
cur.execute(sql)
df = pd.DataFrame.from_records(iter(cur), columns=[x[0] for x in cur.description])
my_dict = dict()

for i in df['col1'].unique().tolist():

    df_x = df[df['col1'] == i]

    my_dict[i] = df_x['col_table'].tolist()

    sql_list = []

    for k, v in my_dict.items():

        for v in v:

            sql_list.append([f"INSERT INTO {k}.{v} SELECT * FROM {k}.{v} where col2 = 1;"])

    sql = 'EXECUTE IMMEDIATE %s'

    cur.executemany(sql, sql_list)

    conn.close()

I would like to add exceptions in order to run this code and also add name of the tables to some log file or something for which I had errors. In my for loop k is schema name and v is table name.

Asked By: My80

||

Source

Answer 1

You can replace your executemany by a execute in the loop. Then, you will know exactly which table failed, and you can handle this accordingly.

Note that in your code for v in v: is a terrible idea, since the value will override the list v

sql = "SELECT * FROM TABLE"
cur.execute(sql)
df = pd.DataFrame.from_records(iter(cur), columns=[x[0] for x in cur.description])
my_dict = dict()

for i in df['col1'].unique().tolist():

    df_x = df[df['col1'] == i]
    my_dict[i] = df_x['col_table'].tolist()

    for schema, tables in my_dict.items():
        for table in tables:
            query = f"INSERT INTO {schema}.{table} SELECT * FROM {schema}.{table} where col2 = 1;"
            try:
                cur.execute(query)
            except snowflake.connector.errors.ProgrammingError as e:
                # Something went wrong with the insert
                logging.error(f"Inserting in {schema}.{table}: {e}")

conn.close()

Answered By: Florent Monin

Adding name of the tables with use of logging

Question:

Answers: