copy temp from stdin with skip id

Question

I have a huge file with some data and need to insert it into the crm database.

I have tested it with pandas.to_sql, but I also need to check for duplications and update data in duplications case, so I decided to use this:

SQL_STATEMENT = """
                            CREATE TEMP TABLE temp
                            (
                                LIKE metal
                            )
                            ON COMMIT DROP;

                            COPY temp FROM STDIN WITH
                                CSV
                                HEADER
                                DELIMITER AS ',';

                            INSERT INTO metal
                            SELECT *
                            FROM temp
                            ON CONFLICT (title) DO UPDATE SET main_category = EXCLUDED.main_category
                            """

My issue is, that crm db has autogenerated id’s, and I can’t it add to my file

So can I just write in the statement to skip the first column (the id’s column)?

Asked By: Nero

||

Source

Answer 1

I found the solution to my problem, all I need is to find the latest id and make it +1 in my csv file

max_id = pd.read_sql_table('metal', engine)['id'].max()

And then, using my statement above, if it is new data, it will add a new uniq id, if it is old, it will update the selected column (main_category) in my case

Answered By: Nero

copy temp from stdin with skip id

Question:

Answers: