immediately throw sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')

Question:

I encountered a very strange issue where I consistently receive the following exception at a fixed point in my test data when using the db.session.execute(‘insert into table_a select * from table_b where xxx’) statement within a loop.

sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')

I believe this is unrelated to connection counts or timeout settings. Due to the large join operation, executing the loop 150 times takes about 5 seconds, and the exception occurs exactly on the 150th iteration every time. This is very fast in Java and has never been a problem with same logic and same MySQL. Furthermore, the total amount of data is very small, for about only 5000 rows, less than 300 KB.

In python, I perform a db.session.commit() at the end of each iteration. But with no help.

The connection loss exception is thrown immediately upon reaching the 150th iteration, and I’m not sure what’s causing it.

Here is the test code:

        normal_date_pattern = '%Y-%m-%d'
        day_start_hour_format_pattern = '%Y-%m-%d %H:00:00'
        day_end_hour_format_pattern = '%Y-%m-%d %H:59:59'

        while actual_end_time > start_time:
            start_current_hour_str = start_time.strftime(self.day_start_hour_format_pattern)
            end_current_hour_str = start_time.strftime(self.day_end_hour_format_pattern)
            # log when meet 0'oclock
            if start_current_hour_str.endswith('00:00:00'):
                current_app.logger.info(f'processing date {start_time.strftime(self.normal_date_pattern)}')
            # insert into db
            db.session.execute(
                text(
                    """
                        INSERT INTO table_1
                        SELECT * FROM table_2
                        WHERE table_2.start_time>=:start_time
                        AND table_2.end_time<=:end_time
                        ON DUPLICATE KEY UPDATE
                            snapshot_date = snapshot_date,
                            snapshot_date_loc = VALUES(snapshot_date_loc)
                    """
                ),
                params={
                    'start_time': start_current_hour_str,
                    'end_time': end_current_hour_str
                }
            )
            db.session.commit()

            # add one hour to process next hour
            start_time = start_time + timedelta(hours=1)

This code is running in a flask application, but even I extract it to an individual script and using individual engine and sessionmaker, it crashes as always with the same error.

Asked By: william

||

Answers:

It is due to the version of MySQL server, nothing to do with the code.The MySQL version is too old to support my code. Here is the reason and solution, just update my MySQL version, then there is nothing wrong.see this issue

Answered By: william