pandas dataframe.to_sql throws error when writing to MariaDB
Question:
When I try and execute the code below, python gives me a mariadb.ProgrammingError: Cursor is closed
even though the data was successfully written to the db. I guess there is something wrong with returning the rowcount (see the full traceback below). Is this a possible bug or am I missing something?
code sample
import pandas as pd
from sqlalchemy import create_engine
conn_str = 'mariadb+mariadbconnector://user:password@localhost:3306/database_name'
engine = create_engine(conn_str)
df = pd.DataFrame(data={'col1': [1, 2, 3], 'col2': ['A', 'B', 'C']})
df.to_sql('pandas_test_table', index=False, con=engine, if_exists='replace')
traceback
Traceback (most recent call last):
File "C:Program FilesPython310libcode.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandascoregeneric.py", line 2951, in to_sql
return sql.to_sql(
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 697, in to_sql
return pandas_sql.to_sql(
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 1739, in to_sql
total_inserted = sql_engine.insert_records(
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 1322, in insert_records
return table.insert(chunksize=chunksize, method=method)
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 950, in insert
num_inserted = exec_insert(conn, keys, chunk_iter)
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 858, in _execute_insert
return result.rowcount
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyutillanghelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginecursor.py", line 1691, in rowcount
self.cursor_strategy.handle_exception(self, self.cursor, e)
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginecursor.py", line 846, in handle_exception
raise err
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginecursor.py", line 1689, in rowcount
return self.context.rowcount
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginedefault.py", line 1439, in rowcount
return self.cursor.rowcount
File "C:UsersuserAppDataRoamingPythonPython310site-packagesmariadbcursors.py", line 541, in rowcount
self.check_closed()
File "C:UsersuserAppDataRoamingPythonPython310site-packagesmariadbcursors.py", line 55, in check_closed
raise mariadb.ProgrammingError("Cursor is closed")
mariadb.ProgrammingError: Cursor is closed
package version
mariadb==1.1.4
pandas==1.4.3
SQLAlchemy==1.4.40
Answers:
It’s a bug in MariaDB Connector/Python.
It’s already fixed in 1.1.5 (not released yet). If you don’t want to downgrade to 1.1.3 you have to build 1.1.5 from source.
I had attempted Georg Richter’s recommendation regarding installing MariaDB 1.1.5 from source or rolling back to 1.1.3 and was still receiving the error message.
Eventually, I came across this documentation and had changed the engine DBAPI from
uri = 'mariadb+mariadbconnector://Username:Password@localhost/test'
to
uri = 'mariadb+pymysql://Username:Password@localhost/test'
And it worked.
Hummm… I’m using mariadb version 1.1.5.post3, but I get the same error. Just the last lines of the data insertion code, which doesn’t raise any error in case I use the pymysql connector:
new_employee = {'id': '105', 'first_name': 'Joao',
'last_name': 'da Silva', 'description': 'Lorem Ipsum', 'active': '1'}
emp = {k: [v] for k, v in new_employee.items()}
df = pd.DataFrame(
emp, columns=['id', 'first_name', 'last_name', 'description', 'active'])
df.to_sql('employees', con=cnx, index=False, if_exists='append')
The error:
raise mariadb.ProgrammingError("Cursor is closed")
mariadb.ProgrammingError: Cursor is closed
Just in case, I’ve tried to downgrade to the version 1.1.5 mentioned above (using pip install
, not from the source). Other packages versions:
SQLAlchemy 1.4.46
pandas 1.5.2
P. S. Although we have this error, the data is inserted into the Maria DB table.
When I try and execute the code below, python gives me a mariadb.ProgrammingError: Cursor is closed
even though the data was successfully written to the db. I guess there is something wrong with returning the rowcount (see the full traceback below). Is this a possible bug or am I missing something?
code sample
import pandas as pd
from sqlalchemy import create_engine
conn_str = 'mariadb+mariadbconnector://user:password@localhost:3306/database_name'
engine = create_engine(conn_str)
df = pd.DataFrame(data={'col1': [1, 2, 3], 'col2': ['A', 'B', 'C']})
df.to_sql('pandas_test_table', index=False, con=engine, if_exists='replace')
traceback
Traceback (most recent call last):
File "C:Program FilesPython310libcode.py", line 90, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandascoregeneric.py", line 2951, in to_sql
return sql.to_sql(
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 697, in to_sql
return pandas_sql.to_sql(
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 1739, in to_sql
total_inserted = sql_engine.insert_records(
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 1322, in insert_records
return table.insert(chunksize=chunksize, method=method)
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 950, in insert
num_inserted = exec_insert(conn, keys, chunk_iter)
File "C:UsersuserAppDataRoamingPythonPython310site-packagespandasiosql.py", line 858, in _execute_insert
return result.rowcount
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyutillanghelpers.py", line 1113, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginecursor.py", line 1691, in rowcount
self.cursor_strategy.handle_exception(self, self.cursor, e)
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginecursor.py", line 846, in handle_exception
raise err
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginecursor.py", line 1689, in rowcount
return self.context.rowcount
File "C:UsersuserAppDataRoamingPythonPython310site-packagessqlalchemyenginedefault.py", line 1439, in rowcount
return self.cursor.rowcount
File "C:UsersuserAppDataRoamingPythonPython310site-packagesmariadbcursors.py", line 541, in rowcount
self.check_closed()
File "C:UsersuserAppDataRoamingPythonPython310site-packagesmariadbcursors.py", line 55, in check_closed
raise mariadb.ProgrammingError("Cursor is closed")
mariadb.ProgrammingError: Cursor is closed
package version
mariadb==1.1.4
pandas==1.4.3
SQLAlchemy==1.4.40
It’s a bug in MariaDB Connector/Python.
It’s already fixed in 1.1.5 (not released yet). If you don’t want to downgrade to 1.1.3 you have to build 1.1.5 from source.
I had attempted Georg Richter’s recommendation regarding installing MariaDB 1.1.5 from source or rolling back to 1.1.3 and was still receiving the error message.
Eventually, I came across this documentation and had changed the engine DBAPI from
uri = 'mariadb+mariadbconnector://Username:Password@localhost/test'
to
uri = 'mariadb+pymysql://Username:Password@localhost/test'
And it worked.
Hummm… I’m using mariadb version 1.1.5.post3, but I get the same error. Just the last lines of the data insertion code, which doesn’t raise any error in case I use the pymysql connector:
new_employee = {'id': '105', 'first_name': 'Joao',
'last_name': 'da Silva', 'description': 'Lorem Ipsum', 'active': '1'}
emp = {k: [v] for k, v in new_employee.items()}
df = pd.DataFrame(
emp, columns=['id', 'first_name', 'last_name', 'description', 'active'])
df.to_sql('employees', con=cnx, index=False, if_exists='append')
The error:
raise mariadb.ProgrammingError("Cursor is closed")
mariadb.ProgrammingError: Cursor is closed
Just in case, I’ve tried to downgrade to the version 1.1.5 mentioned above (using pip install
, not from the source). Other packages versions:
SQLAlchemy 1.4.46
pandas 1.5.2
P. S. Although we have this error, the data is inserted into the Maria DB table.