"inconsistent datatypes: expected DATE got CLOB" with .to_sql() to Oracle

Question

I’m attempting to automate a data insert into an Oracle database using data from an excel file. I put my data into a dataframe and have been able to get the statement to look right but I’m getting errors or when I don’t I see there’s no updates after running my code. Below is basically what I’ve written so far and I hope someone can help me figure this out.

import pandas as pd
import numpy as np
import cx_Oracle
import config
from sqlalchemy import types, create_engine, engine_from_config

df = pd.read_excel("Census_columns_renamed.xlsx")
df = df.drop(['Unnamed: 0', 'index'], axis=1)
df_copy = df.copy(deep=True)

ENGINE_PATH_WIN_AUTH = DIALECT + '+' + SQL_DRIVER + '://' + USERNAME + ':' + PASSWORD +'@' + HOST + ':' + PORT + '/?service_name=' + SERVICE



engine = create_engine(ENGINE_PATH_WIN_AUTH)
test_df = pd.read_sql_query('SELECT * FROM ODSMGR.REGISTERED_CENSUS', engine)



intersection_col = df_copy.columns.intersection(test_df.columns.str.upper())
",".join(intersection_col.tolist())


conn = None
try:
    conn = cx_Oracle.connect(
        config.username,
        config.password,
        config.dsn,
        encoding=config.encoding)

  
    print("Database version:", conn.version)
    print("Client version:", cx_Oracle.clientversion())
except cx_Oracle.Error as error:
    print(error)
finally:
    cursor = conn.cursor()

select_for_table_names = "SELECT * FROM ODSMGR.REGISTERED_CENSUS"
cursor.execute(select_for_table_names)
data = cursor.fetchall()
col_names = []
for i in range(0, len(cursor.description)):
    col_names.append(cursor.description[i][0])
    
    



intersection_col = df_copy.columns.intersection(test_df.columns.str.upper())

table_names = ",".join(f"{x}"
                      for x in intersection_col.to_list())


bind_names = ",".join(":"+f"{val}" 
                      for val in intersection_col.tolist())





df1 = df[df_copy.columns.intersection(test_df.columns.str.upper()).values.tolist()]



sql_bind = "INSERT INTO ODSMGR.REGISTERED_STUDENT_FRZ_T_ULV ""(" + table_names +")"" VALUES (" + bind_names + ")"

######  everything seems to work up until this point  ######
## not sure what else I might need to do here

cursor.executemany(sql_bind, df1)

conn.commit()
conn.close()

this is the error I’m getting:

TypeError: parameters should be a list of sequences/dictionaries or an integer specifying the number of times to execute the statement

Basically it’s looking like my data is fine; the intersection gives me the columns that are 1-to-1 for the insert statement but I’m not sure how to approach this. I also tried to use


dtyp = {c:types.VARCHAR(df1[c].str.len().max())
        for c in df1.columns[df1.dtypes == 'object'].tolist()}
## this dtyp wasn't useful

engine.dialect.max_identifier_length = 100000
df1.to_sql('REGISTERED_CENSUS',schema = 'ODSMGR',con=engine, if_exists='append')

but that to_sql isn’t working for me either– I have been getting datatype errors "expected DATE got CLOB’.

I’m open to ANY solution.

I was also playing with the above code as well to try to loop through the df using an executemany() but also I’m considering how to change the datatypes that are needed for insertion.

Asked By: Benjamin Diaz

||

Source

Answer 1

Your Excel sheet has one or more columns with date values stored as strings, and Oracle doesn’t like them. For example, with an Excel sheet where the values in column B are all strings (i.e., the "date" values are not true dates):

	A	B
1	id	a_date
2	1	2022-12-12

.read_excel() with the default parse_dates=None imports them as string and .to_sql() fails

df = pd.read_excel(r"UsersGordDesktopBook1.xlsx")
try:
    df.to_sql("so74775731", engine, if_exists="append", index=False)
    print("Success.")  # this does not get printed
except sa.exc.DatabaseError as de:
    print(de)
    """console output
    (cx_Oracle.DatabaseError) ORA-00932: inconsistent datatypes: expected DATE got CLOB
    [SQL: INSERT INTO so74775731 (id, a_date) VALUES (:id, :a_date)]
    [parameters: {'id': 1, 'a_date': '2022-12-12'}]
    (Background on this error at: https://sqlalche.me/e/14/4xp6)
    """

However, if we use .read_excel(… , parse_dates=[1]) to parse that column as dates then it works

df = pd.read_excel(r"UsersGordDesktopBook1.xlsx", parse_dates=[1])
try:
    df.to_sql("so74775731", engine, if_exists="append", index=False)
    print("Success.")  # this prints
except sa.exc.DatabaseError as de:
    print(de)  # no error

Answered By: Gord Thompson

"inconsistent datatypes: expected DATE got CLOB" with .to_sql() to Oracle

Question:

Answers: