pd.read_sql – Unsupported format character error (0x27)

Question

As above, I’m trying to use pd.read_sql to query our mysql database, and getting an error for double/single quotes.

When I remove the % operators from the LIKE clause (lines 84-87) the query runs, but these are needed. I know I need to format the strings but I don’t know how within such a big query.

Here’s the query:

SELECT
    s.offer_id,
    s.cap_id,
    vi.make,
    vi.model,
    vi.derivative,
    i.vehicle_orders,
    s.lowest_offer,
    CASE
        WHEN f.previous_avg = f.previous_low THEN "n/a"
        ELSE FORMAT(f.previous_avg, 2)
    END as previous_avg,
    f.previous_low,
    CASE
        WHEN ( ( (s.lowest_offer - f.previous_avg) / f.previous_avg) * 100) = ( ( (s.lowest_offer - f.previous_low) / f.previous_low) * 100) THEN "n/a"
        ELSE CONCAT(FORMAT( ( ( (s.lowest_offer - f.previous_avg) / f.previous_avg) * 100), 2), "%")
    END as diff_avg,
    CONCAT(FORMAT( ( ( (s.lowest_offer - f.previous_low) / f.previous_low) * 100), 2), "%") as diff_low,
    s.broker,
    CASE
        WHEN s.in_stock = '1' THEN "In Stock"
        ELSE "Factory Order"
    END as in_stock,
    CASE
        WHEN s.special IS NOT NULL THEN "Already in Specials"
        ELSE "n/a"
    END as special
FROM

    (   SELECT o.id as offer_id,
               o.cap_id as cap_id,
               MIN(o.monthly_payment) as lowest_offer,
               b.name as broker,
               o.stock as in_stock,
               so.id as special
            FROM
                offers o
                INNER JOIN brands b ON ( o.brand_id = b.id )
                LEFT JOIN special_offers so ON ( so.cap_id = o.cap_id )
            WHERE
                ( o.date_modified >= DATE_ADD(NOW(), INTERVAL -1 DAY) OR o.date_created >= DATE_ADD(NOW(), INTERVAL -1 DAY) )
                AND o.deposit_value = 9
                AND o.term = 48
                AND o.annual_mileage = 8000
                AND o.finance_type = 'P'
                AND o.monthly_payment > 100
            GROUP BY 
                o.cap_id
            ORDER BY
                special DESC) s
                
INNER JOIN      
        
    (   SELECT o.cap_id as cap_id,
               AVG(o.monthly_payment) as previous_avg,
               MIN(o.monthly_payment) as previous_low
            FROM
                offers o
            WHERE
                o.date_modified < DATE_ADD(NOW(), INTERVAL -1 DAY)
                AND o.date_modified >= DATE_ADD(NOW(), INTERVAL -1 WEEK)
                AND o.deposit_value = 9
                AND o.term = 48
                AND o.annual_mileage = 8000
                AND o.finance_type = 'P'
                AND o.monthly_payment > 100
            GROUP BY
                o.cap_id ) f ON ( s.cap_id = f.cap_id )
                
LEFT JOIN

    (   SELECT a.cap_id as cap_id,
               v.manufacturer as make,
               v.model as model,
               v.derivative as derivative,
               COUNT(*) as vehicle_orders
            FROM
                    (    SELECT o.id,
                                o.name as name,
                                o.email as email,
                                o.date_created as date,
                                SUBSTRING_INDEX(SUBSTRING(offer_serialized, LOCATE("capId", offer_serialized) +12, 10), '"', 1) as cap_id
                            FROM moneyshake.orders o
                            WHERE o.name NOT LIKE 'test%'
                                  AND o.email NOT LIKE 'jawor%'
                                  AND o.email NOT LIKE 'test%'
                                  AND o.email NOT LIKE '%moneyshake%'
                                  AND o.phone IS NOT NULL
                                  AND o.date_created > DATE_ADD(NOW(), INTERVAL -1 MONTH)
                    ) a JOIN moneyshake.vehicles_view v ON a.cap_id = v.id
            GROUP BY
                v.manufacturer,
                v.model,
                v.derivative,
                a.cap_id) i ON ( f.cap_id = i.cap_id )
                
INNER JOIN
            
    (   SELECT v.id as id,
               v.manufacturer as make,
               v.model as model,
               v.derivative as derivative
            FROM moneyshake.vehicles_view v
            GROUP BY v.id ) vi ON s.cap_id = vi.id

WHERE
    ( ( s.lowest_offer - f.previous_low ) / f.previous_low) * 100 <= -15
GROUP BY
    s.cap_id

Thanks!

Asked By: Alex Midmore

||

Source

Answer 1

That error occurs then the DBAPI layer (e.g., mysqlclient) natively uses the "format" paramstyle and the percent sign (%) is misinterpreted as a format character instead of a LIKE wildcard.

The fix is to wrap the SQL statement in a SQLAlchemy text() object. For example, this will fail:

import pandas as pd
import sqlalchemy as sa

engine = sa.create_engine("mysql+mysqldb://scott:tiger@localhost:3307/mydb")

sql = """
SELECT * FROM million_rows
WHERE varchar_col LIKE 'record00000%'
ORDER BY id
"""
df = pd.read_sql_query(sql, engine)

but simply changing the read_sql_query() call to

df = pd.read_sql_query(sa.text(sql), engine)

will work.

Answered By: Gord Thompson

Answer 2

I faced the same issue with Python 3.7
In my case the solution was simply replacing the % with %% in all the like clauses.

Answered By: Sameer Girolkar

pd.read_sql – Unsupported format character error (0x27)

Question:

Answers: