Unable to convert sql table into dataframe in Python

Question

I am reading data from Azure SQL database and convert into pandas dataframe. SQL connection is working, however, unable to convert into pandas dataframe. I am following this Azure documentation

https://learn.microsoft.com/en-us/sql/machine-learning/data-exploration/python-dataframe-pandas?view=sql-server-ver16

Below is my code

import pyodbc
import pandas as pd 


server = 'xxxx.database.windows.net'
database = 'xxx'
username = 'bxxxx'
password = 'bxxx'

conxn = 'Driver={ODBC Driver 18 for SQL Server};Server='+server+';Database='+database+';Encrypt=yes;UID='+username+';Pwd='+password+';TrustServerCertificate=no;Connection Timeout=30'

conn = pyodbc.connect(conxn)
cursor = conn.cursor()

query = "SELECT * FROM dbo.outsource;"

df = pd.read_sql(query, conxn)
print(df.head(10))

Getting below error

 u = _url.make_url(url)
PS C:Usersdhili> & C:/Users/dhili/AppData/Local/Programs/Python/Python311/python.exe c:/Users/dhili/Downloads/sql.py
Traceback (most recent call last):
  File "c:UsersdhiliDownloadssql.py", line 17, in <module>
    df = pd.read_sql(query, conxn)
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:UsersdhiliAppDataLocalProgramsPythonPython311Libsite-packagespandasiosql.py", line 561, in read_sql
    pandas_sql = pandasSQL_builder(con)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "C:UsersdhiliAppDataLocalProgramsPythonPython311Libsite-packagespandasiosql.py", line 754, in pandasSQL_builder
    con = sqlalchemy.create_engine(con)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in create_engine
  File "C:UsersdhiliAppDataLocalProgramsPythonPython311Libsite-packagessqlalchemyutildeprecations.py", line 375, in warned
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "C:UsersdhiliAppDataLocalProgramsPythonPython311Libsite-packagessqlalchemyenginecreate.py", line 518, in create_engine
    u = _url.make_url(url)
        ^^^^^^^^^^^^^^^^^^
  File "C:UsersdhiliAppDataLocalProgramsPythonPython311Libsite-packagessqlalchemyengineurl.py", line 738, in make_url
    return _parse_url(name_or_url)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:UsersdhiliAppDataLocalProgramsPythonPython311Libsite-packagessqlalchemyengineurl.py", line 799, in _parse_url
    raise exc.ArgumentError(
sqlalchemy.exc.ArgumentError: Could not parse SQLAlchemy URL from string 'Driver={ODBC Driver 18 for SQL Server};Server=xxxxx.database.windows.net;Database=xxxx;Encrypt=yes;UID=xxxxxx;Pwd=xxxx@xxx;TrustServerCertificate=no;Connection Timeout=30'     
PS C:Usersdhili>

Can anyone help?

Asked By: JasonW

||

Source

Answer 1

I am having a table in my database:

enter image description here

I tried to convert sql table to dataframe in python using :

import pyodbc
import pandas as pd 


server = 'xxxx.database.windows.net'
database = 'xxx'
username = 'bxxxx'
password = 'bxxx'

conxn = 'Driver={SQl server};Server='+server+';Database='+database+';Encrypt=yes;UID='+username+';Pwd='+password+';TrustServerCertificate=no;Connection Timeout=30'

conn = pyodbc.connect(conxn)
cursor = conn.cursor()

query = "SELECT * FROM dbo.outsource;"

df = pd.read_sql(query, conxn)
print(df.head(10))

I got below error:

enter image description here

I installed ODBC server using

%sh
curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
curl https://packages.microsoft.com/config/ubuntu/16.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
sudo apt-get update
sudo ACCEPT_EULA=Y apt-get -q -y install msodbcsql17

enter image description here

I get the installed drivers in pyodbc using:

drivers = [item for item in pyodbc.drivers()]
print(drivers)

enter image description here

Then I loop through it to make sure that it matches what I need

for driv in drivers:
print(driver)

if 'ODBC' in driv and 'SQL Server' in driv:
    driver = driv

enter image description here

I connected to sql database using :

import pandas as pd
db = pyodbc.connect(driver=driver, 
                        server='<servername>.database.windows.net', 
                        database='db',
                        user='<userName>', 
                        password= '<password>')

query = """SELECT * FROM products"""
table = pd.read_sql(query, db)
print(table)

Dataframe is created successfully without any error.
Output:

enter image description here

It worked for me. check once from your end.

Answered By: Bhavani

Unable to convert sql table into dataframe in Python

Question:

Answers: