How to do df.to_sql using SQL Server in Azure

Question

I can do a df.to_slq on my local instance of SQL Server just fine. I am getting stuck when trying to do the same df.to_sll using Python and Azure SQL Server. I thought it would essentially be done like this.

import urllib.parse
params = urllib.parse.quote_plus(
    'Driver=%s;' % '{ODBC Driver 17 for SQL Server}' +
    'Server=%s,1433;' % 'ryan-server.database.windows.net' +
    'Database=%s;' % 'ryan_sql_db' +
    'Uid=%s;' % 'UN'  +
    'Pwd={%s};' % 'PW' +
    'Encrypt=no;' +
    'TrustServerCertificate=no;'
    )

from sqlalchemy.engine import create_engine
conn_str = 'mssql+pyodbc:///?odbc_connect=' + params
engine = create_engine(conn_str)
        
connection = engine.connect()
connection

all_data.to_sql('health', engine, if_exists='append', chunksize=100000, method=None,index=False)

That is giving me this error.

OperationalError: (pyodbc.OperationalError) ('08S01', '[08S01] [Microsoft][ODBC Driver 17 for SQL Server]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.rn (10060) (SQLExecDirectW); [08S01] [Microsoft][ODBC Driver 17 for SQL Server]Communication link failure (10060)')
[SQL: INSERT INTO health ([0], [Facility_BU_ID], [Code_Type], [Code], [Description], [UB_Revenue_Code], [UB_Revenue_Description], [Gross_Charge], [Cash_Charge], [Min_Negotiated_Rate], [Max_Negotiated_Rate], etc., etc., etc.

I found this link today:

https://learn.microsoft.com/en-us/sql/machine-learning/data-exploration/python-dataframe-sql-server?view=sql-server-ver15

I tried to do something similar, like this.

import pyodbc
import pandas as pd
df = all_data
# server = 'myserver,port' # to specify an alternate port
server = 'ryan-server.database.windows.net'
database = 'ryan_sql_db' 
username = 'UN' 
password = 'PW' 
cnxn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)
cursor = cnxn.cursor()
# Insert Dataframe into SQL Server:
for index, row in df.iterrows():
     cursor.execute(all_data.to_sql('health', cnxn, if_exists='append', chunksize=100000, method=None,index=False))
cnxn.commit()
cursor.close()

When I run that, I get this error.

DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ('42S02', "[42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'sqlite_master'. (208) (SQLExecDirectW); [42S02] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")

What I’m really hoping to to is df.to_sql, not Insert Into. I am working in Spyder and trying to send the data from my local machine to the cloud.

Asked By: ASH

||

Source

Answer 1

I read the two links below, and got it working.

Basically, you need to open your command window on your local machine, enter ‘ipconfig’, and grab two IP addresses. Then, enter those into SQL Server in Azure.

EXECUTE sp_set_database_firewall_rule 
N'health', 
'192.0.1.1', 
'192.0.0.5';

Finally, run the small script below, in SQL Server, to confirm that the changes were made correctly.

USE [ryan_sql_db]
GO
SELECT * FROM sys.database_firewall_rules
ORDER BY modify_date DESC

Answered By: ASH

How to do df.to_sql using SQL Server in Azure

Question:

Answers: