Connect CloudSQL Postgres Db using Python

Question

So I am DevOps engineer please excuse for my lack of knowledge in python development. I am trying to connect a CloudSQL Postgres 14 Db using Python. So that I can insert data and read data from my application. Can someone help me understand why I am getting error when I try to run the python script.

Reference used: Stackoverflow reference used for connecting CloudSQL with python

# Copyright 2021 Google LLC.
# SPDX-License-Identifier: Apache-2.0

import os
from google.cloud.sql.connector import connector

# Connect to the database
conn = connector.connect(
    os.getenv("angelic-hexagon-307814:us-central1:sqlpg1234"),
    "pg8000",
    user=os.getenv("test1"),
    password=os.getenv("test1"),
    db=os.getenv("postgres")
)

# Execute a query
cursor = conn.cursor()
cursor.execute("SELECT * from accounts")

# Fetch the results
result = cursor.fetchall()

# Do something with the results
for row in result:
    print(row)

I have this file with filename sql.py and I run the command python3 sql.py to run the python script. But I am getting below error but not able to debug the reason behind this. Can someone help?

sidharth@Sidharths-Air python % python3 sql.py
Traceback (most recent call last):
  File "/Users/sidharth/python/sql.py", line 8, in <module>
    conn = connector.connect(
AttributeError: module 'google.cloud.sql.connector.connector' has no attribute 'connect'. Did you mean: 'Connector'?

After trying out the solution mentioned by @pensive

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/core.py", line 199, in _make_socket
    sock = socket.create_connection((host, port), timeout, source_address)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socket.py", line 845, in create_connection
    raise err
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/socket.py", line 833, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3292, in raw_connection
    return self.pool.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1269, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get
    with util.safe_reraise():
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
    return self._create_connection()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
    with util.safe_reraise():
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 616, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/__init__.py", line 111, in connect
    return Connection(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/legacy.py", line 443, in __init__
    super().__init__(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/core.py", line 312, in __init__
    self.channel_binding, self._usock = _make_socket(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/core.py", line 201, in _make_socket
    raise InterfaceError(
pg8000.exceptions.InterfaceError: Can't create a connection to host localhost and port 5432 (timeout is None and source_address is None).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sidharth/python/sql.py", line 15, in <module>
    with db.connect() as conn:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3268, in connect
    return self._connection_cls(self)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 147, in __init__
    Connection._handle_dbapi_exception_noconnection(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2430, in _handle_dbapi_exception_noconnection
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3292, in raw_connection
    return self.pool.connect()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1269, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get
    with util.safe_reraise():
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
    return self._create_connection()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
    with util.safe_reraise():
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/create.py", line 637, in connect
    return dialect.connect(*cargs, **cparams)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 616, in connect
    return self.loaded_dbapi.connect(*cargs, **cparams)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/__init__.py", line 111, in connect
    return Connection(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/legacy.py", line 443, in __init__
    super().__init__(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/core.py", line 312, in __init__
    self.channel_binding, self._usock = _make_socket(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pg8000/core.py", line 201, in _make_socket
    raise InterfaceError(
sqlalchemy.exc.InterfaceError: (pg8000.exceptions.InterfaceError) Can't create a connection to host localhost and port 5432 (timeout is None and source_address is None).
(Background on this error at: https://sqlalche.me/e/20/rvf5)

After executing what Jack has mentioned getting this error

(venv) bash-3.2$ python main.py 
['project:region:instance-name']: An error occurred while performing refresh. Scheduling another refresh attempt immediately
Traceback (most recent call last):
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/instance.py", line 389, in _refresh_task
    refresh_data = await refresh_task
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/instance.py", line 313, in _perform_refresh
    metadata = await metadata_task
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/refresh_utils.py", line 103, in _get_metadata
    resp = await client_session.get(url, headers=headers, raise_for_status=True)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/aiohttp/client.py", line 669, in _request
    resp.raise_for_status()
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1011, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://sqladmin.googleapis.com/sql/v1beta4/projects/project/instances/instance-name/connectSettings')
Traceback (most recent call last):
  File "/Users/sidharth/PycharmProjects/pythonapp/main.py", line 27, in <module>
    with pool.connect() as db_conn:
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3268, in connect
    return self._connection_cls(self)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 145, in __init__
    self._dbapi_connection = engine.raw_connection()
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 3292, in raw_connection
    return self.pool.connect()
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 452, in connect
    return _ConnectionFairy._checkout(self)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 1269, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 716, in checkout
    rec = pool._do_get()
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 169, in _do_get
    with util.safe_reraise():
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/impl.py", line 167, in _do_get
    return self._create_connection()
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 393, in _create_connection
    return _ConnectionRecord(self)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 678, in __init__
    self.__connect()
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 902, in __connect
    with util.safe_reraise():
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 898, in __connect
    self.dbapi_connection = connection = pool._invoke_creator(self)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/sqlalchemy/pool/base.py", line 365, in <lambda>
    return lambda rec: creator_fn()
  File "/Users/sidharth/PycharmProjects/pythonapp/main.py", line 10, in getconn
    conn = connector.connect(
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/connector.py", line 163, in connect
    return connect_task.result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/connector.py", line 244, in connect_async
    instance_data, ip_address = await instance.connect_info(ip_type)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/instance.py", line 446, in connect_info
    instance_data = await self._current
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/instance.py", line 389, in _refresh_task
    refresh_data = await refresh_task
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/instance.py", line 313, in _perform_refresh
    metadata = await metadata_task
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/google/cloud/sql/connector/refresh_utils.py", line 103, in _get_metadata
    resp = await client_session.get(url, headers=headers, raise_for_status=True)
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/aiohttp/client.py", line 669, in _request
    resp.raise_for_status()
  File "/Users/sidharth/PycharmProjects/pythonapp/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1011, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message='Bad Request', url=URL('https://sqladmin.googleapis.com/sql/v1beta4/projects/project/instances/instance-name/connectSettings')

The link said something like this

{
  "error": {
    "code": 401,
    "message": "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
    "errors": [
      {
        "message": "Login Required.",
        "domain": "global",
        "reason": "required",
        "location": "Authorization",
        "locationType": "header"
      }
    ],
    "status": "UNAUTHENTICATED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ErrorInfo",
        "reason": "CREDENTIALS_MISSING",
        "domain": "googleapis.com",
        "metadata": {
          "service": "sqladmin.googleapis.com",
          "method": "google.cloud.sql.v1beta4.SqlConnectService.GetConnectSettings"
        }
      }
    ]
  }
}

Asked By: sidharth vijayakumar

||

Source

Answer 1

To connect to a Cloud SQL database, you can use the pg8000 library along with the sqlalchemy library

to install pg8000

pip install sqlalchemy pg8000

here is sample for connection

import os
from sqlalchemy import create_engine

db_user = os.getenv("test")
db_pass = os.getenv("test")
db_name = os.getenv("postgres")
cloud_sql_connection_name = os.getenv("angelic-hexagon-307814:us-central1:test-db")

connection_name = f"/cloudsql/{cloud_sql_connection_name}"

db = create_engine(
    f"postgresql+pg8000://{db_user}:{db_pass}@{connection_name}/{db_name}"
)

with db.connect() as conn:
    result = conn.execute("SELECT * from accounts")
    for row in result:
        print(row)

Answered By: pensive

Answer 2

@sidharth vijayajumar The post and code sample you are referencing is over 2 years old and using v0.1.0 of the Cloud SQL Python Connector. As of writing this the Cloud SQL Python Connector is now on v1.4.2. There have beens changes to the API since v0.1.0 (major version upgrade.)

I would recommend taking look at this interactive Python notebook to step you through the process of establishing a connection to a Postgres Cloud SQL instance with up-to-date code samples.
The official Github README is also always up-to-date with usage examples.

Her is a quick code sample showcasing the current way to connect to a Postgres instance using the Cloud SQL Python Connector:

import sqlalchemy
from google.cloud.sql.connector import Connector, IPTypes


# initialize Python Connector object
connector = Connector()

# Python Connector database connection function
def getconn():
    conn = connector.connect(
        "project:region:instance-name", # Cloud SQL Instance Connection Name
        "pg8000",
        user="my-user",
        password="my-password",
        db="my-database",
        ip_type= IPTypes.PUBLIC  # IPTypes.PRIVATE for private IP
    )
    return conn

# create connection pool with 'creator' argument to our connection object function
pool = sqlalchemy.create_engine(
    "postgresql+pg8000://",
    creator=getconn,
)

# interact with Cloud SQL database using connection pool
with pool.connect() as db_conn:
    # insert statement
    insert_stmt = sqlalchemy.text(
        "INSERT INTO my_table (id, title) VALUES (:id, :title)",
    )
    # insert into database
    db_conn.execute(insert_stmt, parameters={"id": "book1", "title": "Book One"})

    # commit transaction (SQLAlchemy v2.X.X is commit as you go)
    db_conn.commit()

    # query database
    result = db_conn.execute(sqlalchemy.text("SELECT * from my_table")).fetchall()

    # Do something with the results
    for row in result:
       print(row)

Answered By: Jack Wotherspoon

Connect CloudSQL Postgres Db using Python

Question:

Answers: