Trigger Cloud Composer Using Google Cloud Function
Question:
I have ran this exact code below but get an error in when attempting to trigger the dag using a cloud function. The error and code are described below:
gcs-dag-trigger-function
8bhxprce8hze
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 171, in view_func
function(data, context)
File "/workspace/main.py", line 52, in trigger_dag
make_iap_request(
File "/workspace/main.py", line 91, in make_iap_request
raise Exception(
Exception: Bad response from application: 404 / {'Date': 'Mon, 18 Jul 2022 15:03:44 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Vary': 'Accept-Encoding', 'Server': 'gunicorn', 'X-Robots-Tag': 'noindex, nofollow', 'Set-Cookie': 'session=616e9fda-1cd1-4a96-b9e2-57a3ea0f78bb.tblTOCMLoOZPdPTHgbbMepCbRbI; Expires=Wed, 17-Aug-2022 15:03:44 GMT; HttpOnly; Path=/; SameSite=Lax', 'Content-Encoding': 'gzip', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'} / 'nn<!DOCTYPE html>n<html lang="en">n <head>n <title>Airflow 404</title>n <link rel="icon" type="image/png" href="/static/pin_32.png">n </head>n <body>n <div style="font-family: verdana; text-align: center; margin-top: 200px;">n <img src="/static/pin_100.png" width="50px" alt="pin-logo" />n <h1>Airflow 404</h1>n <p>Page cannot be found.</p>n <a href="/">Return to the main page</a>n <p>fbffada7b897</p>n </div>n </body>n</html>'
from google.auth.transport.requests import Request
from google.oauth2 import id_token
import requests
IAM_SCOPE = 'https://www.googleapis.com/auth/iam'
OAUTH_TOKEN_URI = 'https://www.googleapis.com/oauth2/v4/token'
# If you are using the stable API, set this value to False
# For more info about Airflow APIs see https://cloud.google.com/composer/docs/access-airflow-api
USE_EXPERIMENTAL_API = True
def trigger_dag(data, context=None):
"""Makes a POST request to the Composer DAG Trigger API
When called via Google Cloud Functions (GCF),
data and context are Background function parameters.
For more info, refer to
https://cloud.google.com/functions/docs/writing/background#functions_background_parameters-python
To call this function from a Python script, omit the ``context`` argument
and pass in a non-null value for the ``data`` argument.
This function is currently only compatible with Composer v1 environments.
"""
# Fill in with your Composer info here
# Navigate to your webserver's login page and get this from the URL
# Or use the script found at
# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/composer/rest/get_client_id.py
client_id = 'xxxxxxxxx-gtld8n5rbu8ncs3l80fvnb2903pdq8p2.apps.googleusercontent.com'
# This should be part of your webserver's URL:
# {tenant-project-id}.appspot.com
webserver_id = 'xxxxxxxxxxxxxxxx-tp'
# The name of the DAG you wish to trigger
dag_name = 'GcsToBigQueryTriggered'
if USE_EXPERIMENTAL_API:
endpoint = f'api/experimental/dags/{dag_name}/dag_runs'
json_data = {'conf': data, 'replace_microseconds': 'false'}
else:
endpoint = f'api/v1/dags/{dag_name}/dagRuns'
json_data = {'conf': data}
webserver_url = (
'https://'
+ webserver_id
+ '.appspot.com/'
+ endpoint
)
# Make a POST request to IAP which then Triggers the DAG
make_iap_request(
webserver_url, client_id, method='POST', json=json_data)
# This code is copied from
# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/iap/make_iap_request.py
# START COPIED IAP CODE
def make_iap_request(url, client_id, method='GET', **kwargs):
"""Makes a request to an application protected by Identity-Aware Proxy.
Args:
url: The Identity-Aware Proxy-protected URL to fetch.
client_id: The client ID used by Identity-Aware Proxy.
method: The request method to use
('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
Returns:
The page body, or raises an exception if the page couldn't be retrieved.
"""
# Set the default timeout, if missing
if 'timeout' not in kwargs:
kwargs['timeout'] = 90
# Obtain an OpenID Connect (OIDC) token from metadata server or using service
# account.
google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
method, url,
headers={'Authorization': 'Bearer {}'.format(
google_open_id_connect_token)}, **kwargs)
if resp.status_code == 403:
raise Exception('Service account does not have permission to '
'access the IAP-protected application.')
elif resp.status_code != 200:
raise Exception(
'Bad response from application: {!r} / {!r} / {!r}'.format(
resp.status_code, resp.headers, resp.text))
else:
return resp.text
# END COPIED IAP CODE
Any assistance would be greatly appreciated. I have removed sensitive personal information and replaced them with xxx’s. I have attempted several times to trigger the cloud composer dag to no avail. I am not sure why the error keeps appearing.
Answers:
In your code, you have USE_EXPERIMENTAL_API as True, which means your function tries to call the experimental REST API of Airflow (api/experimental/dags/{dag_name}/dag_runs).
Since you are running Airflow 2, you will need to override the following Airflow configuration fields to accept requests at the experimental API as follows:
auth_backend = default.api.auth.backend.default
enable_experimental_api = True
The cURL command you showed did not work since the Identity Token requires the client ID as the --audiences parameter. I tested the following request and I could see the DAG being queued (provided the configuration is overridden for experimental API as stated):
curl -X "POST"
"https://<WEBSERVER_ID>.appspot.com/api/experimental/dags/<DAG_NAME>/dag_runs"
-H "accept: application/json"
-H "Content-Type: application/json"
-H "Authorization: Bearer $(gcloud auth print-identity-token --audiences=<CLIENT_ID_STRING>.apps.googleusercontent.com)"
-d '{"conf": {}, "replace_microseconds": "false"}'
If you aren’t using the experimental API, you’d need to change the USE_EXPERIMENTAL_API variable in your function to False. Depending on the length of your runtime service account email, you must manually add it as an Airflow user either through the console, or with the gcloud command shown (the service account must also have the Composer User role assigned).
A cURL request would instead target the stable REST endpoint:
curl -X "POST"
"https://<WEBSERVER_ID>.appspot.com/api/v1/dags/<DAG_NAME>/dagRuns"
-H "accept: application/json"
-H "Content-Type: application/json"
-H "Authorization: Bearer $(gcloud auth print-identity-token --audiences=<CLIENT_ID_STRING>.apps.googleusercontent.com)"
-d '{"conf": {}}'
Let me know if this information was useful!
I have ran this exact code below but get an error in when attempting to trigger the dag using a cloud function. The error and code are described below:
gcs-dag-trigger-function
8bhxprce8hze
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 2073, in wsgi_app
response = self.full_dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1518, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py", line 171, in view_func
function(data, context)
File "/workspace/main.py", line 52, in trigger_dag
make_iap_request(
File "/workspace/main.py", line 91, in make_iap_request
raise Exception(
Exception: Bad response from application: 404 / {'Date': 'Mon, 18 Jul 2022 15:03:44 GMT', 'Content-Type': 'text/html; charset=utf-8', 'Vary': 'Accept-Encoding', 'Server': 'gunicorn', 'X-Robots-Tag': 'noindex, nofollow', 'Set-Cookie': 'session=616e9fda-1cd1-4a96-b9e2-57a3ea0f78bb.tblTOCMLoOZPdPTHgbbMepCbRbI; Expires=Wed, 17-Aug-2022 15:03:44 GMT; HttpOnly; Path=/; SameSite=Lax', 'Content-Encoding': 'gzip', 'Via': '1.1 google', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'} / 'nn<!DOCTYPE html>n<html lang="en">n <head>n <title>Airflow 404</title>n <link rel="icon" type="image/png" href="/static/pin_32.png">n </head>n <body>n <div style="font-family: verdana; text-align: center; margin-top: 200px;">n <img src="/static/pin_100.png" width="50px" alt="pin-logo" />n <h1>Airflow 404</h1>n <p>Page cannot be found.</p>n <a href="/">Return to the main page</a>n <p>fbffada7b897</p>n </div>n </body>n</html>'
from google.auth.transport.requests import Request
from google.oauth2 import id_token
import requests
IAM_SCOPE = 'https://www.googleapis.com/auth/iam'
OAUTH_TOKEN_URI = 'https://www.googleapis.com/oauth2/v4/token'
# If you are using the stable API, set this value to False
# For more info about Airflow APIs see https://cloud.google.com/composer/docs/access-airflow-api
USE_EXPERIMENTAL_API = True
def trigger_dag(data, context=None):
"""Makes a POST request to the Composer DAG Trigger API
When called via Google Cloud Functions (GCF),
data and context are Background function parameters.
For more info, refer to
https://cloud.google.com/functions/docs/writing/background#functions_background_parameters-python
To call this function from a Python script, omit the ``context`` argument
and pass in a non-null value for the ``data`` argument.
This function is currently only compatible with Composer v1 environments.
"""
# Fill in with your Composer info here
# Navigate to your webserver's login page and get this from the URL
# Or use the script found at
# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/composer/rest/get_client_id.py
client_id = 'xxxxxxxxx-gtld8n5rbu8ncs3l80fvnb2903pdq8p2.apps.googleusercontent.com'
# This should be part of your webserver's URL:
# {tenant-project-id}.appspot.com
webserver_id = 'xxxxxxxxxxxxxxxx-tp'
# The name of the DAG you wish to trigger
dag_name = 'GcsToBigQueryTriggered'
if USE_EXPERIMENTAL_API:
endpoint = f'api/experimental/dags/{dag_name}/dag_runs'
json_data = {'conf': data, 'replace_microseconds': 'false'}
else:
endpoint = f'api/v1/dags/{dag_name}/dagRuns'
json_data = {'conf': data}
webserver_url = (
'https://'
+ webserver_id
+ '.appspot.com/'
+ endpoint
)
# Make a POST request to IAP which then Triggers the DAG
make_iap_request(
webserver_url, client_id, method='POST', json=json_data)
# This code is copied from
# https://github.com/GoogleCloudPlatform/python-docs-samples/blob/main/iap/make_iap_request.py
# START COPIED IAP CODE
def make_iap_request(url, client_id, method='GET', **kwargs):
"""Makes a request to an application protected by Identity-Aware Proxy.
Args:
url: The Identity-Aware Proxy-protected URL to fetch.
client_id: The client ID used by Identity-Aware Proxy.
method: The request method to use
('GET', 'OPTIONS', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE')
**kwargs: Any of the parameters defined for the request function:
https://github.com/requests/requests/blob/master/requests/api.py
If no timeout is provided, it is set to 90 by default.
Returns:
The page body, or raises an exception if the page couldn't be retrieved.
"""
# Set the default timeout, if missing
if 'timeout' not in kwargs:
kwargs['timeout'] = 90
# Obtain an OpenID Connect (OIDC) token from metadata server or using service
# account.
google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
method, url,
headers={'Authorization': 'Bearer {}'.format(
google_open_id_connect_token)}, **kwargs)
if resp.status_code == 403:
raise Exception('Service account does not have permission to '
'access the IAP-protected application.')
elif resp.status_code != 200:
raise Exception(
'Bad response from application: {!r} / {!r} / {!r}'.format(
resp.status_code, resp.headers, resp.text))
else:
return resp.text
# END COPIED IAP CODE
Any assistance would be greatly appreciated. I have removed sensitive personal information and replaced them with xxx’s. I have attempted several times to trigger the cloud composer dag to no avail. I am not sure why the error keeps appearing.
In your code, you have USE_EXPERIMENTAL_API as True, which means your function tries to call the experimental REST API of Airflow (api/experimental/dags/{dag_name}/dag_runs).
Since you are running Airflow 2, you will need to override the following Airflow configuration fields to accept requests at the experimental API as follows:
auth_backend = default.api.auth.backend.default
enable_experimental_api = True
The cURL command you showed did not work since the Identity Token requires the client ID as the --audiences parameter. I tested the following request and I could see the DAG being queued (provided the configuration is overridden for experimental API as stated):
curl -X "POST"
"https://<WEBSERVER_ID>.appspot.com/api/experimental/dags/<DAG_NAME>/dag_runs"
-H "accept: application/json"
-H "Content-Type: application/json"
-H "Authorization: Bearer $(gcloud auth print-identity-token --audiences=<CLIENT_ID_STRING>.apps.googleusercontent.com)"
-d '{"conf": {}, "replace_microseconds": "false"}'
If you aren’t using the experimental API, you’d need to change the USE_EXPERIMENTAL_API variable in your function to False. Depending on the length of your runtime service account email, you must manually add it as an Airflow user either through the console, or with the gcloud command shown (the service account must also have the Composer User role assigned).
A cURL request would instead target the stable REST endpoint:
curl -X "POST"
"https://<WEBSERVER_ID>.appspot.com/api/v1/dags/<DAG_NAME>/dagRuns"
-H "accept: application/json"
-H "Content-Type: application/json"
-H "Authorization: Bearer $(gcloud auth print-identity-token --audiences=<CLIENT_ID_STRING>.apps.googleusercontent.com)"
-d '{"conf": {}}'
Let me know if this information was useful!