How do I retrieve airflow url link to my latest task log in a DAG?

Question

In individual DAG task, how do I set up the url link with the help from python operator as I am intending to send an url link of the latest log directly to the user whenever errors occur so that they can access to the page directly skipping the step of navigating.

Asked By: Shlim

||

Source

Answer 1

You can define a callback function, which you can then pass to DAG() operator as default arugments.

def on_failure_callback_slack(context):
    message = f"Failue! airflow task: {context['task_instance'].task_id} failed" 
f"dag: {base_url}?dag_id={context['dag'].dag_id} " 
f"{str(context['task_instance'])}"

    operator = PythonOperator(task_id="failure", python_callable=post_to_slack, op_kwargs={'message': message}
    
    return operator.execute(context=context)

In the above code, post_to_slack() is just a utility function to post to slack with requests.post(...)

You can pass this function to DAG, and it will post with the url to slack (or other medium of your choice). Note you will have to provide base_url for the url to work properly.

default_args = {"on_failure_callback": on_failure_callback_slack}
dag=DAG(dag_id='some_id', default_args=default_args)

For more you can read here: https://airflow.apache.org/docs/apache-airflow/2.2.1/logging-monitoring/callbacks.html

Answered By: monte

Answer 2

Task instances have a log_url attribute. In the callback function, it can be accessed like so:

def on_failure_callback(context):
    dag_run = context.get("dag_run")
    log_url = dag_run.get_task_instance({your_task_id}).log_url

    # Do whatever with the log_url

Depending on the Airflow setup, note that log_url might point to the local address (e.g. http://localhost:8080), which potentially should be replaced with whichever URL is being exposed to users.

Answered By: swimmer

How do I retrieve airflow url link to my latest task log in a DAG?

Question:

Answers: