How do I retrieve airflow url link to my latest task log in a DAG?


enter image description here

In individual DAG task, how do I set up the url link with the help from python operator as I am intending to send an url link of the latest log directly to the user whenever errors occur so that they can access to the page directly skipping the step of navigating.

Asked By: Shlim



You can define a callback function, which you can then pass to DAG() operator as default arugments.

def on_failure_callback_slack(context):
    message = f"Failue! airflow task: {context['task_instance'].task_id} failed" 
f"dag: {base_url}?dag_id={context['dag'].dag_id} " 

    operator = PythonOperator(task_id="failure", python_callable=post_to_slack, op_kwargs={'message': message}
    return operator.execute(context=context)

In the above code, post_to_slack() is just a utility function to post to slack with

You can pass this function to DAG, and it will post with the url to slack (or other medium of your choice). Note you will have to provide base_url for the url to work properly.

default_args = {"on_failure_callback": on_failure_callback_slack}
dag=DAG(dag_id='some_id', default_args=default_args)

For more you can read here:

Answered By: monte

Task instances have a log_url attribute. In the callback function, it can be accessed like so:

def on_failure_callback(context):
    dag_run = context.get("dag_run")
    log_url = dag_run.get_task_instance({your_task_id}).log_url

    # Do whatever with the log_url

Depending on the Airflow setup, note that log_url might point to the local address (e.g. http://localhost:8080), which potentially should be replaced with whichever URL is being exposed to users.

Answered By: swimmer
Categories: questions Tags: ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.