Is It possible to trigger a Dag from an on_failure_callback? Question: I would like to trigger a Dag when a task is failed. I want to use the "on_failure_callback", however, I have not found information about it. Do you know if it is possible to trigger the dag from the "on_failure_callback"? In the past, I …
Pass returned value from a previous Python operator task to another in airflow Question: I am a new user to Apache Airflow. I am building a DAG like the following to schedule tasks: def add(): return 1 + 1 def multiply(a): return a * 999 dag_args = { ‘owner’: ‘me’, ‘depends_on_past’: False, ‘start_date’: datetime(2023, 2, …
How to have multiple input and output connection for an Airflow DAG task use a global variable pandas data frame with in @task.external_python? Question: GOAL I use the Docker 2.4.1 version of Airflow I use my external python virtual environment for each task I have a normal python integer that I want to pass on …
How to use a python list as global variable with in @task.external_python? Question: GOAL: Have a python list as a global variable between tasks. Currently it crashes at the 1st task. 1.) I am trying to have a simple python list that is carried from 1 task to the next and append a few string …
Broken DAG issue (Airflow 2.5.0) Question: Broken DAG: [/opt/airflow/dags/dag.py] Traceback (most recent call last): File "/opt/airflow/dags/dag.py", line 7, in <module> from training import training File "/opt/airflow/dags/training.py", line 6, in <module> from joblib import dump ModuleNotFoundError: No module named ‘joblib’ I have ‘joblib’ module installed already then why it is showing this module not found error?? …
Get weight combining edges in Networkx Question: Hello I am using Networkx with Python to make some graphs. After extracting the edges from a .CSV structured as "source" and "target" I got a list of edges like this: edges = [(a,b), (a,c), (b,c), (a,b), (a,b), (a,c)] Is there any way to build ( after creating …
Airflow task stuck in 'queued' state. Dependencies Blocking Task From Getting Scheduled Question: I have the following Airflow DAG which uploads a single local file into an S3 bucket. # airflow related from airflow import DAG from airflow.operators.python import PythonOperator # other packages from datetime import datetime import boto3 with DAG( dag_id=’upload_to_s3′, start_date=datetime(2020, 5, 5), …
Airflow DAG can't find local file to upload on s3 Question: I have created a DAG to upload a local file into a personal S3 Bucket. However, when accessing http://localhost:9099/home I get the following error: FileNotFoundError: [Errno 2] No such file or directory: ‘C:UsersplataOneDriveΥπολογιστήςprojects backupsairflow-sqlserverdagspricedata.xlsx’ Ariflow error – broken dag I have a Windows PC …
Airflow DAG: How to insert data into a table using Python operator, not BigQuery operator? Question: I am trying to insert some data into a table using a simple Python operator, not the BigQuery operator, but I am unsure how to implement this. I am trying to implement this in the form of an Airflow …
How to load a BigQuery table from a file in GCS Bucket using Airflow? Question: I am new to Airflow, and I am wondering, how do I load a file from a GCS Bucket to BigQuery? So far, I have managed to do BigQuery to GCS Bucket: bq_recent_questions_query = bigquery_operator.BigQueryOperator( task_id=’bq_recent_questions_query’, sql=""" SELECT owner_display_name, title, …
