How can the default list of email addresses be rendered from a variable in Airflow? Question: I have a lot of Airflow DAGs and I’d like to automatically send email notifications to the same list of recipients (stored in an Airflow variable) on any task failure, so I’m using the following default operator configuration defined …
Airflow task stuck in 'queued' state. Dependencies Blocking Task From Getting Scheduled Question: I have the following Airflow DAG which uploads a single local file into an S3 bucket. # airflow related from airflow import DAG from airflow.operators.python import PythonOperator # other packages from datetime import datetime import boto3 with DAG( dag_id=’upload_to_s3′, start_date=datetime(2020, 5, 5), …
Airflow DAG can't find local file to upload on s3 Question: I have created a DAG to upload a local file into a personal S3 Bucket. However, when accessing http://localhost:9099/home I get the following error: FileNotFoundError: [Errno 2] No such file or directory: ‘C:UsersplataOneDriveΥπολογιστήςprojects backupsairflow-sqlserverdagspricedata.xlsx’ Ariflow error – broken dag I have a Windows PC …
Import modules from Airflow Question: Need help importing modules from dags. I have a dag, next to which in the same folder is my python script. For some reason airflow does not see this file and it shows import error. Below you can see that status_dict.by is next to my dag. I partially solved the …
No module named 'airflow.sensors.python_sensor' Question: I am trying to use PythonSensor in my dag but I am unable to import it. from airflow.sensors.python_sensor import PythonSensor wait_for_stg_completion = PythonSensor( task_id=’wait_for_stg_completion’, python_callable=fetch_stg_qa_status ) How can I import it? What else can I try? Asked By: x89 || Source Answers: You have incorrect import. Use from airflow.sensors.python import …
Load JSON data into postgres table using airflow Question: I have an Airflow DAG that runs a spark file (reads two parquet files, performs transformations on them, and loads the data into a single JSON file). Now the data from this JSON file needs to be pushed into a Postgres table. At first, I was …
How Airflow DockerOperator send notification when Docker image creation error Question: I want to send notification when Airflow DockerOperator can’t create Docker image. Notification can be sent in case of DockerOperator execution errors using on_failure_callback. To be more specific, I want to catch 2 errors. private Docker repository is not running(10.11.12.13 is not running in …
How to implement CSV Writer in a named temporary – Python? Question: Basically, I need final a csv file, whose contents look like Header, and then the data: (Boto3 upload_file does the job of writing temporary file, into csv) Expectation: Name,Code Adam,12 I am able to get this, by using: with tempfile.NamedTemporaryFile(mode="a+t", suffix =".csv", delete=True) …
conditionally_trigger for TriggerDagRunOperator Question: I have 2 DAGs: dag_a and dag_b (dag_a -> dag_b) After dag_a is executed, TriggerDagRunOperator is called, which starts dag_b. The problem is, when dag_b is off (paused), dag_a’s TriggerDagRunOperator creates scheduled runs in dag_b that queue up for as long as dag_a is running. After turning dag_b back ON, the …
Airflow DAG: How to insert data into a table using Python operator, not BigQuery operator? Question: I am trying to insert some data into a table using a simple Python operator, not the BigQuery operator, but I am unsure how to implement this. I am trying to implement this in the form of an Airflow …
