airflow

How to install packages(Pandas) in Airflow? Question: Airflow is installed on Linux(Debian), just following the official tutorial in the most clumsy way – no docker, etc. (official tutorial: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) I created a DAG and it has a Python Operator that uses the Pandas package. But I am getting an error: Broken DAG: [/home/airflow/airflow/dags/air_etl.py] Traceback (most …

Airflow: BigQuery SQL Insert empty data to the table Question: Using Airflow, I am trying to get the data from one table to insert it into another in BigQuery. I have 5 origin tables and 5 destination tables. My SQL query and python logic work for the 4 tables where it successfully gets the data …

Multiple inheritance using `BaseBranchOperator` in Airflow Question: Can one use multiple inheritance using BaseBranchOperator in Airflow? I want to define an operator like: from airflow.models import BaseOperator from airflow.operators.branch import BaseBranchOperator class MyOperator(BaseOperator, BaseBranchOperator): def execute(self, context): print(‘hi’) def choose_branch(self, context): if True: return ‘task_A’ else: return ‘task_B’ In that case, is it accurate to …

Broken DAG issue (Airflow 2.5.0) Question: Broken DAG: [/opt/airflow/dags/dag.py] Traceback (most recent call last): File "/opt/airflow/dags/dag.py", line 7, in <module> from training import training File "/opt/airflow/dags/training.py", line 6, in <module> from joblib import dump ModuleNotFoundError: No module named ‘joblib’ I have ‘joblib’ module installed already then why it is showing this module not found error?? …

How can the default list of email addresses be rendered from a variable in Airflow? Question: I have a lot of Airflow DAGs and I’d like to automatically send email notifications to the same list of recipients (stored in an Airflow variable) on any task failure, so I’m using the following default operator configuration defined …

Airflow DAG can't find local file to upload on s3 Question: I have created a DAG to upload a local file into a personal S3 Bucket. However, when accessing http://localhost:9099/home I get the following error: FileNotFoundError: [Errno 2] No such file or directory: ‘C:UsersplataOneDriveΥπολογιστήςprojects backupsairflow-sqlserverdagspricedata.xlsx’ Ariflow error – broken dag I have a Windows PC …

Import modules from Airflow Question: Need help importing modules from dags. I have a dag, next to which in the same folder is my python script. For some reason airflow does not see this file and it shows import error. Below you can see that status_dict.by is next to my dag. I partially solved the …

No module named 'airflow.sensors.python_sensor' Question: I am trying to use PythonSensor in my dag but I am unable to import it. from airflow.sensors.python_sensor import PythonSensor wait_for_stg_completion = PythonSensor( task_id=’wait_for_stg_completion’, python_callable=fetch_stg_qa_status ) How can I import it? What else can I try? Asked By: x89 || Source Answers: You have incorrect import. Use from airflow.sensors.python import …

Load JSON data into postgres table using airflow Question: I have an Airflow DAG that runs a spark file (reads two parquet files, performs transformations on them, and loads the data into a single JSON file). Now the data from this JSON file needs to be pushed into a Postgres table. At first, I was …

How Airflow DockerOperator send notification when Docker image creation error Question: I want to send notification when Airflow DockerOperator can’t create Docker image. Notification can be sent in case of DockerOperator execution errors using on_failure_callback. To be more specific, I want to catch 2 errors. private Docker repository is not running(10.11.12.13 is not running in …