Restrict/Exclude specific logs from Airflow to Datadog Question: We are observing Airflow is sending large amount of logs to Datadog and we want to restrict/Reduce these logs by excluding logs from below tasks: pod_manager.py base.py base_aws.py logging_mixin.py Do we have any configuration settings where I can define this requirement? We have Airflow-2.0 running on Kubernetes. …
How to have multiple input and output connection for an Airflow DAG task use a global variable pandas data frame with in @task.external_python? Question: GOAL I use the Docker 2.4.1 version of Airflow I use my external python virtual environment for each task I have a normal python integer that I want to pass on …
How to set the state of a past TaskInstance to success and continue the pipeline inside a airflow DAG? Question: I’m trying to make a DAG with params that can be triggered with a dag_id/task_id. The goal of this DAG is to set the state of the last executed task to "success" and to continue …
How to use a python list as global variable with in @task.external_python? Question: GOAL: Have a python list as a global variable between tasks. Currently it crashes at the 1st task. 1.) I am trying to have a simple python list that is carried from 1 task to the next and append a few string …
How to set the two types of authentication for airflow: AUTH_LDAP and AUTH_DB? If airflow can't find an user in AUTH_LDAP it has to search in AUTH_DB Question: I would like to set up two types of authentification in the Airflow webserver. The first preferred type is LDAP. If airflow doesn’t find a user in …
How to remove a downstream or upstream task dependency in Airflow Question: Assuming we have the two following Airflow tasks in a DAG, from airflow.operators.dummy import DummyOperator t1 = DummyOperator(task_id=’dummy_1′) t2 = DummyOperator(task_id=’dummy_2′) we can specify dependencies as: # Option A t1 >> t2 # Option B t2.set_upstream(t1) # Option C t1.set_downstream(t2) My question is …
How to install packages(Pandas) in Airflow? Question: Airflow is installed on Linux(Debian), just following the official tutorial in the most clumsy way – no docker, etc. (official tutorial: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html) I created a DAG and it has a Python Operator that uses the Pandas package. But I am getting an error: Broken DAG: [/home/airflow/airflow/dags/air_etl.py] Traceback (most …
Airflow: BigQuery SQL Insert empty data to the table Question: Using Airflow, I am trying to get the data from one table to insert it into another in BigQuery. I have 5 origin tables and 5 destination tables. My SQL query and python logic work for the 4 tables where it successfully gets the data …
Multiple inheritance using `BaseBranchOperator` in Airflow Question: Can one use multiple inheritance using BaseBranchOperator in Airflow? I want to define an operator like: from airflow.models import BaseOperator from airflow.operators.branch import BaseBranchOperator class MyOperator(BaseOperator, BaseBranchOperator): def execute(self, context): print(‘hi’) def choose_branch(self, context): if True: return ‘task_A’ else: return ‘task_B’ In that case, is it accurate to …
Broken DAG issue (Airflow 2.5.0) Question: Broken DAG: [/opt/airflow/dags/dag.py] Traceback (most recent call last): File "/opt/airflow/dags/dag.py", line 7, in <module> from training import training File "/opt/airflow/dags/training.py", line 6, in <module> from joblib import dump ModuleNotFoundError: No module named ‘joblib’ I have ‘joblib’ module installed already then why it is showing this module not found error?? …
