How to add new DAGs to Airflow?
Question:
I have defined a DAG in a file called tutorial_2.py
(actually a copy of the tutorial.py
provided in the airflow
tutorial, except with the dag_id
changed to tutorial_2
).
When I look inside my default, unmodified airflow.cfg
(located in ~/airflow
), I see that dags_folder
is set to /home/alex/airflow/dags
.
I do cd /home/alex/airflow; mkdir dags; cd dags; cp [...]/tutorial_2.py tutorial_2.py
. Now I have a dags
folder matching the path set in airflow.cfg
, containing the tutorial_2.py
file I created earlier.
However, when I run airflow list_dags
, I only get the names corresponding with the default, tutorial DAGs.
I would like to have tutorial_2
show up in my DAG list, so that I can begin interacting with. Neither python tutorial_2.py
nor airflow resetdb
have caused it to appear in the list.
How do I remedy this?
Answers:
I think the reason for this is because you haven’t exported AIRFLOW_HOME
.
Try doing:
AIRFLOW_HOME="/home/alex/airflow/dags" airflow list_dags
.
If that’s not working than do two steps
export AIRFLOW_HOME="/home/alex/airflow/dags"
airflow list_dags
I believe this should work. Give it a go?
In my understanding, AIRFLOW_HOME
should link to the directory where airflow.cfg
is stored. Then, airflow.cfg
can apply and set the dag directory to the value you put in it.
The important point is : airflow.cfg
is useless if your AIRFLOW_HOME
is not set
The issue is that you might have two airflow configs existing in your directories, so check for /root/airflow/dags and if yes you require to change the dags_folder path in both airflow.cfg s
I might be using the latest airflow, the command has changed. What works for me is:
export AIRFLOW_HOME="~/airflow"
- Then run
airflow dags list
I have defined a DAG in a file called tutorial_2.py
(actually a copy of the tutorial.py
provided in the airflow
tutorial, except with the dag_id
changed to tutorial_2
).
When I look inside my default, unmodified airflow.cfg
(located in ~/airflow
), I see that dags_folder
is set to /home/alex/airflow/dags
.
I do cd /home/alex/airflow; mkdir dags; cd dags; cp [...]/tutorial_2.py tutorial_2.py
. Now I have a dags
folder matching the path set in airflow.cfg
, containing the tutorial_2.py
file I created earlier.
However, when I run airflow list_dags
, I only get the names corresponding with the default, tutorial DAGs.
I would like to have tutorial_2
show up in my DAG list, so that I can begin interacting with. Neither python tutorial_2.py
nor airflow resetdb
have caused it to appear in the list.
How do I remedy this?
I think the reason for this is because you haven’t exported AIRFLOW_HOME
.
Try doing:
AIRFLOW_HOME="/home/alex/airflow/dags" airflow list_dags
.
If that’s not working than do two steps
export AIRFLOW_HOME="/home/alex/airflow/dags"
airflow list_dags
I believe this should work. Give it a go?
In my understanding, AIRFLOW_HOME
should link to the directory where airflow.cfg
is stored. Then, airflow.cfg
can apply and set the dag directory to the value you put in it.
The important point is : airflow.cfg
is useless if your AIRFLOW_HOME
is not set
The issue is that you might have two airflow configs existing in your directories, so check for /root/airflow/dags and if yes you require to change the dags_folder path in both airflow.cfg s
I might be using the latest airflow, the command has changed. What works for me is:
export AIRFLOW_HOME="~/airflow"
- Then run
airflow dags list