How to add new DAGs to Airflow?


I have defined a DAG in a file called (actually a copy of the provided in the airflow tutorial, except with the dag_id changed to tutorial_2).

When I look inside my default, unmodified airflow.cfg (located in ~/airflow), I see that dags_folder is set to /home/alex/airflow/dags.

I do cd /home/alex/airflow; mkdir dags; cd dags; cp [...]/ Now I have a dags folder matching the path set in airflow.cfg , containing the file I created earlier.

However, when I run airflow list_dags, I only get the names corresponding with the default, tutorial DAGs.

I would like to have tutorial_2 show up in my DAG list, so that I can begin interacting with. Neither python nor airflow resetdb have caused it to appear in the list.

How do I remedy this?

Asked By: Aleksey Bilogur



I think the reason for this is because you haven’t exported AIRFLOW_HOME.
Try doing:
AIRFLOW_HOME="/home/alex/airflow/dags" airflow list_dags.
If that’s not working than do two steps

export AIRFLOW_HOME="/home/alex/airflow/dags"
airflow list_dags

I believe this should work. Give it a go?

Answered By: Ace Haidrey

In my understanding, AIRFLOW_HOME should link to the directory where airflow.cfg is stored. Then, airflow.cfg can apply and set the dag directory to the value you put in it.

The important point is : airflow.cfg is useless if your AIRFLOW_HOME is not set

Answered By: Pierre CORBEL

The issue is that you might have two airflow configs existing in your directories, so check for /root/airflow/dags and if yes you require to change the dags_folder path in both airflow.cfg s

Answered By: ahsen_mukhtar_maken

I might be using the latest airflow, the command has changed. What works for me is:

  • export AIRFLOW_HOME="~/airflow"
  • Then run airflow dags list
Answered By: Cherry Wu
Categories: questions Tags: , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.