AIRFLOW: use .replace() or relativedelta() in jinja template for {{ds}}

Question

My goal is to return 1st day of previous month based on airflow macro variable {{ds}} and use it e.g. in HiveOperator.

E.g. For ds = 2020-05-09 I expect to return: 2020-04-01

The solution I found and tried were:

SET hivevar_LAST_MONTH='{{ (ds.replace(day=1) - macros.timedelta(days=1)).replace(day=1) }}';
SET hivevar_LAST_MONTH='{{ ds + macros.dateutil.relativedelta.relativedelta(months=-1, day=1) }}'

But both resulted in errors:

Error rendering template: replace() takes no keyword arguments 

Error rendering template: must be str, not relativedelta

and when rendered didn’t show any dates.

What am I doing wrong?

Asked By: Mateusz Konopelski

||

Source

Answer 1

You can use:

{{ (execution_date + macros.dateutil.relativedelta.relativedelta(months=-1, day=1)).strftime("%Y-%m-%d") }}

Example:

from airflow.operators.bash_operator import BashOperator
default_args = {
    'owner': 'airflow',
    'start_date': datetime(2020, 4, 1),

}
with DAG(dag_id='stackoverflow',
         default_args=default_args,
         schedule_interval=None,
         catchup=False
         ) as dag:
    run_this = BashOperator(
        task_id='example',
        bash_command='echo ds is {{ ds }} modified ds is {{ (execution_date + macros.dateutil.relativedelta.relativedelta(months=-1, day=1)).strftime("%Y-%m-%d") }}',
    )

Answered By: Elad Kalif

Answer 2

According to airflow doc {{ execution_date }} is deprecated. So, you can keep using {{ ds }}, but you should convert it into datetime. Of course {{ ds }} depends on you cron time.

If you cron time is monthly, you could use only replace() function:

"{{ macros.datetime.strptime(ds, '%Y-%m-%d').replace(day=1) }}"

If you cron time is daily, use replace() and then relativedelta():

"{{ (macros.datetime.strptime(ds, '%Y-%m-%d').replace(day=1) - macros.dateutil.relativedelta.relativedelta(months=1)).strftime('%Y-%m-%d') }}

Answered By: moneretin

AIRFLOW: use .replace() or relativedelta() in jinja template for {{ds}}

Question:

Answers: