How do I force a task on airflow to fail?
Question:
I have a python callable process_csv_entries that processes csv file entries. I want my task to complete successfully only if all entries were processed successfully. Task should fail otherwise
def process_csv_entries(csv_file):
# Boolean
file_completely_parsed = <call_to_module_to_parse_csv>
return not file_completely_parsed
CSV_FILE=<Sets path to csv file>
t1 = PythonOperator(dag=dag,
task_id='parse_csv_completely',
python_operator=process_csv_entries,
op_args=[CSV_FILE])
t1 seems to complete successfully irrespective of returned value.
How do I force PythonOperator task to fail?
Answers:
raise exception when you meet the error condition ( in your case: when file is not sucesfully parsed)
raise ValueError('File not parsed completely/correctly')
raise relevant error type with suitable message
Yes, raise AirflowException, this will cause the task to move immediately to failure state.
from airflow import AirflowException
ValueError can be used for fail and retry.
AirflowFailException is now available in Airflow 1.10.11 to make the task fail without retries
if you want to fail the task without retries use AirflowFailException :-
Example :-
from airflow.exceptions import AirflowFailException
def task_to_fail():
raise AirflowFailException("Our api key is bad!")
If you are looking for retries use AirflowException :-
Example:-
from airflow import AirflowException
def task_to_fail():
raise AirflowException("Error msj")
I have a python callable process_csv_entries that processes csv file entries. I want my task to complete successfully only if all entries were processed successfully. Task should fail otherwise
def process_csv_entries(csv_file):
# Boolean
file_completely_parsed = <call_to_module_to_parse_csv>
return not file_completely_parsed
CSV_FILE=<Sets path to csv file>
t1 = PythonOperator(dag=dag,
task_id='parse_csv_completely',
python_operator=process_csv_entries,
op_args=[CSV_FILE])
t1 seems to complete successfully irrespective of returned value.
How do I force PythonOperator task to fail?
raise exception when you meet the error condition ( in your case: when file is not sucesfully parsed)
raise ValueError('File not parsed completely/correctly')
raise relevant error type with suitable message
Yes, raise AirflowException, this will cause the task to move immediately to failure state.
from airflow import AirflowException
ValueError can be used for fail and retry.
AirflowFailException is now available in Airflow 1.10.11 to make the task fail without retries
if you want to fail the task without retries use AirflowFailException :-
Example :-
from airflow.exceptions import AirflowFailException
def task_to_fail():
raise AirflowFailException("Our api key is bad!")
If you are looking for retries use AirflowException :-
Example:-
from airflow import AirflowException
def task_to_fail():
raise AirflowException("Error msj")