How to use Xcom within python script file executed by BashOperator

Question

I am executing a python script file using a Bash Operator as following:

op = BashOperator(task_id='python',
                  bash_command='python3 py_script.py')

In the python file, I want to send a value to xcom if a condition is met in a loop, something like:

for i in range(10):
    if i == 2:
        print(i)
        xcom_push('hello')

Since it is in a loop and I want it to continue after i reach 2, I can’t use a return. I have tried with:

from airflow.models import XCom

context['ti'].xcom_push('hello')
kwargs['ti'].xcom_push('hello')

But in both cases I receive an error like NameError: name 'kwargs' is not defined

What would be the solution? Thanks

Asked By: Javier Lopez Tomas

||

Source

Answer 1

If you set do_xcom_push=True in BashOperator then the last line written to stdout by the bash command will be pushed to an XCom (docs).

So, if after print(i) in your py_script.py no other lines are printed then the string representation of i will be pushed to an XCom for the pythonk task.

Note 1: In Airflow 1.10, the argument name is xcom_push and not do_xcom_push.

Note 2: In Airflow 2.x, do_xcom_push=True is the default setting.

Answered By: SergiyKolesnikov

How to use Xcom within python script file executed by BashOperator

Question:

Answers: