Airflow execute python script through bash operator

Question

I have a python script test2.py to connect to a remote server and execute the command. as below. This works on the command line.

Passing parameters as JSON and getting the response in JSON this works when executed as below in the command line.

python3.6 test2.py  '{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'

I am trying to execute same through the airflow

from __future__ import print_function
from airflow.operators import BashOperator
from airflow.models import DAG
from datetime import datetime, timedelta


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime(2018, 9, 1),
    'email_on_failure': False,
    'email_on_retry': False,
    'schedule_interval': '@daily',
    'retries': 1,
    'retry_delay': timedelta(seconds=5),
}

dag = DAG(
    dag_id='DAG-3',
    default_args=default_args,
    dagrun_timeout=timedelta(minutes=10)
    )

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"


t = BashOperator(
     task_id = 'some_id',
     bash_command = cmd_command,
     dag = dag)

I am seeing below error related to syntax.?

cmd_command = "python3.6 /root/test2.py '{{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}}'"
                                                       ^
SyntaxError: invalid syntax

Can you please help

Thank you

Asked By: goe

||

Source

Answer 1

You use double quotes for JSON, but Python interprets them as start or end of a string. One way to resolve this is to escape double quotes inside JSON:

cmd_command = "python3.6 /root/test2.py '{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'"

Answered By: SergiyKolesnikov

Answer 2

I agree with Sergiy, you’ve got repeated " in your python line.

  cmd_args= r'{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}'
  cmd_command = f"python3.6 /root/test2.py '{cmd_args}'"

where

r'string' r is Python’s RAW string, to switch off processing of spec chars, like
f"string {python_var} " – new python3 explicit formatting string

another approach is to pass arguments by env variables:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command="$PYTHON /root/test2.py '$my_params'",
    env={"my_params": r'{"hostname": "<server>", "username":"<test>", "password":"<test1>", "command1":"hostname"}', 
         "PYTHON": 'python3.6'},
)

Answered By: Fedor

Airflow execute python script through bash operator

Question:

Answers: