How can I pass and than get the passed arguments in databricks job

Question

I’m trying to pass and get arguments in my databricks job it’s a spark_python_task type IT IS NOT A NOTEBOOK. I deployed my job with dbx from pycharm. I have deployment.json file where I configure deployment stuff.

Asked By: Borislav Blagoev

||

Source

Answer 1

If you follow documentation about deployment file, you can see that you can specify parameters as parameters array:

{
  "name": "this-parameter-is-required!",
  "spark_python_task": {
      "python_file": "path/to/entrypoint.py" 
  },
  "parameters": [
     "--conf-file",
     "conf/test/sample.json"
  ]
}

Parameters are passed as command-line parameters, so you can get them from the code just using the sys.argv, or built-in argparse library.

Answered By: Alex Ott

How can I pass and than get the passed arguments in databricks job

Question:

Answers: