Elasticsearch Search Query using python subprocess: Illegal Argument Exception

Question

I am trying to run a curl command using the run command in the subprocess module in python. But it is failing.
However, the curl command for my elasticsearch search query API is working fine-

curl -k XGET -H "Content-Type: application/json" -H "Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==" 'https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search?index=.fleet-enrollment-api-keys-7&size=20&pretty' -d '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}},"fields" : ["_id", "_index"]}'

This is the code that I am trying to run-

import json
from builtins import int
from pydantic import BaseModel, Field
from typing import List, Dict
from subprocess import PIPE, run

def elasticsearch_search_query(host: str, 
                               port: int, 
                               api_key: str, 
                               path: str, 
                               query: dict,
                               index: str,
                               size: int,
                               fields: List):

    es_path = host + ":" + str(port) + path +"?index="+index+"&"+str(size)+"&pretty"
    es_header = "Authorization: ApiKey" + " " + api_key
    es_dict = {"query": query, "fields": fields}
    es_json = json.dumps(es_dict)
    es_replaced = " "+"'"+ es_json.replace("'", """)+"'"

    result = run(["curl", "-k", "-XGET", "-H", "Content-Type: application/json", "-H",
                  es_header,
                  es_path,
                  "-d",
                  es_replaced]
                 # ,stdout=PIPE, stderr=PIPE, universal_newlines=True, shell=False
                )
    print(es_path, es_header, es_replaced)
    print(result.stdout)


elasticsearch_search_query(host="https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search", api_key="Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==",
                           port=9200, size=5, query={"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, fields=["_id", "_index"], path="/_search",index=".fleet-enrollment-api-keys-7")

Here is the error-

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "request [/_search:9200/_search] contains unrecognized parameter: [5]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "request [/_search:9200/_search] contains unrecognized parameter: [5]"
  },
  "status" : 400
}
https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/_search:9200/_search?index=.fleet-enrollment-api-keys-7&5&pretty Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw== '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'
None

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   350  100   350    0     0  10294      0 --:--:-- --:--:-- --:--:-- 10294
curl: (3) nested brace in URL position 12:
'{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'

UPDATE

Output of repr(result):

CompletedProcess(args=['curl', '-k', '-XGET', '-H', 'Content-Type: application/json', '-H', 'Authorization: ApiKey Q2txZ2FvSUJ6Nlcwa3pjbnh0NUM6enBidjdNLTdRQVNhc0tuTEk5UEZmZw==', 'https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200:9200/_search?index=.fleet-enrollment-api-keys-7&size=5&pretty', '-d', '{"query": {"match": {"_id": "c8a537a9-7f98-489b-8f8b-de2450c67ce6"}}, "fields": ["_id", "_index"]}'], returncode=3)

Asked By: Shloka Bhalgat

||

Source

Answer 1

Since the error was

contains unrecognized parameter: [5]

The problem is not in the query but in the way you construct es_path. Instead of

&"+str(size)+

it should be

&size="+str(size)+

UPDATE:

Also from what I see your URL now looks like this

https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200:9200/_search?index=.fleet-enrollment-api-keys-7&size=5&pretty

and it’s wrong for the following reasons:

the port is specified twice
the index is not at the right location

First, host should just be this (i.e. without port and path)

host="https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com"

Then you can build the URL like this instead:

es_path = host + ":" + str(port) + "/" + index + path +"?size="+str(size)+"&pretty"

i.e. it should look like this instead:

https://internal-a02c1b85dade940ef871b0b6d1e65191-1270244841.us-west-2.elb.amazonaws.com:9200/.fleet-enrollment-api-keys-7/_search?size=5&pretty

Answered By: Val

Elasticsearch Search Query using python subprocess: Illegal Argument Exception

Question:

Answers: