Ansible – how to use selectattr with yaml of different keys

Question

im pulling my hairs trying to do a simple thing (i thought it should be easy) with parsing a yaml and filtering for some key in Ansible.

My yaml file looks like this:

---

- vm: "vm1"
  ip: 10.10.10.1
- vm: "vm2"
  ip: 10.10.10.2
- test_vm: something
- another_vm: something_other

so i thought than an expression like

lookup('file','my_file.yaml') | from_yaml | selectattr('vm','search','vm1')|list

would work but it gives an error like

fatal: [localhost]: FAILED! => {"msg": "Unexpected templating type error occurred on ({{ lookup('file','{{sysfile}}') | from_yaml | selectattr('vm','search','vm1')|list}}): expected string or bytes-like object"}

If i remove the test_vm and another_vm keys it works fine.

ok: [localhost] => {
    "msg": [
        {
            "ip": "10.10.10.1",
            "vm": "vm1"
        }
    ]
}

If i try to search for the test_vm key it fails with:

fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'test_vm'nnThe error appears to be ...

Is selectattr filter expecting all the dicts in the list to have the same keys?Because it does not make any sense not to be able to filter a list of custom dicts with Jinja2.

For example if i had a more complicated yaml (not that flat) am i limited to search and filter within Ansible?

For example if i have a yaml looks like this:

---

- vm: "vm1"
  ip: 10.10.10.1
- vm: "vm2"
  ip: 10.10.10.2
- test_vm: something
   - process_1: X
   - process_2: Y
   - process_3: Z
- another_vm: something_other

how can i quickly filter for the process_2 for example?
Is there a way?

Thanks a lot in advance.

Asked By: jkalivas

||

Source

Answer 1

Is selectattr filter expecting all the dicts in the list to have the same keys?

More precisely, it is expecting all dicts in the list to have the attribute you are selecting on. If not all dict in the list have it, you will have to first filter out the items where it is not defined. This can be done with selectattr as well. (thanks @Randy for making this clearer since my initial answer).

In your situation, the json_query filter (which implements jmespath) can also do the job in sometimes a more compact manner. But it is not a core filter and requires to have the community.general collection installed.

Here are a few examples taken from your above requirements solved with both core filters and json_query solutions.

The playbook:

---
- name: "Filter data with core filters or json query"
  hosts: "localhost"
  gather_facts: false

  vars:
    # Your initial data on a single line for legibility
    test_var: [{"vm":"vm1","ip":"10.10.10.1"},{"vm":"vm2","ip":"10.10.10.2"},{"test_vm":"something","process_1":"X","process_2":"Y","process_3":"Z"},{"another_vm":"something_other"}]

  tasks:
    - name: Get objects having vm==vm1
      vars:
        msg: |-
          With core filters: {{ test_var | selectattr('vm', 'defined') | selectattr('vm', '==', 'vm1') | list }}
          With json_query: {{ test_var | json_query("[?vm=='vm1']") | list }}
      debug:
        msg: "{{ msg.split('n') }}"

    - name: Get all objects having vm attribute
      vars:
        msg: |-
          With core filters: {{ test_var | selectattr('vm', 'defined') | list }}
          With json_query: {{ test_var | json_query("[?vm]") | list }}
      debug:
        msg: "{{ msg.split('n') }}"

    - name: Get all objects having process_2 attribute
      vars:
        msg: |-
          With core filters: {{ test_var | selectattr('process_2', 'defined') | list }}
          With json_query: {{ test_var | json_query("[?process_2]") | list }}
      debug:
        msg: "{{ msg.split('n') }}"

    - name: Get only a list of process_2 attributes
      vars:
        msg: |-
          With core filters: {{ test_var | selectattr('process_2', 'defined') | map(attribute='process_2') | list }}
          With json_query: {{ test_var | json_query("[].process_2") | list }}
      debug:
        msg: "{{ msg.split('n') }}"

which gives:

PLAY [Filter data with core filters or json query] *********************************************************************

TASK [Get objects having vm==vm1] *********************************************************************
ok: [localhost] => {
    "msg": [
        "With core filters: [{'vm': 'vm1', 'ip': '10.10.10.1'}]",
        "With json_query: [{'vm': 'vm1', 'ip': '10.10.10.1'}]"
    ]
}

TASK [Get all objects having vm attribute] *********************************************************************
ok: [localhost] => {
    "msg": [
        "With core filters: [{'vm': 'vm1', 'ip': '10.10.10.1'}, {'vm': 'vm2', 'ip': '10.10.10.2'}]",
        "With json_query: [{'vm': 'vm1', 'ip': '10.10.10.1'}, {'vm': 'vm2', 'ip': '10.10.10.2'}]"
    ]
}

TASK [Get all objects having process_2 attribute] *********************************************************************
ok: [localhost] => {
    "msg": [
        "With core filters: [{'test_vm': 'something', 'process_1': 'X', 'process_2': 'Y', 'process_3': 'Z'}]",
        "With json_query: [{'test_vm': 'something', 'process_1': 'X', 'process_2': 'Y', 'process_3': 'Z'}]"
    ]
}

TASK [Get only a list of process_2 attributes] *********************************************************************
ok: [localhost] => {
    "msg": [
        "With core filters: ['Y']",
        "With json_query: ['Y']"
    ]
}

PLAY RECAP *********************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Answered By: Zeitounator

Answer 2

More precisely, it is expecting all dicts in the list to have the attribute you are selecting on.

It is not 100% true for all filter functions, to select objects by an attribute not defined by all elements:

{{ test_var | selectattr('vm','defined') |selectattr('vm','equalto','vm1') | list }}

Answered By: Randy

Ansible – how to use selectattr with yaml of different keys

Question:

Answers: