Hydra combine lists in config

Question

Say I have the following hydra config test.yaml:

list1 : [0]
list2 : [1,2,3]

is it possible to merge list1 and list2 into a new list that contains [0,1,2,3], perhaps using variable interpolation?

Here is the hydra code:

import hydra
from omegaconf import OmegaConf
@hydra.main(config_name="test.yaml", config_path="./")
def main(cfg):
    OmegaConf.resolve(cfg)
    print(cfg)
if __name__ == "__main__":
    main()

Attempt (failed):

list1 : [0]
list2 : [1,2,3]
list3 : 
  - ${list1} 
  - ${list2}

list3 gives [0,[1,2,3]]

The reason I want this is because I have some lists of unknown length in other config files and want to merge them to create an argument for object creation. I’d prefer not to do this in the code and just rely directly on hydras object instantiation (otherwise I’ll be doing list merging everywhere!).

Asked By: BenedictWilkins

||

Source

Answer 1

Turns out this is not too difficult, one can indeed rely on OmegaConfs variable interpolation with a custom resolver.

import hydra
from omegaconf import OmegaConf
# custom list merge resolver
OmegaConf.register_new_resolver("merge", lambda x, y : x + y)

@hydra.main(config_name="test.yaml", config_path="./")
def main(cfg):
    OmegaConf.resolve(cfg)
    print(cfg)

if __name__ == "__main__":
    main()

Config file test.yaml

list1 : [0]
list2 : [1,2,3]

list3 : ${merge:${list1},${list2}}

list3 is now [0,1,2,3] as desired.

Answered By: BenedictWilkins

Answer 2

I tried this on version 1.1.2.
and I’m getting this error:
any clue on how to fix it?

Traceback (most recent call last):
File "C:venvscolor-correctionlibsite-packageshydra_internalutils.py", line 377, in _run_hydra
    run_and_report(
  File "C:venvscolor-correctionlibsite-packageshydra_internalutils.py", line 214, in run_and_report
    raise ex
  File "C:venvscolor-correctionlibsite-packageshydra_internalutils.py", line 211, in run_and_report
    return func()
  File "C:venvscolor-correctionlibsite-packageshydra_internalutils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "C:venvscolor-correctionlibsite-packageshydra_internalhydra.py", line 88, in run
    cfg = self.compose_config(
  File "C:venvscolor-correctionlibsite-packageshydra_internalhydra.py", line 559, in compose_config
    cfg = self.config_loader.load_configuration(
  File "C:venvscolor-correctionlibsite-packageshydra_internalconfig_loader_impl.py", line 141, in load_configuration
    return self._load_configuration_impl(
  File "C:venvscolor-correctionlibsite-packageshydra_internalconfig_loader_impl.py", line 231, in _load_configuration_impl
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "C:venvscolor-correctionlibsite-packageshydracoreoverride_parseroverrides_parser.py", line 96, in parse_overrides
    raise OverrideParseException(
hydra.errors.OverrideParseException: Error parsing override 'list3=${merge:${list1},${list2}}'
extraneous input '}' expecting <EOF>
See https://hydra.cc/docs/next/advanced/override_grammar/basic for details
python-BaseException

Answered By: Ori Noked

Hydra combine lists in config

Question:

Answers: