How can I have parallel Snakemake jobs fit in limited memory?

Question

The Snakemake‘s scheduler ignores my mem_mb declaration and executes in parallel jobs which summed requirements exceed the available memory (e.g. three jobs with mem_mb=53000 in a 128 GB system). Moreover, it runs even jobs which declared requirements (over 1TB when I run snakemake -T10) cannot be met in my system even if run serially. Snakemake also seems to keep a job running even if it allocates much more memory than it declared.

What I have in mind is to tell Snakemake to expect a job allocating up to certain amount of memory, plan the workflow accordingly and enforce the declared constraints on memory consumption. Is there any way to do so with Snakemake without resorting to serial execution?

I have a workflow with a lot of calls to a rule which may be either quite memory-light or memory-heavy. As I want to benefit from parallel execution, I have declared that the job requires 1000 MiB in first attempt to run, and more in subsequent attempts. The rule looks like this:

def get_mem_mb(wildcards, attempt):
    return 1_000 if attempt == 1 else (51_000 + 1_000 * 2 ** (attempt - 1))

rule Rulename:
    input:
        "{CONFIG}.ini"
    output:
        "{CONFIG}.h5"
    resources:
        mem_mb=get_mem_mb
    shell:
        "python script.py -o {output} -i {input}"

This is not a duplicate of Snakemake memory limiting, as the only answer there is incomplete (it does not cover the "memory limiting" part). At the moment, it is my question which has the complete (but split) answer:

Asked By: abukaj

||

Source

Answer 1

I’m basing this on a previous answer of mine, but I am not sure if I understand the question correctly. Perhaps you can run snakemake with e.g. --restart-times 10 and dynamic memory constraints:

rule all:
    input:
        "a"

def get_mem_mb(wildcards, attempt):
    """
    First attempt uses 10 MB, second attempt uses 100MB, third 1GB,
    etc etc
    """
    return 10**attempt

rule:
    output:
        "a"
    threads: 1
    resources:
        mem_mb = get_mem_mb
    shell:
        """
        ulimit -v {resources.mem_mb}
        python3 -c 'import numpy; x=numpy.ones(1_000_000_000)'
        touch a
        """

Note that this will only work on linux machines. What this approach does it allocates a certain amount of memory for a rule, 10MB on the first try. If the rule tries to allocate more than that, the rule crashes, and it gets restarted with 100MB of memory. If that fails 1GB, etc. etc. You probably want to change how the get_mem_mb rule scales into e.g. 2**attempt.

Answered By: Maarten-vd-Sande

Answer 2

I have found a hint in a comment to related question: How to set binding memory limits in snakemake

"ressources" in snakemake are […] an arbitrary value that you assign
to the whole snakemake run, and snakemake will not run simultaneously
jobs whose total in that ressource goes above the assigned amount for
the run

Thus, to prevent Snakemake from running memory-heavy jobs simultaneously, I need to tell it what the limit is (e.g. --res mem_mb=100000).

Answered By: abukaj

How can I have parallel Snakemake jobs fit in limited memory?

Question:

Answers: