How to ensure file has been written to disk before reading it

Question

I want to read a plan.txt file generated by the fast-downward planner, to do so I’m using subprocess as follows:

import subprocess

search_options = "lama-first"
## Call Planner
cmd = [
    "python3",
    self.planner_path,
    "--alias",
    search_options,
    "--plan-file",
    self.plan_path,
    self.domain_path,
    self.problem_path,
]
## Read plan.txt
cmd2 = ["cat", self.plan_path]
subprocess.run(cmd, shell=False, stdout=subprocess.PIPE)

plan = []
try:
    # Huge dealy to be sure that the plan is written to the txt file
    time.sleep(1)
    result2 = subprocess.run(cmd2, shell=False, stdout=subprocess.PIPE)
    plan = result2.stdout.decode("utf-8")
except:
    pass

Can you please tell me what is the best approach to be sure that the plan was written to disk before trying to read it, rather than adding huge time delay? thanks in advance.

Asked By: Bilal

||

Source

Answer 1

No need to call time.sleep(...), because (1) subprocess.run waits for the subprocess to exit; (2) the libc and the kernel together make sure that all pending data is flushed to the output files when the (sub)process exits.

The modified code looks like this:

import os
import subprocess

try:
    # Remove output of previous invocation.
    os.remove(self.plan_path)
except OSError:
    pass
search_options = "lama-first"
cmd = [
    "python3",
    self.planner_path,
    "--alias",
    search_options,
    "--plan-file",
    self.plan_path,
    self.domain_path,
    self.problem_path,
]
subprocess.run(cmd).check_returncode()
with open(self.plan_path, encoding="utf-8") as f:
    plan = f.read()

Some comments about the code in the question:

After subprocess.run(...), call .check_returncode(), which aborts if the subprocess hasn’t exited successfully (with exit code 0, indicating success).
Don’t use except: pass. It hides all exceptions, so you won’t be notified about relevant errors.
For simplicity and portability (e.g. to Windows), use open(…) + read() rather than cat to read a file in Python.
No need to specify shell=False, that’s the default when cmd is a list or tuple.

FYI Because of kernel buffers and caches, data may be flushed to permanent storage (e.g. filesystem on the HDD or SSD) several seconds later. However, you don’t have to wait for that in your program, because the kernel will make sure that any subsequent read following a (flushed) file write will see the output of the write, even if the output is only in buffers or caches.

Answered By: pts

How to ensure file has been written to disk before reading it

Question:

Answers: