os.replace unexpected behaviour

Question

To write to a log file on a linux system, I first create a temporary file with all of the content. Afterward, I replace the log file with os.replace because it is said to be atomic.

if os.path.exists(log_file_path):
    with open(log_file_path, 'r') as f:
        existing = json.load(f)
else:
    existing = list()
existing.append(event.error_log())    
with open(temp_file_name, 'a+') as f:
    json.dump(existing, f, indent=2, default=vars)

os.replace(temp_file_name, log_file_path)

The output of this code was unexpected when a signal 15 was sent to my service due to a reboot. After the execution, the file was not decodable as json anymore and had content, which was set together as follows:

File content before write access:

[complete log]

File content after write access:

[partial logs
[complete log]

The first list, does not have a closing bracket and contains only a part of the events which were in the file before the write access. The second list is the complete log from before the write access, plus the added events.

Does anybody know, what happened here?

Asked By: Sascha Lüthi

||

Source

Answer 1

You wrote some data to the temporary file, then the process got shut down before it could finish.

Then on the next run, you appended new data to the half-written garbage from the previous run, instead of discarding that garbage, because your code opens the temporary file in a+ mode instead of w or w+ mode.

Then you replaced the original file with the file still containing half-written garbage.

Answered By: user2357112

os.replace unexpected behaviour

Question:

Answers: