Atomic file write operations (cross platform)
Question:
How do I build up an atomic file write operation? The file is to be written by a Java service and read by python scripts.
For the record, reads are far greater than writes. But the write happens in batches and tend to be long. The file size amounts to mega bytes.
Right now my approach is:
- Write file contents to a temp file in
same directory
- Delete old file
- Rename temp file to old filename.
Is this the right approach? How can avoid conditions where the old file is deleted but the new filename is yet to be renamed?
Do these programming languages (python and java) offer constructs to lock and avoid this situation?
Answers:
AFAIK no.
And the reason is that for such an atomic operation to be possible, there has to be OS support in the form of a transactional file system. And none of the mainstream operating system offer a transactional file system.
EDIT – I’m wrong for POSIX-compliant systems at least. The POSIX rename syscall performs an atomic replace if a file with the target name already exists … as pointed out by @janneb. That should be sufficient to do the OP’s operation atomically.
However, the fact remains that the Java File.renameTo() method is explicitly not guaranteed to be atomic, so it does not provide a cross-platform solution to the OP’s problem.
EDIT 2 – With Java 7 you can use java.nio.file.Files.move(Path source, Path target, CopyOption... options) with copyOptions and ATOMIC_MOVE. If this is not supported (by the OS / file system) you should get an exception.
You could try and use an extra file to act as a lock, but I’m not sure if that will work out ok. (It would force you to create lock-checking and retry logic at both sides, java and python)
Another solution might be to not create files at all, maybe you could make your java process listen on a port and serve data from there rather than from a file?
At least on POSIX platforms, leave out step 3 (delete old file). In POSIX, rename within a filesystem is guaranteed to be atomic, and renaming on top of an existing file replaces it atomically.
Have the python scripts request permission from the service. While the service is writing it would place a lock on the file. If the lock exists, the service would reject the python request.
It’s a classic producer/consumer problem. You should be able to solve this by using file renaming, which is atomic on POSIX systems.
In Linux, Solaris, Unix this is easy. Just use rename() from your program or mv. The files need to be on the same filesystem.
On Windows, this is possible if you can control both programs. LockFileEx. For reads, open a shared lock on the lockfile. For writes, open an exclusive lock on the lockfile. Locking is weird in Windows, so I recommend using a separate lock file for this.
How do I build up an atomic file write operation? The file is to be written by a Java service and read by python scripts.
For the record, reads are far greater than writes. But the write happens in batches and tend to be long. The file size amounts to mega bytes.
Right now my approach is:
- Write file contents to a temp file in
same directory - Delete old file
- Rename temp file to old filename.
Is this the right approach? How can avoid conditions where the old file is deleted but the new filename is yet to be renamed?
Do these programming languages (python and java) offer constructs to lock and avoid this situation?
AFAIK no.
And the reason is that for such an atomic operation to be possible, there has to be OS support in the form of a transactional file system. And none of the mainstream operating system offer a transactional file system.
EDIT – I’m wrong for POSIX-compliant systems at least. The POSIX rename syscall performs an atomic replace if a file with the target name already exists … as pointed out by @janneb. That should be sufficient to do the OP’s operation atomically.
However, the fact remains that the Java File.renameTo() method is explicitly not guaranteed to be atomic, so it does not provide a cross-platform solution to the OP’s problem.
EDIT 2 – With Java 7 you can use java.nio.file.Files.move(Path source, Path target, CopyOption... options) with copyOptions and ATOMIC_MOVE. If this is not supported (by the OS / file system) you should get an exception.
You could try and use an extra file to act as a lock, but I’m not sure if that will work out ok. (It would force you to create lock-checking and retry logic at both sides, java and python)
Another solution might be to not create files at all, maybe you could make your java process listen on a port and serve data from there rather than from a file?
At least on POSIX platforms, leave out step 3 (delete old file). In POSIX, rename within a filesystem is guaranteed to be atomic, and renaming on top of an existing file replaces it atomically.
Have the python scripts request permission from the service. While the service is writing it would place a lock on the file. If the lock exists, the service would reject the python request.
It’s a classic producer/consumer problem. You should be able to solve this by using file renaming, which is atomic on POSIX systems.
In Linux, Solaris, Unix this is easy. Just use rename() from your program or mv. The files need to be on the same filesystem.
On Windows, this is possible if you can control both programs. LockFileEx. For reads, open a shared lock on the lockfile. For writes, open an exclusive lock on the lockfile. Locking is weird in Windows, so I recommend using a separate lock file for this.