How to create a temporary file that can be read by a subprocess?

Question

I’m writing a Python script that needs to write some data to a temporary file, then create a subprocess running a C++ program that will read the temporary file. I’m trying to use NamedTemporaryFile for this, but according to the docs,

Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).

And indeed, on Windows if I flush the temporary file after writing, but don’t close it until I want it to go away, the subprocess isn’t able to open it for reading.

I’m working around this by creating the file with delete=False, closing it before spawning the subprocess, and then manually deleting it once I’m done:

fileTemp = tempfile.NamedTemporaryFile(delete = False)
try:
    fileTemp.write(someStuff)
    fileTemp.close()
    # ...run the subprocess and wait for it to complete...
finally:
    os.remove(fileTemp.name)

This seems inelegant. Is there a better way to do this? Perhaps a way to open up the permissions on the temporary file so the subprocess can get at it?

Asked By: Nathan Reed

||

Source

Answer 1

You can always go low-level, though am not sure if it’s clean enough for you:

fd, filename = tempfile.mkstemp()
try:
    os.write(fd, someStuff)
    os.close(fd)
    # ...run the subprocess and wait for it to complete...
finally:
    os.remove(filename)

Answered By: tshepang

Answer 2

At least if you open a temporary file using existing Python libraries, accessing it from multiple processes is not possible in case of Windows. According to MSDN you can specify a 3rd parameter (dwSharedMode) shared mode flag FILE_SHARE_READ to CreateFile() function which:

Enables subsequent open operations on a file or device to request read
access. Otherwise, other processes cannot open the file or device if
they request read access. If this flag is not specified, but the file
or device has been opened for read access, the function fails.

So, you can write a Windows specific C routine to create a custom temporary file opener function, call it from Python and then you can make your sub-process access the file without any error. But I think you should stick with your existing approach as it is the most portable version and will work on any system and thus is the most elegant implementation.

Discussion on Linux and windows file locking can be found here.

EDIT: Turns out it is possible to open & read the temporary file from multiple processes in Windows too. See Piotr Dobrogost’s answer.

Answered By: Nilanjan Basu

Answer 3

According to Richard Oudkerk

(…) the only reason that trying to reopen a NamedTemporaryFile fails on
Windows is because when we reopen we need to use O_TEMPORARY.

and he gives an example of how to do this in Python 3.3+

import os, tempfile

DATA = b"hello bob"

def temp_opener(name, flag, mode=0o777):
    return os.open(name, flag | os.O_TEMPORARY, mode)

with tempfile.NamedTemporaryFile() as f:
    f.write(DATA)
    f.flush()
    with open(f.name, "rb", opener=temp_opener) as f:
        assert f.read() == DATA

assert not os.path.exists(f.name)

Because there’s no opener parameter in the built-in open() in Python 2.x, we have to combine lower level os.open() and os.fdopen() functions to achieve the same effect:

import subprocess
import tempfile

DATA = b"hello bob"

with tempfile.NamedTemporaryFile() as f:
    f.write(DATA)
    f.flush()

    subprocess_code = 
    """import os
       f = os.fdopen(os.open(r'{FILENAME}', os.O_RDWR | os.O_BINARY | os.O_TEMPORARY), 'rb')
       assert f.read() == b'{DATA}'
    """.replace('n', ';').format(FILENAME=f.name, DATA=DATA)

    subprocess.check_output(['python', '-c', subprocess_code]) == DATA

Answered By: Piotr Dobrogost

Answer 4

Since nobody else appears to be interested in leaving this information out in the open…

tempfile does expose a function, mkdtemp(), which can trivialize this problem:

try:
    temp_dir = mkdtemp()
    temp_file = make_a_file_in_a_dir(temp_dir)
    do_your_subprocess_stuff(temp_file)
    remove_your_temp_file(temp_file)
finally:
    os.rmdir(temp_dir)

I leave the implementation of the intermediate functions up to the reader, as one might wish to do things like use mkstemp() to tighten up the security of the temporary file itself, or overwrite the file in-place before removing it. I don’t particularly know what security restrictions one might have that are not easily planned for by perusing the source of tempfile.

Anyway, yes, using NamedTemporaryFile on Windows might be inelegant, and my solution here might also be inelegant, but you’ve already decided that Windows support is more important than elegant code, so you might as well go ahead and do something readable.

Answered By: Corbin

Answer 5

Using mkstemp() instead with os.fdopen() in a with statement avoids having to call close():

fd, path = tempfile.mkstemp()
try:
    with os.fdopen(fd, 'wb') as fileTemp:
        fileTemp.write(someStuff)
    # ...run the subprocess and wait for it to complete...
finally:
    os.remove(path)

Answered By: Jon-Eric

Answer 6

I know this is a really old post, but I think it’s relevant today given that the API is changing and functions like mktemp and mkstemp are being replaced by functions like TemporaryFile() and TemporaryDirectory(). I just wanted to demonstrate in the following sample how to make sure that a temp directory is still available downstream:

Instead of coding:

tmpdirname = tempfile.TemporaryDirectory()

and using tmpdirname throughout your code, you should trying to use your code in a with statement block to insure that it is available for your code calls… like this:

with tempfile.TemporaryDirectory() as tmpdirname:
    [do dependent code nested so it's part of the with statement]

If you reference it outside of the with then it’s likely that it won’t be visible anymore.

Answered By: Clint Box

How to create a temporary file that can be read by a subprocess?

Question:

Answers: