How to construct an in-memory virtual file system and then write this structure to disk
Question:
I’m looking for a way to create a virtual file system in Python for creating directories and files, before writing these directories and files to disk.
Using PyFilesystem I can construct a memory filesystem using the following:
>>> import fs
>>> temp_dir = fs.open_fs('mem://')
>>> temp_dir.makedirs('fruit')
SubFS(MemoryFS(), '/fruit')
>>> temp_dir.makedirs('vegetables')
SubFS(MemoryFS(), '/vegetables')
>>> with temp_dir.open('fruit/apple.txt', 'w') as apple: apple.write('braeburn')
...
8
>>> temp_dir.tree()
├── fruit
│ └── apple.txt
└── vegetables
Ideally, I want to be able to do something like:
temp_dir.write_to_disk('<base path>')
To write this structure to disk, where <base path>
is the parent directory in which this structure will be created.
As far as I can tell, PyFilesystem has no way of achieving this. Is there anything else I could use instead or would I have to implement this myself?
Answers:
If you just want to stage a file system tree in memory, look at the tarfile
module.
Creating files and directories is a bit involved:
tarblob = io.BytesIO()
tar = tarfile.TarFile(mode="w", fileobj=tarblob)
dirinfo = tarfile.TarInfo("directory")
dirinfo.mode = 0o755
dirinfo.type = tarfile.DIRTYPE
tar.addfile(dirinfo, None)
filedata = io.BytesIO(b"Hello, world!n")
fileinfo = tarfile.TarInfo("directory/file")
fileinfo.size = len(filedata.getbuffer())
tar.addfile(fileinfo, filedata)
tar.close()
But then you can create the file system hierarchy using TarFile.extractall
:
tarblob.seek(0) # Rewind to the beginning of the buffer.
tar = tarfile.TarFile(mode="r", fileobj=tarblob)
tar.extractall()
You can use fs.copy.copy_fs()
to copy from one filesystem to another, or fs.move.move_fs()
to move the filesystem altogether.
Given that PyFilesystem also abstracts around the underlying system filesystem – OSFS
– in fact, it’s the default protocol, all you need is to copy your in-memory filesystem (MemoryFS
) to it and, in effect, you’ll have it written to the disk:
import fs
import fs.copy
mem_fs = fs.open_fs('mem://')
mem_fs.makedirs('fruit')
mem_fs.makedirs('vegetables')
with mem_fs.open('fruit/apple.txt', 'w') as apple:
apple.write('braeburn')
# write to the CWD for testing...
with fs.open_fs(".") as os_fs: # use a custom path if you want, i.e. osfs://<base_path>
fs.copy.copy_fs(mem_fs, os_fs)
I’m looking for a way to create a virtual file system in Python for creating directories and files, before writing these directories and files to disk.
Using PyFilesystem I can construct a memory filesystem using the following:
>>> import fs
>>> temp_dir = fs.open_fs('mem://')
>>> temp_dir.makedirs('fruit')
SubFS(MemoryFS(), '/fruit')
>>> temp_dir.makedirs('vegetables')
SubFS(MemoryFS(), '/vegetables')
>>> with temp_dir.open('fruit/apple.txt', 'w') as apple: apple.write('braeburn')
...
8
>>> temp_dir.tree()
├── fruit
│ └── apple.txt
└── vegetables
Ideally, I want to be able to do something like:
temp_dir.write_to_disk('<base path>')
To write this structure to disk, where <base path>
is the parent directory in which this structure will be created.
As far as I can tell, PyFilesystem has no way of achieving this. Is there anything else I could use instead or would I have to implement this myself?
If you just want to stage a file system tree in memory, look at the tarfile
module.
Creating files and directories is a bit involved:
tarblob = io.BytesIO()
tar = tarfile.TarFile(mode="w", fileobj=tarblob)
dirinfo = tarfile.TarInfo("directory")
dirinfo.mode = 0o755
dirinfo.type = tarfile.DIRTYPE
tar.addfile(dirinfo, None)
filedata = io.BytesIO(b"Hello, world!n")
fileinfo = tarfile.TarInfo("directory/file")
fileinfo.size = len(filedata.getbuffer())
tar.addfile(fileinfo, filedata)
tar.close()
But then you can create the file system hierarchy using TarFile.extractall
:
tarblob.seek(0) # Rewind to the beginning of the buffer.
tar = tarfile.TarFile(mode="r", fileobj=tarblob)
tar.extractall()
You can use fs.copy.copy_fs()
to copy from one filesystem to another, or fs.move.move_fs()
to move the filesystem altogether.
Given that PyFilesystem also abstracts around the underlying system filesystem – OSFS
– in fact, it’s the default protocol, all you need is to copy your in-memory filesystem (MemoryFS
) to it and, in effect, you’ll have it written to the disk:
import fs
import fs.copy
mem_fs = fs.open_fs('mem://')
mem_fs.makedirs('fruit')
mem_fs.makedirs('vegetables')
with mem_fs.open('fruit/apple.txt', 'w') as apple:
apple.write('braeburn')
# write to the CWD for testing...
with fs.open_fs(".") as os_fs: # use a custom path if you want, i.e. osfs://<base_path>
fs.copy.copy_fs(mem_fs, os_fs)