Rsync, used from Python subprocess.run(), cannot read source (works from bash)
Question:
I’m trying to use rsync from Python. I’m calling it with subprocess.run(). (From what I have read, if I use subprocess.popen(), my program doesn’t wait for execution to finish, and I want it to wait.) I’m copying the /boot partition and the system partition, with some excludes. (Those are the only two partitions on the drive.)
The command line version of the two rsync commands I’m using are:
/usr/bin/rsync -aAXv --dry-run /boot/* /media/nboot/
/usr/bin/rsync -aAXv --dry-run --exclude=/boot/* --exclude=/dev/* --exclude=/home/* --exclude=/home/pi/tmp/* --exclude=/lost+found/* --exclude=/media/* --exclude=/mnt/* --exclude=/proc/* --exclude=/run/* --exclude=/sys/* tmp /* /media/nsys/
For reference, I’m creating a list in Python for the actual call and to create these commands I used ' '.join(rsync_boot) (or rsync_sys), so there’s no room for typos. I then copied the commands from the terminal, where they were printed out, and pasted them. Both commands worked just as they should in bash.
The actual calls I’m making in Python are:
info = subprocess.run(rsync_boot, text=True, capture_output=True) and
info = subprocess.run(rsync_sys, text=True, capture_output=True).
After the call, I pass info to a subroutine that prints info as a whole, then prints info.stdout and info.stderr.
I get this output for the first one:
Rsync Boot info: CompletedProcess(args=['/usr/bin/rsync', '-aAXv', '--dry-run', '/boot/*', '/media/nboot/'], returncode=23, stdout='sending incremental file listncreated directory /media/nbootnnsent 18 bytes received 47 bytes 130.00 bytes/secntotal size is 0 speedup is 0.00 (DRY RUN)n', stderr='rsync: link_stat "/boot/*" failed: No such file or directory (2)nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]n')
(That’s directly printing the object returned from the subprocess.run() command, using text=True, capture_output=True in the call.)
The output from stdout is:
sending incremental file list
created directory /media/nboot
sent 18 bytes received 47 bytes 130.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
The output from stderr is:
rsync: link_stat "/boot/*" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
For the 2nd one (syncing / and using excludes), the returned object, when printed, is this:
Rsync System info: CompletedProcess(args=['/usr/bin/rsync', '-aAXv', '--dry-run', '--exclude=/boot/*', '--exclude=/dev/*', '--exclude=/home/*', '--exclude=/home/pi/tmp/*', '--exclude=/lost+found/*', '--exclude=/media/*', '--exclude=/mnt/*', '--exclude=/proc/*', '--exclude=/run/*', '--exclude=/sys/*', 'tmp', '/*', '/media/nsys/'], returncode=23, stdout='sending incremental file listncreated directory /media/nsysnnsent 18 bytes received 46 bytes 128.00 bytes/secntotal size is 0 speedup is 0.00 (DRY RUN)n', stderr='rsync: link_stat "/home/pi/tmp" failed: No such file or directory (2)nrsync: link_stat "/*" failed: No such file or directory (2)nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]n')
The output from stdout is:
sending incremental file list
created directory /media/nsys
sent 18 bytes received 46 bytes 128.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
And the output from stderr is:
rsync: link_stat "/home/pi/tmp" failed: No such file or directory (2)
rsync: link_stat "/*" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
Note that there is a 2nd error here. Even though I have excluded "/home/*" it is, for some reason, still trying to read "/home/pi/tmp." I’m not clear why it wants to read anything in a directory tree that’s excluded.
The common error for both is that rsync is not able to properly handle the source directory, even though it exists and it can be read fine when I run the command for either version from bash.
I have read that sometimes there are issues with the ‘*’ character in this situation. If so, that would be an issue for the excludes as well. Also, I have read about using shell=True and that it can be a security risk. I’m building up the commands I’m using within the program, without using external output for any source material, so I’m wondering if that might be a good fix in this case, since there’s no input to sanitize.
This gives me with 3 questions:
- How can I get rsync, when run this way, to see and be able to work with the actual source directories?
- If the asterisk is an issue, won’t it also be an issue for the excludes? And if so, how do I address that?
- In this situation, without user input, would
shell=True be a good solution?
Answers:
- How can I get rsync, when run this way, to see and be able to work with the actual source directories?
The easiest: don’t use a wildcard (that would regularly be expanded by the shell) with the actual source/target specifications.
rsync /boot /media/nboot/ (with the rest of the extra flags) will be enough; adapt for the other command. (You may need to fiddle with whether /boot or the target has a trailing slash, I can never remember how the behavior changes with or without the slash.)
- If the asterisk is an issue, won’t it also be an issue for the excludes? And if so, how do I address that?
No. Those patterns are evaluated by rsync itself, and not having them expanded by the shell is the correct thing to do. On the command line, you’d want to pass them as --exclude '*foo*' (the single quotes preventing the shell from expanding the wildcards).
- In this situation, without user input, would shell=True be a good solution?
See point 1 – it’s not necessary.
By the way, you might want to add , check=True to your .run() call, so subprocess raises an exception if the subprocess fails.
I’m trying to use rsync from Python. I’m calling it with subprocess.run(). (From what I have read, if I use subprocess.popen(), my program doesn’t wait for execution to finish, and I want it to wait.) I’m copying the /boot partition and the system partition, with some excludes. (Those are the only two partitions on the drive.)
The command line version of the two rsync commands I’m using are:
/usr/bin/rsync -aAXv --dry-run /boot/* /media/nboot/
/usr/bin/rsync -aAXv --dry-run --exclude=/boot/* --exclude=/dev/* --exclude=/home/* --exclude=/home/pi/tmp/* --exclude=/lost+found/* --exclude=/media/* --exclude=/mnt/* --exclude=/proc/* --exclude=/run/* --exclude=/sys/* tmp /* /media/nsys/
For reference, I’m creating a list in Python for the actual call and to create these commands I used ' '.join(rsync_boot) (or rsync_sys), so there’s no room for typos. I then copied the commands from the terminal, where they were printed out, and pasted them. Both commands worked just as they should in bash.
The actual calls I’m making in Python are:
info = subprocess.run(rsync_boot, text=True, capture_output=True) and
info = subprocess.run(rsync_sys, text=True, capture_output=True).
After the call, I pass info to a subroutine that prints info as a whole, then prints info.stdout and info.stderr.
I get this output for the first one:
Rsync Boot info: CompletedProcess(args=['/usr/bin/rsync', '-aAXv', '--dry-run', '/boot/*', '/media/nboot/'], returncode=23, stdout='sending incremental file listncreated directory /media/nbootnnsent 18 bytes received 47 bytes 130.00 bytes/secntotal size is 0 speedup is 0.00 (DRY RUN)n', stderr='rsync: link_stat "/boot/*" failed: No such file or directory (2)nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]n')
(That’s directly printing the object returned from the subprocess.run() command, using text=True, capture_output=True in the call.)
The output from stdout is:
sending incremental file list
created directory /media/nboot
sent 18 bytes received 47 bytes 130.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
The output from stderr is:
rsync: link_stat "/boot/*" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
For the 2nd one (syncing / and using excludes), the returned object, when printed, is this:
Rsync System info: CompletedProcess(args=['/usr/bin/rsync', '-aAXv', '--dry-run', '--exclude=/boot/*', '--exclude=/dev/*', '--exclude=/home/*', '--exclude=/home/pi/tmp/*', '--exclude=/lost+found/*', '--exclude=/media/*', '--exclude=/mnt/*', '--exclude=/proc/*', '--exclude=/run/*', '--exclude=/sys/*', 'tmp', '/*', '/media/nsys/'], returncode=23, stdout='sending incremental file listncreated directory /media/nsysnnsent 18 bytes received 46 bytes 128.00 bytes/secntotal size is 0 speedup is 0.00 (DRY RUN)n', stderr='rsync: link_stat "/home/pi/tmp" failed: No such file or directory (2)nrsync: link_stat "/*" failed: No such file or directory (2)nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]n')
The output from stdout is:
sending incremental file list
created directory /media/nsys
sent 18 bytes received 46 bytes 128.00 bytes/sec
total size is 0 speedup is 0.00 (DRY RUN)
And the output from stderr is:
rsync: link_stat "/home/pi/tmp" failed: No such file or directory (2)
rsync: link_stat "/*" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1207) [sender=3.1.3]
Note that there is a 2nd error here. Even though I have excluded "/home/*" it is, for some reason, still trying to read "/home/pi/tmp." I’m not clear why it wants to read anything in a directory tree that’s excluded.
The common error for both is that rsync is not able to properly handle the source directory, even though it exists and it can be read fine when I run the command for either version from bash.
I have read that sometimes there are issues with the ‘*’ character in this situation. If so, that would be an issue for the excludes as well. Also, I have read about using shell=True and that it can be a security risk. I’m building up the commands I’m using within the program, without using external output for any source material, so I’m wondering if that might be a good fix in this case, since there’s no input to sanitize.
This gives me with 3 questions:
- How can I get rsync, when run this way, to see and be able to work with the actual source directories?
- If the asterisk is an issue, won’t it also be an issue for the excludes? And if so, how do I address that?
- In this situation, without user input, would
shell=Truebe a good solution?
- How can I get rsync, when run this way, to see and be able to work with the actual source directories?
The easiest: don’t use a wildcard (that would regularly be expanded by the shell) with the actual source/target specifications.
rsync /boot /media/nboot/ (with the rest of the extra flags) will be enough; adapt for the other command. (You may need to fiddle with whether /boot or the target has a trailing slash, I can never remember how the behavior changes with or without the slash.)
- If the asterisk is an issue, won’t it also be an issue for the excludes? And if so, how do I address that?
No. Those patterns are evaluated by rsync itself, and not having them expanded by the shell is the correct thing to do. On the command line, you’d want to pass them as --exclude '*foo*' (the single quotes preventing the shell from expanding the wildcards).
- In this situation, without user input, would shell=True be a good solution?
See point 1 – it’s not necessary.
By the way, you might want to add , check=True to your .run() call, so subprocess raises an exception if the subprocess fails.