Why use Python's os module methods instead of executing shell commands directly?

There are four strong cases for preferring Python’s more-specific methods in the os module over using os.system or the subprocess module when executing a command:

• Redundancy – spawning another process is redundant and wastes time and resources.
• Portability – Many of the methods in the os module are available in multiple platforms while many shell commands are os-specific.
• Understanding the results – Spawning a process to execute arbitrary commands forces you to parse the results from the output and understand if and why a command has done something wrong.
• Safety – A process can potentially execute any command it’s given. This is a weak design and it can be avoided by using specific methods in the os module.

Redundancy (see redundant code):

You’re actually executing a redundant “middle-man” on your way to the eventual system calls (chmod in your example). This middle man is a new process or sub-shell.

From os.system:

Execute the command (a string) in a subshell …

And subprocess is just a module to spawn new processes.

You can do what you need without spawning these processes.

Portability (see source code portability):

The os module’s aim is to provide generic operating-system services and it’s description starts with:

This module provides a portable way of using operating system dependent functionality.

You can use os.listdir on both windows and unix. Trying to use os.system / subprocess for this functionality will force you to maintain two calls (for ls / dir) and check what operating system you’re on. This is not as portable and will cause even more frustration later on (see Handling Output).

Understanding the command’s results:

Suppose you want to list the files in a directory.

If you’re using os.system("ls") / subprocess.call(['ls']), you can only get the process’s output back, which is basically a big string with the file names.

How can you tell a file with a space in it’s name from two files?

What if you have no permission to list the files?

How should you map the data to python objects?

These are only off the top of my head, and while there are solutions to these problems – why solve again a problem that was solved for you?

This is an example of following the Don’t Repeat Yourself principle (Often reffered to as “DRY”) by not repeating an implementation that already exists and is freely available for you.

Safety:

os.system and subprocess are powerful. It’s good when you need this power, but it’s dangerous when you don’t. When you use os.listdir, you know it can not do anything else other then list files or raise an error. When you use os.system or subprocess to achieve the same behaviour you can potentially end up doing something you did not mean to do.

Injection Safety (see shell injection examples):

If you use input from the user as a new command you’ve basically given him a shell. This is much like SQL injection providing a shell in the DB for the user.

An example would be a command of the form:

# ... read some user input
os.system(user_input + " some continutation")

This can be easily exploited to run any arbitrary code using the input: NASTY COMMAND;# to create the eventual:

os.system("NASTY COMMAND; # some continuation")

There are many such commands that can put your system at risk.

For a simple reason – when you call a shell function, it creates a sub-shell which is destroyed after your command exists, so if you change directory in a shell – it does not affect your environment in Python.

Besides, creating sub-shell is time consuming, so using OS commands directly will impact your performance

EDIT

I had some timing tests running:

In [379]: %timeit os.chmod('Documents/recipes.txt', 0755)
10000 loops, best of 3: 215 us per loop

In [380]: %timeit os.system('chmod 0755 Documents/recipes.txt')
100 loops, best of 3: 2.47 ms per loop

In [382]: %timeit call(['chmod', '0755', 'Documents/recipes.txt'])
100 loops, best of 3: 2.93 ms per loop

Internal function runs more than 10 time faster

EDIT2

There may be cases when invoking external executable may yield better results than Python packages – I just remembered a mail sent by a colleague of mine that performance of gzip called through subprocess was much higher than the performance of a Python package he used. But certainly not when we are talking about standard OS packages emulating standard OS commands

It is safer. To give you an idea here is an example script

import os
file = raw_input("Please enter a file: ")
os.system("chmod 777 " + file)

If the input from the user was test; rm -rf ~ this would then delete the home directory.

This is why it is safer to use the built in function.

Hence why you should use subprocess instead of system too.

Shell call are OS specific whereas Python os module functions are not, in most of the case. And it avoid spawning a subprocess.

It’s far more efficient. The “shell” is just another OS binary which contains a lot of system calls. Why incur the overhead of creating the whole shell process just for that single system call?

The situation is even worse when you use os.system for something that’s not a shell built-in. You start a shell process which in turn starts an executable which then (two processes away) makes the system call. At least subprocess would have removed the need for a shell intermediary process.

It’s not specific to Python, this. systemd is such an improvement to Linux startup times for the same reason: it makes the necessary system calls itself instead of spawning a thousand shells.