Subprocess command shows FileNotFoundError: [Errno 2] No such file or directory

Question

I’m trying to run shell commands using python by using subprocess module in the below code, but I don’t why my script is throwing an error like below. Can someone help me what I’m missing?

Traceback (most recent call last):
  File "/Scripts/test_file.py", line 6, in <module>
    p2 = subprocess.Popen('sed s'/&quot;/ /g'', stdin=p1.stdout, stdout=subprocess.PIPE)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 854, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/subprocess.py", line 1702, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: "sed s'/&quot;/ /g'"`

import subprocess
#output3.txt='/Users/boggulv/Desktop/output3.txt'
p1 = subprocess.Popen( ['cat', 'output3.txt'], stdout=subprocess.PIPE)
print(p1)
p2 = subprocess.Popen('sed s'/&quot;/ /g'', stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen('grep "sO"', stdin=p2.stdout,stdout=subprocess.PIPE)
p4 = subprocess.Popen('grep -v "get"', stdin=p3.stdout, stdout=subprocess.PIPE)
p5 = subprocess.Popen('cut -d ',' -f2', stdin=p4.stdout, stdout=subprocess.PIPE)
p6 = subprocess.Popen('sed 's/"//g'', stdin=p5.stdout, stdout=subprocess.PIPE)
p7 = subprocess.Popen('sort', stdin=p6.stdout, stdout=subprocess.PIPE)
p8 = subprocess.Popen('sort', stdin=p8.stdout, stdout=subprocess.PIPE)
p9 = subprocess.Popen('uniq -c', stdin=p8.stdout, stdout=subprocess.PIPE)
p0 = subprocess.Popen('sort -nr', stdin=p9.stdout, stdout=subprocess.PIPE)
print(p01.communicate())

Tried now changing to lists.

p2 = subprocess.Popen('sed 's/&quot;/ /g'', stdin=p1.stdout, stdout=subprocess.PIPE, shell=True)
p3 = subprocess.Popen(['grep','"shipOption"'], stdin=p2.stdout,stdout=subprocess.PIPE,shell = True)
p4 = subprocess.Popen(['grep','-v', '"getShipMethod"'], stdin=p3.stdout, stdout=subprocess.PIPE,shell = True)
p5 = subprocess.Popen(['cut','-d','','', '-f2'], stdin=p4.stdout, stdout=subprocess.PIPE,shell = True)
p6 = subprocess.Popen(['sed',''s/"//g''],stdin=p5.stdout, stdout=subprocess.PIPE,shell = True)
p7 = subprocess.Popen(['sort'], stdin=p6.stdout, stdout=subprocess.PIPE,shell = True)
p8 = subprocess.Popen(['uniq', '-c'], stdin=p7.stdout, stdout=subprocess.PIPE,shell = True)
p9 = subprocess.Popen(['sort', '-nr'], stdin=p8.stdout, stdout=subprocess.PIPE,shell = True)
p0 = subprocess.Popen(['head', '-10'], stdin=p9.stdout, stdout=subprocess.PIPE,shell = True)```

New Error:

`usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context[=num]] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]
usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
        [-e pattern] [-f file] [--binary-files=value] [--color=when]
        [--context[=num]] [--directories=action] [--label] [--line-buffered]
        [--null] [pattern] [file ...]
usage: cut -b list [-n] [file ...]
       cut -c list [file ...]
       cut -f list [-s] [-w | -d delim] [file ...]
(b'', None)
cat: stdin: Input/output error`

Asked By: Tech Geek

||

Source

Answer 1

As per Gordon above – by default Popen() treats

sed 's/&quot;/ /g'

as the name of a command to run rather than a command name plus one argument. So you need to do one of the following:

p2 = subprocess.Popen(['sed', 's/&quot;/ /g'], stdin=p1.stdout, stdout=subprocess.PIPE)

or

p2 = subprocess.Popen('sed 's/&quot;/ /g'', stdin=p1.stdout, stdout=subprocess.PIPE, shell=True)

The use of shell=True gets the Popen function to split the string into a list containing the command and arguments for you.

Also note, that p0 is your last result, but you call p01.communicate()

Answered By: David Castell

Answer 2

Your commands are still wrong. If you just want to run these commands like the shell does, the absolutely easiest way to do that is to … use the shell.

result = subprocess.run('''
# useless cat, but bear with
cat output3.txt |
sed 's/&quot;/ /g' |
grep "shipOption" |
grep -v "getShipMethod" |
cut -d ',' -f2 |
sed 's/"//g' |
sort |
uniq -c |
sort -nr |
head -10
    ''',
    # Probably add these too
    check=True,
    capture_output=True,
    # We are using the shell for piping etc
    shell=True)

If you want to remove the shell=True and manually run all these processes, you have to understand how the shell works. In particular, you need to fix the quoting so that the commands you run have the quotes which remain after the shell has processed the syntactic quotes.

p1 = subprocess.Popen(['cat', 'output3.txt'], stdout=subprocess.PIPE)  # still useless
p2 = subprocess.Popen(['sed','s/&quot;/ /g'], stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(['grep', "shipOption"], stdin=p2.stdout,stdout=subprocess.PIPE)
p4 = subprocess.Popen(['grep', '-v', "getShipMethod"], stdin=p3.stdout, stdout=subprocess.PIPE)
p5 = subprocess.Popen(['cut', '-d', ',', '-f2'], stdin=p4.stdout, stdout=subprocess.PIPE)
p6 = subprocess.Popen(['sed', 's/"//g'],stdin=p5.stdout, stdout=subprocess.PIPE)
p7 = subprocess.Popen(['sort'], stdin=p6.stdout, stdout=subprocess.PIPE)
p8 = subprocess.Popen(['uniq', '-c'], stdin=p7.stdout, stdout=subprocess.PIPE)
p9 = subprocess.Popen(['sort', '-nr'], stdin=p8.stdout, stdout=subprocess.PIPE)
p0 = subprocess.Popen(['head', '-10'], stdin=p9.stdout, stdout=subprocess.PIPE)

Notice in particular how the arguments to sed and grep have their outer quotes removed, and how we removed shell=True everywhere. As a rule of thumb, if the first argument to Popen (or other subprocess methods) is a list, you should not use shell=True, and vice versa. (There are situations where you can pass a list to shell=True but … let’s not even begin to go there.)

All of this seems rather moot, though, since Python can eminently well do all of these things.

from collections import Counter

counts = Counter()
with open('output3.txt', 'r', encoding='utf-8') as lines:
    for line in lines:
        line = line.rstrip('n').replace('&quot;', ' ')
        if "shipOption" in line and "getShipMethod" not in line:
            field = line.split(',')[1].replace('"', '')
            counts[field] += 1
print(counts.most_common(10))

Probably you would want to put the rstrip and replace inside the if to avoid unnecessary work. The same refactoring could be done to the shell pipeline, of course.

Answered By: tripleee

Answer 3

Suppose you use command ‘python hello.py –main’ to run your hello.py file with –main as an argument.

this will throw an error that no such or file or directory is found.

subprocess.run(["python", "hello.py --main"],stdout=subprocess.PIPE)

in subprocess you would write it as

subprocess.run(["python", "hello.py", "--main"],stdout=subprocess.PIPE)

you can put this in a variable and print the value returned from the script like this

data = subprocess.run(["python", "hello.py", "--main"],stdout=subprocess.PIPE)

print(data.stdout.decode('utf-8').strip())

Answered By: GloatingCoder

Subprocess command shows FileNotFoundError: [Errno 2] No such file or directory

Question:

Answers: