piping from stdin to a python code in a bash script

Question

I have a bash script, f, that contains python code. That python code reads from standard input. I want to be able to call my bash script as follows:

f input.txt > output.txt

In the example above, the python code will read from input.txt and will write to output.txt.

I’m not sure how to do this. I know that if I wanted to just write to a file, then my bash script would look like this

#!/bin/bash
python << EOPYTHON > output.txt
#python code goes here
EOPYTHON

I tried changing the second line in the code above to the following, but without luck

python << EOPYTHON $*

I’m not sure how else to go about doing this. Any suggestions?

EDIT
I’ll give a more concrete example. Consider the following bash script, f

#!/bin/bash
python << EOPYTHON 
import sys
import fileinput
for i in fileinput.input():
    sys.stdout.write(i + 'n')
EOPYTHON

I want to run my code with the following command

f input.txt > output.txt

How do I change my bash script so that it uses “input.txt” as the input stream?

Asked By: user2699231

||

Source

Answer 1

You can just check it against the file descriptor list for the process, i.e. on a proc filesystem you can print the redirection location for stdout with

readlink /proc/$$/fd/1

For example

> cat test.sh
#!/bin/bash
readlink /proc/$$/fd/1
> ./test.sh
/dev/pts/3
> ./test.sh > out.txt
> cat out.txt 
/home/out.txt

Answered By: Reinstate Monica Please

Answer 2

Updated Answer

If you absolutely must run the way you ask, you could do something like this:

#!/bin/bash
python -c 'import os
for i in range(3):
   for j in range(3):
     print(i + j)
'  < "$1"

Original Answer

Save your python code in a file called script.py and change your script f to this:

#!/bin/bash
python script.py < "$1"

Answered By: Mark Setchell

Answer 3

-c option from @Mark Setchell’s answer should work.

Here’s an alternative where you embed bash in Python script:

#!/usr/bin/env python
import sys
import fileinput
from subprocess import call

# shell command before the python code
rc = call(r"""
some bash-specific commands here
...
""", shell=True, executable='/bin/bash')

for line in fileinput.input():
    sys.stdout.write(line) #NOTE: `line` already has a newline

# shell command after the python code
rc = call(r"""
some /bin/sh commands here
...
""", shell=True)

Answered By: jfs

Answer 4

As no one mentioned this, here is what author requested. The magic is to pass “-” as argument to cpython (instruction to read source code from stdin):

With output to file:

python - << EOF > out.txt
print("hello")
EOF

Execution sample:

# python - << EOF
> print("hello")
> EOF
hello

As data can’t be passed via stdin anymore, here is another trick:

data=`cat input.txt`

python - <<EOF

data="""${data}"""
print(data)

EOF

Answered By: Reishin

Answer 5

Use process substitution for passing the python code. This frees up regular io indirection which can be used as usual.

❯ seq 1 5 > input.txt
❯ python3 <input.txt <(<< EOS
import sys
print("Running script with args: ", sys.argv)
for line in sys.stdin:
  print("line is %s" % line, end='')
EOS
)
Running script with args:  ['/proc/self/fd/11']
line is 1
line is 2
line is 3
line is 4
line is 5

Answered By: balki

Answer 6

For the sake of completeness, you can also use bash process substitution to trick python into reading from the substituted file descriptor. An example I used recently:

IFUPDOWN_VERSION=$(apt-cache show ifupdown | grep -Po '(?<=^Version: ).*')

if ! python <(cat<<EOF
from distutils.version import LooseVersion, StrictVersion
if LooseVersion("${IFUPDOWN_VERSION}") < LooseVersion("0.8.36"):
  exit(1)
EOF
              ); then
    echo "ifupdown version doesn't support 'client no' option!"
    exit ${LINENO}
fi

This is similar to calling python myscript.py.

Answered By: ealfonso

piping from stdin to a python code in a bash script

Question:

Answers: