Python in Emacs shell-mode turns on stty echo and breaks C-d

Question:

When I run an interactive Python inside an Emacs shell buffer (M-x shell), it does two surprising things to the TTY. First, it turns on input echo, which persists after Python exits, until I do stty -echo. Secondly, it doesn’t accept C-d (or C-q C-d, i.e. ^D) as EOF: I have to type quit() to leave the Python. How can I stop these two behaviours?

I know that I could run python-shell, but I don’t want to: I’m noodling about in the shell and I want to do five lines of Python and then C-d out of it. So “run python-shell” is not an answer to my question.

Python running in a Terminal window is fine: ^D keeps working and echo doesn’t change.

Python 2.7.5, GNU Emacs 24.3.1, OS X 10.8.5

Edited to add this snippet from a shell buffer:

bash-3.2$ echo foo 
foo                     # no echo.
bash-3.2$ cat
foo                     # I typed this.
foo                     # cat returned it; no additional echo.
bash-3.2$ python
Python 2.7.5 (default, May 19 2013, 13:26:46) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>                   # C-d has no effect. C-q C-d has no effect.
                      # not sure where this blank link has come from.
>>> quit()              # I have to type this to get out of Python
quit()                  # note that it is echoed, like anything I now type.
bash-3.2$ echo foo
echo foo                # now I am getting my input echoed.
foo
bash-3.2$ cat
cat                     # echo of the 'cat' command.
foo                     # my input
foo                     # echo of my input.
foo                     # cat's output.
bash-3.2$ stty -echo    # turn echo back off.
stty -echo
bash-3.2$ echo foo
foo                     # and it's off.
bash-3.2$ 
Asked By: Nick Barnes

||

Answers:

0. Summary

If you installed Python via Macports, install the py27-gnureadline port (or py37-gnureadline, or whatever your version is) and the problem is fixed.

1. Reproduction

I can reproduce this (GNU Emacs 23.4.1; OS X 10.8.5; Python 3.3.2). Here’s a session in a fresh emacs -Q showing the problem:

$ stty -a > stty-before
$ python3.3
Python 3.3.2 (default, May 21 2013, 11:50:47) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
quit()
$ stty -a > stty-after
stty -a > stty-after
bash-3.2$ diff stty-before stty-after
diff stty-before stty-after
2c2
< lflags: icanon isig iexten -echo echoe -echok echoke -echonl echoctl
---
> lflags: icanon isig iexten echo echoe -echok echoke -echonl echoctl
7c7
< oflags: opost -onlcr -oxtabs -onocr -onlret
---
> oflags: opost onlcr -oxtabs -onocr -onlret
11,13c11,13
<   eol2 = <undef>; erase = <undef>; intr = ^C; kill = <undef>;
<   lnext = ^V; min = 1; quit = ^; reprint = ^R; start = ^Q;
<   status = ^T; stop = ^S; susp = ^Z; time = 0; werase = ^W;
---
>   eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
>   min = 1; quit = ^; reprint = ^R; start = ^Q; status = ^T;
>   stop = ^S; susp = ^Z; time = 0; werase = ^W;

So you can see that Python turned on the ECHO and ONLCR flags. Why did it do that? And why does it only do that on OS X?

2. What’s calling tcsetattr?

I ran Python under GDB and set a breakpoint on tcsetattr to see what’s calling it. Here are the relevant parts of the backtraces:

#0  0x00007fff898e7e63 in tcsetattr ()
#1  0x00000001007cbe96 in tty_init ()
#2  0x00000001007c19cf in el_init ()
#3  0x00000001007d1bb7 in rl_initialize ()
#4  0x00000001003f10ea in PyInit_readline ()
#0  0x00007fff898e7e63 in tcsetattr ()
#1  0x00000001007cc812 in tty_rawmode ()
#2  0x00000001007c610f in read_prepare ()
#3  0x00000001007c203d in el_wset ()
#4  0x00000001007d554d in el_set ()
#5  0x00000001003f128a in call_readline ()

PyInit_readline and call_readline are functions in readline.c, but you can see from the backtraces that this is not the real readline library that is being called here, but the mostly-compatible editline library. OS X ships with the BSD-licensed editline rather than the GPL-licensed readline, so this would explain why the behaviour is different on OS X from other Unixes.

3. It’s nothing to do with Python

The same thing happens with other interactive interpreters. I find that the Lua, Ruby and Sqlite3 command-line interpreters also turn on terminal echo when run inside Emacs. So it seems to be some kind of “feature” of the editline library. Let’s test that theory by running this short program:

#include <readline/readline.h>

int main() {
    char *line = readline("> ");
    return 0;
}

and sure enough, when compiled with

$ clang rl.c -lreadline

this program also turns on terminal echo when run inside Emacs. But when compiled with

$ clang rl.c -L/opt/local/lib -lreadline

which causes it to link with the real (GNU) readline library, installed by MacPorts, it works as expected (not turning on echo).

4. Bug and workaround

So this looks like a bug in the editline library. Let’s check that this really is the system version of the library, and not (say) the MacPorts version, using DYLD_PRINT_LIBRARIES:

$ export DYLD_PRINT_LIBRARIES=1
$ /usr/bin/python
dyld: loaded: /usr/bin/python
dyld: loaded: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
[... many lines omitted ...]
dyld: loaded: /usr/lib/libstdc++.6.dylib
Python 2.6.7 (r267:88850, Oct 11 2012, 20:15:00) 
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload/readline.so
dyld: loaded: /usr/lib/libedit.3.dylib
dyld: loaded: /usr/lib/libncurses.5.4.dylib
>>> 

I have reported this to Apple as bug 15184759. I understand that Apple use the number of people reporting an issue as an indicator of the issue’s severity, so please report the issue yourself if you want it to be fixed.

Now, I believe this broke in a fairly recent upgrade to OS X, so it seems likely that a recent change to libedit introduced the error. Here are the versions of libedit installed by MacPorts:

$ port installed libedit
The following ports are currently installed:
  libedit @20110802-3.0_0
  libedit @20120601-3.0_0
  libedit @20121213-3.0_0 (active)

If I go back in time to the June 2012 version:

$ sudo port activate [email protected]_0
--->  Computing dependencies for libedit
--->  Deactivating libedit @20121213-3.0_0
--->  Cleaning libedit
--->  Activating libedit @20120601-3.0_0
--->  Cleaning libedit

Then this fixes both problems (the terminal ECHO flag and the broken C-d) in the MacPorts versions of all the interactive interpreters I tested (Python, Ruby, Sqlite3).

So if you are looking for a workaround for your problems, this is it: use MacPorts to revert to a version of libedit before it broke, and put /opt/local/bin on your PATH so that when you type python you get the MacPorts installation of Python rather than the system one. (Possibly you do this already since I see that your Python is 2.7.5 whereas the system version is 2.6.7.)

5. Reporting to upstream

I downloaded the latest version of libedit from upstream to see if the problem has been fixed there. But it hasn’t. So I contacted Jess Thrysoee and reported the bug.

6. Update

As of December 2018, the problem has not yet been fixed in libedit. However, if you are using Macports then there is a workaround (see issue #48807): you can install the pyXX-gnureadline port (where XX is your Python version, for example py27-gnureadline or py35-gnureadline) which links Python against the GNU readline library instead of libedit. Now the terminal settings are unchanged:

$ sudo port install py37-gnureadline
[...]
$ stty -a > stty-before
$ python3.7
Python 3.7.1 (default, Oct 21 2018, 09:01:26) 
[Clang 10.0.0 (clang-1000.11.45.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
$ stty -a > stty-after
$ diff stty-before stty-after 
$ 
Answered By: Gareth Rees

Installing Python from MacPorts with the readline variant will work around this and other bugs in libedit.

$ sudo port install python27 +readline

I was having problems with Python 2.7.11 interleaving prompts incorrectly with output and leaving the tty in an insane state on exit. Configuring Python to use readline solved these problems.

Answered By: rptb1

This doesn’t seem to work any more (no readline variant for the python27 port). The underlying problem, of Python leaving the stty settings broken, seems to have gone away, although it’s still the case that C-d doesn’t do the right thing.

Answered By: Nick Barnes
Categories: questions Tags: , , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.