How to get both REPL/arrow key functionality and Unicode characters working with Python3 under Git Bash/WinPTY
Question:
Windows 7, restricted user account running
Git git version 2.14.1.windows.1
MINGW/MSYS MINGW64_NT-6.1 REDACTED 2.8.2(0.313/5/3) 2017-07-12 15:35 x86_64 Msys
Python Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
locale has been set:
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=
Python IO Encoding env variable has been set
$ env | grep "PYTHON"
PYTHONIOENCODING=utf-8
Now for the ‘tests’:
$ /c/Python34/python.exe -i
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'utf-8'
>>> print('u2660')
♠
>>>
Unicode works but the arrow key functionality has been lost in the REPL. So instead, execute using WinPTY
$ winpty /c/Python34/python.exe -i
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'utf-8'
>>> print('u2660')
ΓÖá
>>>
Now I can navigate the history in the REPL, but Unicode is wonky.
So seemingly I can either be able to use the arrow keys in the REPL or proper unicode encoding, but not both. Anyone else have this issue or did I miss something here that will make this all work?
Note: I am restricted to this environment; I cannot readily install/update/modify apps at will.
Answers:
In the WinPTY case, the console is still using cp437 (default US OEM Windows encoding) but you’ve force Python to output UTF-8 via PYTHONIOENCODING. Switch to Python 3.6 since that version now uses the Windows Unicode console APIs to output Unicode and shouldn’t have issues with encodings.
Windows 7, restricted user account running
Git git version 2.14.1.windows.1
MINGW/MSYS MINGW64_NT-6.1 REDACTED 2.8.2(0.313/5/3) 2017-07-12 15:35 x86_64 Msys
Python Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
locale has been set:
$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=
Python IO Encoding env variable has been set
$ env | grep "PYTHON"
PYTHONIOENCODING=utf-8
Now for the ‘tests’:
$ /c/Python34/python.exe -i
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'utf-8'
>>> print('u2660')
♠
>>>
Unicode works but the arrow key functionality has been lost in the REPL. So instead, execute using WinPTY
$ winpty /c/Python34/python.exe -i
Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'utf-8'
>>> print('u2660')
ΓÖá
>>>
Now I can navigate the history in the REPL, but Unicode is wonky.
So seemingly I can either be able to use the arrow keys in the REPL or proper unicode encoding, but not both. Anyone else have this issue or did I miss something here that will make this all work?
Note: I am restricted to this environment; I cannot readily install/update/modify apps at will.
In the WinPTY case, the console is still using cp437 (default US OEM Windows encoding) but you’ve force Python to output UTF-8 via PYTHONIOENCODING. Switch to Python 3.6 since that version now uses the Windows Unicode console APIs to output Unicode and shouldn’t have issues with encodings.