UnicodeDecodeError: 'utf-8' when debugging Python files in PyCharm Community
Question:
Current conclusion:
The encoding of the converted file is utf-8->utf-8 big->ansi -> utf-8. Reopen the file after each conversion.
After observing for a period of time, there is no such error.
When I use PyCharm to debug .py files, the same file sometimes has UnicodeDecodeError, sometimes it’s normal. My operating system is Windows 10, PyCharm version is 2020.3.3 Community edition.
The error is as follows:
Traceback (most recent call last):
File "D:Program FilesJetBrainsPyCharm Community Edition 2020.3.3pluginspython-cehelperspydev_pydevd_bundlepydevd_comm.py", line 301, in _on_run
r = r.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1022-1023: unexpected end of data
I tried to add the following code to the header of the file, but sometimes I still get an error, how to solve it?
#!/usr/bin/env Python
# coding=utf-8
I found another way to save as a UTF-8 document with Notepad. I tried it, but there are still errors sometimes.
Answers:
Add ‘ignore’. It worked.
r = r.decode('utf-8', 'ignore')
There isn’t one single answer to the problem as it is described in the question. A number of issues can cause the indicated error, so it’s best to address the several possible factors in the context of the PyCharm IDE.
-
Every Python file .py (or any other file for that matter) has an encoding. The default encoding of a .py source code file is Unicode UTF-8. This problem is frequently faced by beginners, so lets pinpoint the relevant quotes from the official documentation (to shorten any unnecessary reading time):
The default encoding for Python source code is UTF-8, so you can simply include a Unicode character in a string literal.
This means in most circumstances you shouldn’t need the encoding string, see Python Source Code Encodings – PEP 263. Current practice is having the source files encoded by default in UTF-8 and omitting the encoding string at the top of the module (this is also more concise).
-
The PyCharm IDE has a number of encoding configurations that can successively be refined, going from global, to project, to file path. By default, everything should be set to UTF-8, especially the source code. See the official PyCharm documentation Configure file encoding settings.
-
The exception to the above should be if you are processing external data files, in which case your source code should still be kept as UTF-8 and the data file opened with whatever encoding it requires. Most questions about the UnicodeDecodeError are about specifying the right file encoding when using the open() function to open some data file (they are not about the encoding of the source files where you are writing your code).
-
When your source files cause this error, a frequent cause is after copy-pasting, or opening, a source code file that is not encoded in UTF-8. (The copy-paste is especially unexpected, when you copy from a file that isn’t encoded in UTF-8 and the IDE doesn’t automatically convert what you are copy-pasting into the editor). This can cause the said error. So you should narrow down which source code file has the encoding that isn’t UTF-8 and convert it.
We don’t have access to your project files, but the error message to me reads as the debugger trying to open a user source code file that isn’t encoded in UTF-8, contrary to the IDE configurations and module encoding.
File "D:Program FilesJetBrainsPyCharm Community Edition 2020.3.3pluginspython-cehelperspydev_pydevd_bundlepydevd_comm.py"
I faced the same issue and finally fixed it after changing the case of the python executable used in my interpreter settings, as explained here. Long story short, sometimes PyCharm try to execute the symlink in the venv directory using Python (with the "P" in uppercase) instead of python. After changing this I could debug again.
Look at Stefan Ukena’s answer on this thread:
https://youtrack.jetbrains.com/issue/PY-14497#focus=Comments-27-5243196.0-0
Quote in case the link dies:
You might need to change your Python Interpreter in the Pycharm > settings. In my case (using pipenv), it was pointing to /Library/…/bin/Python with an uppercase P. Opening the folder and checking, I found that the file or symlink was actually python with a lowercase p. Changing it from …/Python to …/python in the Pycharm settings fixed this problem. (I had to restart Pycharm afterwards.)
It helped me too, but only when I switched to pipenv instead of the usual venv. I changed Python to python and the debugger worked, but I’m still getting errors/warnings:
OSError: [Errno 9] Bad file descriptor
But it works anyway. Without the debugger, it works as expected and without the error above.
Current conclusion:
The encoding of the converted file is utf-8->utf-8 big->ansi -> utf-8. Reopen the file after each conversion.
After observing for a period of time, there is no such error.
When I use PyCharm to debug .py files, the same file sometimes has UnicodeDecodeError, sometimes it’s normal. My operating system is Windows 10, PyCharm version is 2020.3.3 Community edition.
The error is as follows:
Traceback (most recent call last):
File "D:Program FilesJetBrainsPyCharm Community Edition 2020.3.3pluginspython-cehelperspydev_pydevd_bundlepydevd_comm.py", line 301, in _on_run
r = r.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 1022-1023: unexpected end of data
I tried to add the following code to the header of the file, but sometimes I still get an error, how to solve it?
#!/usr/bin/env Python
# coding=utf-8
I found another way to save as a UTF-8 document with Notepad. I tried it, but there are still errors sometimes.
Add ‘ignore’. It worked.
r = r.decode('utf-8', 'ignore')
There isn’t one single answer to the problem as it is described in the question. A number of issues can cause the indicated error, so it’s best to address the several possible factors in the context of the PyCharm IDE.
-
Every Python file
.py(or any other file for that matter) has an encoding. The default encoding of a.pysource code file is Unicode UTF-8. This problem is frequently faced by beginners, so lets pinpoint the relevant quotes from the official documentation (to shorten any unnecessary reading time):The default encoding for Python source code is UTF-8, so you can simply include a Unicode character in a string literal.
This means in most circumstances you shouldn’t need the encoding string, see Python Source Code Encodings – PEP 263. Current practice is having the source files encoded by default in UTF-8 and omitting the encoding string at the top of the module (this is also more concise).
-
The PyCharm IDE has a number of encoding configurations that can successively be refined, going from global, to project, to file path. By default, everything should be set to UTF-8, especially the source code. See the official PyCharm documentation Configure file encoding settings.
-
The exception to the above should be if you are processing external data files, in which case your source code should still be kept as UTF-8 and the data file opened with whatever encoding it requires. Most questions about the
UnicodeDecodeErrorare about specifying the right file encoding when using theopen()function to open some data file (they are not about the encoding of the source files where you are writing your code). -
When your source files cause this error, a frequent cause is after copy-pasting, or opening, a source code file that is not encoded in UTF-8. (The copy-paste is especially unexpected, when you copy from a file that isn’t encoded in UTF-8 and the IDE doesn’t automatically convert what you are copy-pasting into the editor). This can cause the said error. So you should narrow down which source code file has the encoding that isn’t UTF-8 and convert it.
We don’t have access to your project files, but the error message to me reads as the debugger trying to open a user source code file that isn’t encoded in UTF-8, contrary to the IDE configurations and module encoding.
File "D:Program FilesJetBrainsPyCharm Community Edition 2020.3.3pluginspython-cehelperspydev_pydevd_bundlepydevd_comm.py"
I faced the same issue and finally fixed it after changing the case of the python executable used in my interpreter settings, as explained here. Long story short, sometimes PyCharm try to execute the symlink in the venv directory using Python (with the "P" in uppercase) instead of python. After changing this I could debug again.
Look at Stefan Ukena’s answer on this thread:
https://youtrack.jetbrains.com/issue/PY-14497#focus=Comments-27-5243196.0-0
Quote in case the link dies:
You might need to change your Python Interpreter in the Pycharm > settings. In my case (using pipenv), it was pointing to /Library/…/bin/Python with an uppercase P. Opening the folder and checking, I found that the file or symlink was actually python with a lowercase p. Changing it from …/Python to …/python in the Pycharm settings fixed this problem. (I had to restart Pycharm afterwards.)
It helped me too, but only when I switched to pipenv instead of the usual venv. I changed Python to python and the debugger worked, but I’m still getting errors/warnings:
OSError: [Errno 9] Bad file descriptor
But it works anyway. Without the debugger, it works as expected and without the error above.
