UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte when deploying to Heroku
Question:
I’m trying to follow this tutorial: http://tutorial.djangogirls.org/en/index.html
I’m up to this part: http://tutorial.djangogirls.org/en/deploy/README.html
Where I am to push it up to heroku via git. I’m familiar with git just not heroku and while I know python I’m a django beginner.
When I do the command git push heroku master i get this output which prevents the app from being deployed.
Here is the error I am receiving:
(myvenv) $> git push heroku master
Counting objects: 19, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (16/16), done.
Writing objects: 100% (19/19), 3.81 KiB | 0 bytes/s, done.
Total 19 (delta 0), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: -----> Installing runtime (python-3.4.1)
remote: -----> Installing dependencies with pip
remote: Exception:
remote: Traceback (most recent call last):
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip- 6.0.6-py3.4.egg/pip/basecommand.py", lin
, in main
remote: status = self.run(options, args)
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip- 6.0.6-py3.4.egg/pip/commands/install.py"
e 321, in run
remote: finder=finder, options=options, session=session):
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip-6.0.6-py3.4.egg/pip/req/req_file.py", li
, in parse_requirements
remote: session=session,
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip- 6.0.6-py3.4.egg/pip/download.py", line 4
n get_file_content
remote: content = f.read()
remote: File "/app/.heroku/python/lib/python3.4/codecs.py", line 313, in decode
remote: (result, consumed) = self._buffer_decode(data, self.errors, final)
remote: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
remote:
remote:
remote: ! Push rejected, failed to compile Python app
remote:
remote: Verifying deploy...
remote:
remote: ! Push rejected to chsdjangoblog.
remote:
To https://git.heroku.com/chsdjangoblog.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/chsdjangoblog.git'
does anyone have any idea why this is occurring? heroku seems nice to use, is there a better alternative/what are the best use cases for heroku? I really just want to solve this issue so I can continue the tutorial. Learning django has been a goal of mine for a while as I’m sick of Word Press and PHP development and have been a long time Python lover.
After that error when I try the next step: heroku ps:scale web=1 i get this output:
Scaling dynos... failed
! App mus tbe deployed before dynos can be scaled.
Thanks in advance.
EDIT:
Here’s my requirements.txt:
Django==1.8
dj-database-url==0.3.0
gunicorn==19.3.0
heroku==0.1.4
python-dateutil==1.5
requests==2.6.0
whitenoise==1.0.6
psycopg2==2.5.4`
I have tried saving as UTF-8, ANSI, UTF-16. Same message for all of them. I even rewrote it without copy paste. Why is my first byte always 0xff regardless of encoding? What is heroku expecting and is there a way/tool to check the bytes in a txt file?
Answers:
See this answer: Deploying Django/Python 3.4 to Heroku
pip is crashing because your requirements.txt file is encoded wrong. Save it in UTF-8 ANSI encoding.
On the shoulders of the other contributors to this question, it looks like your requirements.txt file is encoded as UTF-16 little endian.
0xFF is the first character of the Byte Order Mark for UTF-16-LE, the second character being 0xFE. The traceback states that the first character is 0xFF in position 0, and it is common in Windows for files to be stored as UTF-16 with the BOM.
Try saving the requirements.txt file as UTF-8 without BOM, or as ASCII. Simple old notepad.exe might do the trick.
Edit
Not working in notepad, so use Python 3 instead:
with open('requirements.txt', encoding='utf-16') as old, open('requirements_new.txt', 'w', encoding='utf-8') as new:
new.write(old.read())
requirements_new.txt will now be encoded as UTF-8 and should work (it will probably end up as ASCII anyway).
Note that this is based on the comments and answers of others which have suggested that the troublesome file is requirements.txt.
I fixed it, the problem seemed to occur only when requirements.txt was generated on windows no matter what encoding I picked. I generated the file in ascii on Linux and it worked. I then transferred the file to windows and it worked there as well. Therefore the problem must be the requirements.txt encoding as was mentioned in comments. However, the correct encoding appears to be ASCII.
I’m trying to follow this tutorial: http://tutorial.djangogirls.org/en/index.html
I’m up to this part: http://tutorial.djangogirls.org/en/deploy/README.html
Where I am to push it up to heroku via git. I’m familiar with git just not heroku and while I know python I’m a django beginner.
When I do the command git push heroku master i get this output which prevents the app from being deployed.
Here is the error I am receiving:
(myvenv) $> git push heroku master
Counting objects: 19, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (16/16), done.
Writing objects: 100% (19/19), 3.81 KiB | 0 bytes/s, done.
Total 19 (delta 0), reused 0 (delta 0)
remote: Compressing source files... done.
remote: Building source:
remote:
remote: -----> Python app detected
remote: -----> Installing runtime (python-3.4.1)
remote: -----> Installing dependencies with pip
remote: Exception:
remote: Traceback (most recent call last):
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip- 6.0.6-py3.4.egg/pip/basecommand.py", lin
, in main
remote: status = self.run(options, args)
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip- 6.0.6-py3.4.egg/pip/commands/install.py"
e 321, in run
remote: finder=finder, options=options, session=session):
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip-6.0.6-py3.4.egg/pip/req/req_file.py", li
, in parse_requirements
remote: session=session,
remote: File "/app/.heroku/python/lib/python3.4/site-packages/pip- 6.0.6-py3.4.egg/pip/download.py", line 4
n get_file_content
remote: content = f.read()
remote: File "/app/.heroku/python/lib/python3.4/codecs.py", line 313, in decode
remote: (result, consumed) = self._buffer_decode(data, self.errors, final)
remote: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
remote:
remote:
remote: ! Push rejected, failed to compile Python app
remote:
remote: Verifying deploy...
remote:
remote: ! Push rejected to chsdjangoblog.
remote:
To https://git.heroku.com/chsdjangoblog.git
! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://git.heroku.com/chsdjangoblog.git'
does anyone have any idea why this is occurring? heroku seems nice to use, is there a better alternative/what are the best use cases for heroku? I really just want to solve this issue so I can continue the tutorial. Learning django has been a goal of mine for a while as I’m sick of Word Press and PHP development and have been a long time Python lover.
After that error when I try the next step: heroku ps:scale web=1 i get this output:
Scaling dynos... failed
! App mus tbe deployed before dynos can be scaled.
Thanks in advance.
EDIT:
Here’s my requirements.txt:
Django==1.8
dj-database-url==0.3.0
gunicorn==19.3.0
heroku==0.1.4
python-dateutil==1.5
requests==2.6.0
whitenoise==1.0.6
psycopg2==2.5.4`
I have tried saving as UTF-8, ANSI, UTF-16. Same message for all of them. I even rewrote it without copy paste. Why is my first byte always 0xff regardless of encoding? What is heroku expecting and is there a way/tool to check the bytes in a txt file?
See this answer: Deploying Django/Python 3.4 to Heroku
pip is crashing because your requirements.txt file is encoded wrong. Save it in UTF-8 ANSI encoding.
On the shoulders of the other contributors to this question, it looks like your requirements.txt file is encoded as UTF-16 little endian.
0xFF is the first character of the Byte Order Mark for UTF-16-LE, the second character being 0xFE. The traceback states that the first character is 0xFF in position 0, and it is common in Windows for files to be stored as UTF-16 with the BOM.
Try saving the requirements.txt file as UTF-8 without BOM, or as ASCII. Simple old notepad.exe might do the trick.
Edit
Not working in notepad, so use Python 3 instead:
with open('requirements.txt', encoding='utf-16') as old, open('requirements_new.txt', 'w', encoding='utf-8') as new:
new.write(old.read())
requirements_new.txt will now be encoded as UTF-8 and should work (it will probably end up as ASCII anyway).
Note that this is based on the comments and answers of others which have suggested that the troublesome file is requirements.txt.
I fixed it, the problem seemed to occur only when requirements.txt was generated on windows no matter what encoding I picked. I generated the file in ascii on Linux and it worked. I then transferred the file to windows and it worked there as well. Therefore the problem must be the requirements.txt encoding as was mentioned in comments. However, the correct encoding appears to be ASCII.