Getting Killed error in aws but works in Mac
Question:
When I try to run the following pickle program in aws ubuntu instance, I am getting ‘killed’ message and it is not generating pickle. However the same works when I try it in my local machine (Mac). I am using python3 to run the program:
import nltk
import random
from nltk.classify.scikitlearn import SklearnClassifier
import pickle
from nltk.tokenize import word_tokenize
documents_f = open("documents.pickle", "rb")
documents = pickle.load(documents_f)
documents_f.close()
word_features5k_f = open("word_features5k.pickle", "rb")
word_features = pickle.load(word_features5k_f)
word_features5k_f.close()
def find_features(document):
words = word_tokenize(document)
features = {}
for w in word_features:
features[w] = (w in words)
#print (features)
return features
featuresets = [(find_features(rev), category) for (rev, category) in documents]
save_featuresets = open("featuresets.pickle","wb")
pickle.dump(featuresets, save_featuresets)
save_featuresets.close()
I believe it might be due to memory issue, because I am using aws free tire. Someone please let me know how to fix this?
Issue:
$ python3 my_pickle.py
Killed
Answers:
Your EC2 instance is running out of memory, and your process is being killed by the OOM killer.
Upgrade to a larger instance type, or find a way to make your process use less memory. (For example, you may want to consider processing one document at a time instead of loading them all at once.)
You can check the error you’ve encountered on the ec2 instance using the following command :
$ sudo dmesg -e
Most probably it will be ‘out of memory’ error.
To solve this issue instead of upgrading your ec2 instance computing resources you can try the following fix.
You can enable paging by allocating swap space on your instance. Enable swap by using the following command :
$ sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
$ sudo /sbin/mkswap /var/swap.1
$ sudo chmod 600 /var/swap.1
$ sudo /sbin/swapon /var/swap.1
The above commands will allocate a swap space of 1GB. If you want more space, I recommend allocating 4096(4GB).
One important thing to note is, swap is not enabled by default after every reboot. Which means if you restart your ec2 instance, the swap space you previously allocated will vanish.
To fix this, we enable swap by default after every instance reboot.
sudo chmod 777 /etc/fstab
nano /etc/fstab
Check if the below line is present in the file :
/var/swap.1 swap swap defaults 0 0
If not, add it. Save write changes to the file and rerun your python code. Insert print statements in between to check how your code proceeds during execution.
When I try to run the following pickle program in aws ubuntu instance, I am getting ‘killed’ message and it is not generating pickle. However the same works when I try it in my local machine (Mac). I am using python3 to run the program:
import nltk
import random
from nltk.classify.scikitlearn import SklearnClassifier
import pickle
from nltk.tokenize import word_tokenize
documents_f = open("documents.pickle", "rb")
documents = pickle.load(documents_f)
documents_f.close()
word_features5k_f = open("word_features5k.pickle", "rb")
word_features = pickle.load(word_features5k_f)
word_features5k_f.close()
def find_features(document):
words = word_tokenize(document)
features = {}
for w in word_features:
features[w] = (w in words)
#print (features)
return features
featuresets = [(find_features(rev), category) for (rev, category) in documents]
save_featuresets = open("featuresets.pickle","wb")
pickle.dump(featuresets, save_featuresets)
save_featuresets.close()
I believe it might be due to memory issue, because I am using aws free tire. Someone please let me know how to fix this?
Issue:
$ python3 my_pickle.py
Killed
Your EC2 instance is running out of memory, and your process is being killed by the OOM killer.
Upgrade to a larger instance type, or find a way to make your process use less memory. (For example, you may want to consider processing one document at a time instead of loading them all at once.)
You can check the error you’ve encountered on the ec2 instance using the following command :
$ sudo dmesg -e
Most probably it will be ‘out of memory’ error.
To solve this issue instead of upgrading your ec2 instance computing resources you can try the following fix.
You can enable paging by allocating swap space on your instance. Enable swap by using the following command :
$ sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
$ sudo /sbin/mkswap /var/swap.1
$ sudo chmod 600 /var/swap.1
$ sudo /sbin/swapon /var/swap.1
The above commands will allocate a swap space of 1GB. If you want more space, I recommend allocating 4096(4GB).
One important thing to note is, swap is not enabled by default after every reboot. Which means if you restart your ec2 instance, the swap space you previously allocated will vanish.
To fix this, we enable swap by default after every instance reboot.
sudo chmod 777 /etc/fstab
nano /etc/fstab
Check if the below line is present in the file :
/var/swap.1 swap swap defaults 0 0
If not, add it. Save write changes to the file and rerun your python code. Insert print statements in between to check how your code proceeds during execution.