Restore original text from Keras’s imdb dataset

This happened because of a basic NLP data preparation. Loads of the so called stop words were removed from text in order to make learning feasible. Usually – also the most of puntuation and less frequent words are removed from text during preprocessing. I think that the only way to restore original text is to find the most matching texts at IMDB using e.g. a Google’s browser API.

You can get the original dataset without stop words removed using get_file from keras.utils.data_utils:

path = get_file('imdb_full.pkl',
               origin='https://s3.amazonaws.com/text-datasets/imdb_full.pkl',
                md5_hash='d091312047c43cf9e4e38fef92437263')
f = open(path, 'rb')
(training_data, training_labels), (test_data, test_labels) = pickle.load(f)

Credit – Jeremy Howards fast.ai course lesson 5

The indices are offset by 3 because 0, 1 and 2 are reserved indices for “padding”, “start of sequence” and “unknown”. The following should work.

imdb = tf.keras.datasets.imdb
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)

word_index = imdb.get_word_index()
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])

review = [reverse_word_index.get(i-3, "?") for i in train_data[0]]

This works for me:

word_index = imdb.get_word_index()                                    
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])            
decoded_review = ' '.join([reverse_word_index.get(i - 3, "") for i in train_data[0]])

This encoding will work along with the labels:

from keras.datasets import imdb
(x_train,y_train),(x_test,y_test) = imdb.load_data()
word_index = imdb.get_word_index() # get {word : index}
index_word = {v : k for k,v in word_index.items()} # get {index : word}

index = 1
print(" ".join([index_word[idx] for idx in x_train[index]]))
print("positve" if y_train[index]==1 else "negetive")

Upvote if helps. 🙂

To get an equivalent array of all the reviews:

def decode_imdb_reviews(text_data):
    result = [0 for x in range(len(text_data))]
    word_index = imdb.get_word_index()
    reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])
    for review in range(0,len(text_data)):
        for index in enumerate(text_data[review]):
            decoded_review = ' '.join([reverse_word_index.get(index - 3, '#') for index in text_data[review]])
        result[review] = decoded_review
    return result

text_data = []
text_data = decode_imdb_reviews(train_data)

from keras.datasets import imdb
NUM_WORDS=1000 # only use top 1000 words

(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=NUM_WORDS)

Get the index details:

word_to_id = keras.datasets.imdb.get_word_index()

Build the Key-Value pair:

id_to_word = {value:key for key,value in word_to_id.items()}
print(' '.join(id_to_word[id] for id in x_train[0] ))

Try below code. this code worked.

# load dataset 
from tensorflow.keras.datasets import imdb 
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=500)


# vec 2 num 
imdb_vec2num = imdb.get_word_index(path="imdb_word_index.json")
# num 2 vec 
imdb_num2vec = {value+3:key for key, value in imdb_vec2num.items()}
imdb_num2vec[0] = "<PAD>"
imdb_num2vec[1] = "<START>"
imdb_num2vec[2] = "<UNK>"
imdb_num2vec[3] = "<UNUSED>"

# index-word table
imdb_num2vec = {value:key for key, value in imdb_index.items()}

# change encoded sentences to sentences
def restore_sentences(num2word, encoded_sentences): 
    sentences = []
    for encoded_sentence in encoded_sentences:
        sentences.append([num2word[ele] for ele in encoded_sentence])
    return sentences

# example 
sentences = restore_sentences(imdb_num2vec, x_train)