# Scikit Learn TfidfVectorizer : How to get top n terms with highest tf-idf score

## Question:

I am working on keyword extraction problem. Consider the very general case

``````from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english')

t = """Two Travellers, walking in the noonday sun, sought the shade of a widespreading tree to rest. As they lay looking up among the pleasant leaves, they saw that it was a Plane Tree.

"How useless is the Plane!" said one of them. "It bears no fruit whatever, and only serves to litter the ground with leaves."

"Ungrateful creatures!" said a voice from the Plane Tree. "You lie here in my cooling shade, and yet you say I am useless! Thus ungratefully, O Jupiter, do men receive their blessings!"

Our best blessings are often the least appreciated."""

tfs = tfidf.fit_transform(t.split(" "))
str = 'tree cat travellers fruit jupiter'
response = tfidf.transform([str])
feature_names = tfidf.get_feature_names()

for col in response.nonzero()[1]:
print(feature_names[col], ' - ', response[0, col])
``````

and this gives me

``````  (0, 28)   0.443509712811
(0, 27)   0.517461475101
(0, 8)    0.517461475101
(0, 6)    0.517461475101
tree  -  0.443509712811
travellers  -  0.517461475101
jupiter  -  0.517461475101
fruit  -  0.517461475101
``````

which is good. For any new document that comes in, is there a way to get the top n terms with the highest tfidf score?

You have to do a little bit of a song and dance to get the matrices as numpy arrays instead, but this should do what you’re looking for:

``````feature_array = np.array(tfidf.get_feature_names())
tfidf_sorting = np.argsort(response.toarray()).flatten()[::-1]

n = 3
top_n = feature_array[tfidf_sorting][:n]
``````

This gives me:

``````array([u'fruit', u'travellers', u'jupiter'],
dtype='<U13')
``````

The `argsort` call is really the useful one, here are the docs for it. We have to do `[::-1]` because `argsort` only supports sorting small to large. We call `flatten` to reduce the dimensions to 1d so that the sorted indices can be used to index the 1d feature array. Note that including the call to `flatten` will only work if you’re testing one document at at time.

Also, on another note, did you mean something like `tfs = tfidf.fit_transform(t.split("nn"))`? Otherwise, each term in the multiline string is being treated as a “document”. Using `nn` instead means that we are actually looking at 4 documents (one for each line), which makes more sense when you think about tfidf.

Solution using sparse matrix itself (without `.toarray()`)!

``````import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english')
corpus = [
'I would like to check this document',
'How about one more document',
'Aim is to capture the key words from the corpus',
'frequency of words in a document is called term frequency'
]

X = tfidf.fit_transform(corpus)
feature_names = np.array(tfidf.get_feature_names())

new_doc = ['can key words in this new document be identified?',
'idf is the inverse document frequency caculcated for each of the words']
responses = tfidf.transform(new_doc)

def get_top_tf_idf_words(response, top_n=2):
sorted_nzs = np.argsort(response.data)[:-(top_n+1):-1]
return feature_names[response.indices[sorted_nzs]]

print([get_top_tf_idf_words(response,2) for response in responses])

#[array(['key', 'words'], dtype='<U9'),
array(['frequency', 'words'], dtype='<U9')]
``````

Here is a quick code for that:
(`documents` is a list)

``````def get_tfidf_top_features(documents,n_top=10):
tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2,  stop_words='english')
tfidf = tfidf_vectorizer.fit_transform(documents)
importance = np.argsort(np.asarray(tfidf.sum(axis=0)).ravel())[::-1]
tfidf_feature_names = np.array(tfidf_vectorizer.get_feature_names())
return tfidf_feature_names[importance[:n_top]]
``````