Scikit Learn TfidfVectorizer : How to get top n terms with highest tf-idf score

Question

I am working on keyword extraction problem. Consider the very general case

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(tokenizer=tokenize, stop_words='english')

t = """Two Travellers, walking in the noonday sun, sought the shade of a widespreading tree to rest. As they lay looking up among the pleasant leaves, they saw that it was a Plane Tree.

"How useless is the Plane!" said one of them. "It bears no fruit whatever, and only serves to litter the ground with leaves."

"Ungrateful creatures!" said a voice from the Plane Tree. "You lie here in my cooling shade, and yet you say I am useless! Thus ungratefully, O Jupiter, do men receive their blessings!"

Our best blessings are often the least appreciated."""

tfs = tfidf.fit_transform(t.split(" "))
str = 'tree cat travellers fruit jupiter'
response = tfidf.transform([str])
feature_names = tfidf.get_feature_names()

for col in response.nonzero()[1]:
    print(feature_names[col], ' - ', response[0, col])

and this gives me

  (0, 28)   0.443509712811
  (0, 27)   0.517461475101
  (0, 8)    0.517461475101
  (0, 6)    0.517461475101
tree  -  0.443509712811
travellers  -  0.517461475101
jupiter  -  0.517461475101
fruit  -  0.517461475101

which is good. For any new document that comes in, is there a way to get the top n terms with the highest tfidf score?

Asked By: AbtPst

||

Source

Answer 1

You have to do a little bit of a song and dance to get the matrices as numpy arrays instead, but this should do what you’re looking for:

feature_array = np.array(tfidf.get_feature_names())
tfidf_sorting = np.argsort(response.toarray()).flatten()[::-1]

n = 3
top_n = feature_array[tfidf_sorting][:n]

This gives me:

array([u'fruit', u'travellers', u'jupiter'], 
  dtype='<U13')

The argsort call is really the useful one, here are the docs for it. We have to do [::-1] because argsort only supports sorting small to large. We call flatten to reduce the dimensions to 1d so that the sorted indices can be used to index the 1d feature array. Note that including the call to flatten will only work if you’re testing one document at at time.

Also, on another note, did you mean something like tfs = tfidf.fit_transform(t.split("nn"))? Otherwise, each term in the multiline string is being treated as a “document”. Using nn instead means that we are actually looking at 4 documents (one for each line), which makes more sense when you think about tfidf.

Answered By: hume

Answer 2

Solution using sparse matrix itself (without .toarray())!

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer(stop_words='english')
corpus = [
    'I would like to check this document',
    'How about one more document',
    'Aim is to capture the key words from the corpus',
    'frequency of words in a document is called term frequency'
]

X = tfidf.fit_transform(corpus)
feature_names = np.array(tfidf.get_feature_names())


new_doc = ['can key words in this new document be identified?',
           'idf is the inverse document frequency caculcated for each of the words']
responses = tfidf.transform(new_doc)


def get_top_tf_idf_words(response, top_n=2):
    sorted_nzs = np.argsort(response.data)[:-(top_n+1):-1]
    return feature_names[response.indices[sorted_nzs]]
  
print([get_top_tf_idf_words(response,2) for response in responses])

#[array(['key', 'words'], dtype='<U9'),
 array(['frequency', 'words'], dtype='<U9')]

Answered By: Venkatachalam

Answer 3

Here is a quick code for that:
(documents is a list)

def get_tfidf_top_features(documents,n_top=10):
  tfidf_vectorizer = TfidfVectorizer(max_df=0.95, min_df=2,  stop_words='english')
  tfidf = tfidf_vectorizer.fit_transform(documents)
  importance = np.argsort(np.asarray(tfidf.sum(axis=0)).ravel())[::-1]
  tfidf_feature_names = np.array(tfidf_vectorizer.get_feature_names())
  return tfidf_feature_names[importance[:n_top]]

Answered By: Sahar Millis

Scikit Learn TfidfVectorizer : How to get top n terms with highest tf-idf score

Question:

Answers: