What are the available tools to summarize or simplify text?
Question:
Is there any library, preferably in python but at least open source, that can summarize and or simplify natural-language text?
Answers:
I’m not sure if there is currently any libraries that do this, as text summarization, or at least understandable text summarization isn’t something that will be easily accomplished by a simple plug & play library.
Here are a few links that I managed to find regarding projects / resources that are related to text summarization to get you started:
- The Lemur Project
- Python Natural Language Toolkit
- O’Reilly’s Book on Natural Language Processing in Python
- Google Resource on Natural Language Processing
- Tutorial : How to create a keyword summary of text in Python
Hope that helps 🙂
Not python but MEAD will do text summarization (it’s in Perl). Usually what comes out is comprehensible, if not always particularly fluent sounding. Also check out summarization.com for a lot of good information on the text summarization task.
Try Open Text Summarizer which is released under the GPL open source license. It works reasonably well but there has been no development work on it since 2007.
The original code is written in C (both a library and a command line utility) but there are wrappers to it in a number of languages:
I needed also the same thing but I couldn’t find anything in Python that helped me have a Comprehensive Result.
So I found this Web Service really useful, and they have a free API which gives a JSON result, and I wanted to share it with you.
Check it out here: http://smmry.com
Maybe you can try sumy. It’s a quite small library that I wrote in Python. There are implemented Luhn’s and Edmundson’s approaches, LSA method, SumBasic, KL-Sum, LexRank and TextRank algorithms. It’s Apache2 licensed and supports Czech, Slovak, English, French, Japanese, Chinese, Portuguese, Spanish and German languages.
Feel free to open an issue or send a pull request if there is something you are missing.
A while back, I wrote a summarization library for python using NLTK, using an algorithm from the Classifier4J library. It’s pretty simple but it may suit the needs of anyone that needs summarization: https://github.com/thavelick/summarize
Take a look at this article which does a detailed study of these methods and packages:
- Lex_rank (sumy)
- LSA (sumy)
- Luhn (sumy)
- PyTeaser
- Gensim TextRank
- PyTextRank
- Google TextSum
The ending of the article does a ‘summary‘.
The author of sumy @miso.belica has given a description in an answer above.
Various other ML techniques have risen, such as Facebook/NAMAS and Google/TextSum but still need extensive training in Gigaword Dataset and about 7000 GPU hours. The dataset itself is quite costly.
In conclusion I would say sumy is the best option in the market right now if you don’t have access to high-end machines. Thanks a lot @miso.belica for this wonderful package.
Is there any library, preferably in python but at least open source, that can summarize and or simplify natural-language text?
I’m not sure if there is currently any libraries that do this, as text summarization, or at least understandable text summarization isn’t something that will be easily accomplished by a simple plug & play library.
Here are a few links that I managed to find regarding projects / resources that are related to text summarization to get you started:
- The Lemur Project
- Python Natural Language Toolkit
- O’Reilly’s Book on Natural Language Processing in Python
- Google Resource on Natural Language Processing
- Tutorial : How to create a keyword summary of text in Python
Hope that helps 🙂
Not python but MEAD will do text summarization (it’s in Perl). Usually what comes out is comprehensible, if not always particularly fluent sounding. Also check out summarization.com for a lot of good information on the text summarization task.
Try Open Text Summarizer which is released under the GPL open source license. It works reasonably well but there has been no development work on it since 2007.
The original code is written in C (both a library and a command line utility) but there are wrappers to it in a number of languages:
I needed also the same thing but I couldn’t find anything in Python that helped me have a Comprehensive Result.
So I found this Web Service really useful, and they have a free API which gives a JSON result, and I wanted to share it with you.
Check it out here: http://smmry.com
Maybe you can try sumy. It’s a quite small library that I wrote in Python. There are implemented Luhn’s and Edmundson’s approaches, LSA method, SumBasic, KL-Sum, LexRank and TextRank algorithms. It’s Apache2 licensed and supports Czech, Slovak, English, French, Japanese, Chinese, Portuguese, Spanish and German languages.
Feel free to open an issue or send a pull request if there is something you are missing.
A while back, I wrote a summarization library for python using NLTK, using an algorithm from the Classifier4J library. It’s pretty simple but it may suit the needs of anyone that needs summarization: https://github.com/thavelick/summarize
Take a look at this article which does a detailed study of these methods and packages:
- Lex_rank (sumy)
- LSA (sumy)
- Luhn (sumy)
- PyTeaser
- Gensim TextRank
- PyTextRank
- Google TextSum
The ending of the article does a ‘summary‘.
The author of sumy @miso.belica has given a description in an answer above.
Various other ML techniques have risen, such as Facebook/NAMAS and Google/TextSum but still need extensive training in Gigaword Dataset and about 7000 GPU hours. The dataset itself is quite costly.
In conclusion I would say sumy is the best option in the market right now if you don’t have access to high-end machines. Thanks a lot @miso.belica for this wonderful package.