How to check whether a sentence is correct (simple grammar check in Python)?
Question:
How to check whether a sentence is valid in Python?
Examples:
I love Stackoverflow - Correct
I Stackoverflow love - Incorrect
Answers:
Check out NLTK. They have support for grammars that you can use to parse your sentence. You can define a grammar, or use one that is provided, along with a context-free parser. If the sentence parses, then it has valid grammar; if not, then it doesn’t. These grammars may not have the widest coverage (eg, it might not know how to handle a word like StackOverflow), but this approach will allow you to say specifically what is valid or invalid in the grammar. Chapter 8 of the NLTK book covers parsing and should explain what you need to know.
An alternative would be to write a python interface to a wide-coverage parser (like the Stanford parser or C&C). These are statistical parsers that will be able to understand sentences even if they haven’t seen all the words or all the grammatical constructions before. The downside is that sometimes the parser will still return a parse for a sentence with bad grammar because it will use the statistics to make the best guess possible.
So, it really depends on exactly what your goal is. If you want very precise control over what is considered grammatical, use a context-free parser with NLTK. If you want robustness and wide-coverage, use a statistical parser.
There are various Web Services providing automated proofreading and grammar checking. Some have a Python library to simplify querying.
As far as I can tell, most of those tools (certainly After the Deadline and LanguageTool) are rule based. The checked text is compared with a large set of rules describing common errors. If a rule matches, the software calls it an error. If a rule does not match, the software does nothing (it cannot detect errors it does not have rules for).
After the Deadline
import ATD
ATD.setDefaultKey("your API key")
errors = ATD.checkDocument("Looking too the water. Fixing your writing typoss.")
for error in errors:
print "%s error for: %s **%s**" % (error.type, error.precontext, error.string)
print "some suggestions: %s" % (", ".join(error.suggestions),)
Expected output:
grammar error for: Looking **too the**
some suggestions: to the
spelling error for: writing **typoss**
some suggestions: typos
It is possible to run the server application on your own machine, 4 GB RAM is recommended.
LanguageTool
https://pypi.python.org/pypi/language-check
>>> import language_check
>>> tool = language_check.LanguageTool('en-US')
>>> text = 'A sentence with a error in the Hitchhiker’s Guide tot he Galaxy'
>>> matches = tool.check(text)
>>> matches[0].fromy, matches[0].fromx
(0, 16)
>>> matches[0].ruleId, matches[0].replacements
('EN_A_VS_AN', ['an'])
>>> matches[1].fromy, matches[1].fromx
(0, 50)
>>> matches[1].ruleId, matches[1].replacements
('TOT_HE', ['to the'])
>>> print(matches[1])
Line 1, column 51, Rule ID: TOT_HE[1]
Message: Did you mean 'to the'?
Suggestion: to the
...
>>> language_check.correct(text, matches)
'A sentence with an error in the Hitchhiker’s Guide to the Galaxy'
It is also possible to run the server side privately.
Ginger
Additionally, this is a hacky (screen scraping) library for Ginger, arguably one of the most polished free-to-use grammar checking options out there.
Microsoft Word
It should be possible to script Microsoft Word and use its grammar checking functionality.
More
There is a curated list of grammar checkers on Open Office website. Noted in comments by Patrick.
Some other answers have mentioned LanguageTool, the largest open-source grammar checker. It didn’t have a reliable, up-to-date Python port until now.
I recommend language_tool_python, a grammar checker that supports Python 3 and the latest versions of Java and LanguageTool. It’s the only up-to-date, free Python grammar checker. (full disclosure, I made this library)
I would suggest the language-tool-python. For example:
import language_tool_python
tool = language_tool_python.LanguageTool('en-US')
text = "Your the best but their are allso good !"
matches = tool.check(text)
len(matches)
and we get:
4
We can have a look at the 4 issues that it found:
1st Issue:
matches[0]
And we get:
Match({'ruleId': 'YOUR_YOU_RE', 'message': 'Did you mean "You're"?', 'replacements': ["You're"], 'context': 'Your the best but their are allso good !', 'offset': 0, 'errorLength': 4, 'category': 'TYPOS', 'ruleIssueType': 'misspelling'})
2nd Issue:
matches[1]
and we get:
Match({'ruleId': 'THEIR_IS', 'message': 'Did you mean "there"?', 'replacements': ['there'], 'context': 'Your the best but their are allso good !', 'offset': 18, 'errorLength': 5, 'category': 'CONFUSED_WORDS', 'ruleIssueType': 'misspelling'})
3rd Issue:
matches[2]
and we get:
Match({'ruleId': 'MORFOLOGIK_RULE_EN_US', 'message': 'Possible spelling mistake found.', 'replacements': ['also', 'all so'], 'context': 'Your the best but their are allso good !', 'offset': 28, 'errorLength': 5, 'category': 'TYPOS', 'ruleIssueType': 'misspelling'})
4th Issue:
matches[3]
and we get:
Match({'ruleId': 'WHITESPACE_RULE', 'message': 'Possible typo: you repeated a whitespace', 'replacements': [' '], 'context': 'Your the best but their are allso good!', 'offset': 33, 'errorLength': 2, 'category': 'TYPOGRAPHY', 'ruleIssueType': 'whitespace'})
If you are looking for a more detailed example you can have a look at the related post of Predictive Hacks
Step 1
pip install Caribe
Step 2
import Caribe as cb
sentence="I is playing football"
output=cb.caribe_corrector(sentence)
print(output)
How to check whether a sentence is valid in Python?
Examples:
I love Stackoverflow - Correct
I Stackoverflow love - Incorrect
Check out NLTK. They have support for grammars that you can use to parse your sentence. You can define a grammar, or use one that is provided, along with a context-free parser. If the sentence parses, then it has valid grammar; if not, then it doesn’t. These grammars may not have the widest coverage (eg, it might not know how to handle a word like StackOverflow), but this approach will allow you to say specifically what is valid or invalid in the grammar. Chapter 8 of the NLTK book covers parsing and should explain what you need to know.
An alternative would be to write a python interface to a wide-coverage parser (like the Stanford parser or C&C). These are statistical parsers that will be able to understand sentences even if they haven’t seen all the words or all the grammatical constructions before. The downside is that sometimes the parser will still return a parse for a sentence with bad grammar because it will use the statistics to make the best guess possible.
So, it really depends on exactly what your goal is. If you want very precise control over what is considered grammatical, use a context-free parser with NLTK. If you want robustness and wide-coverage, use a statistical parser.
There are various Web Services providing automated proofreading and grammar checking. Some have a Python library to simplify querying.
As far as I can tell, most of those tools (certainly After the Deadline and LanguageTool) are rule based. The checked text is compared with a large set of rules describing common errors. If a rule matches, the software calls it an error. If a rule does not match, the software does nothing (it cannot detect errors it does not have rules for).
After the Deadline
import ATD
ATD.setDefaultKey("your API key")
errors = ATD.checkDocument("Looking too the water. Fixing your writing typoss.")
for error in errors:
print "%s error for: %s **%s**" % (error.type, error.precontext, error.string)
print "some suggestions: %s" % (", ".join(error.suggestions),)
Expected output:
grammar error for: Looking **too the**
some suggestions: to the
spelling error for: writing **typoss**
some suggestions: typos
It is possible to run the server application on your own machine, 4 GB RAM is recommended.
LanguageTool
https://pypi.python.org/pypi/language-check
>>> import language_check
>>> tool = language_check.LanguageTool('en-US')
>>> text = 'A sentence with a error in the Hitchhiker’s Guide tot he Galaxy'
>>> matches = tool.check(text)
>>> matches[0].fromy, matches[0].fromx
(0, 16)
>>> matches[0].ruleId, matches[0].replacements
('EN_A_VS_AN', ['an'])
>>> matches[1].fromy, matches[1].fromx
(0, 50)
>>> matches[1].ruleId, matches[1].replacements
('TOT_HE', ['to the'])
>>> print(matches[1])
Line 1, column 51, Rule ID: TOT_HE[1]
Message: Did you mean 'to the'?
Suggestion: to the
...
>>> language_check.correct(text, matches)
'A sentence with an error in the Hitchhiker’s Guide to the Galaxy'
It is also possible to run the server side privately.
Ginger
Additionally, this is a hacky (screen scraping) library for Ginger, arguably one of the most polished free-to-use grammar checking options out there.
Microsoft Word
It should be possible to script Microsoft Word and use its grammar checking functionality.
More
There is a curated list of grammar checkers on Open Office website. Noted in comments by Patrick.
Some other answers have mentioned LanguageTool, the largest open-source grammar checker. It didn’t have a reliable, up-to-date Python port until now.
I recommend language_tool_python, a grammar checker that supports Python 3 and the latest versions of Java and LanguageTool. It’s the only up-to-date, free Python grammar checker. (full disclosure, I made this library)
I would suggest the language-tool-python. For example:
import language_tool_python
tool = language_tool_python.LanguageTool('en-US')
text = "Your the best but their are allso good !"
matches = tool.check(text)
len(matches)
and we get:
4
We can have a look at the 4 issues that it found:
1st Issue:
matches[0]
And we get:
Match({'ruleId': 'YOUR_YOU_RE', 'message': 'Did you mean "You're"?', 'replacements': ["You're"], 'context': 'Your the best but their are allso good !', 'offset': 0, 'errorLength': 4, 'category': 'TYPOS', 'ruleIssueType': 'misspelling'})
2nd Issue:
matches[1]
and we get:
Match({'ruleId': 'THEIR_IS', 'message': 'Did you mean "there"?', 'replacements': ['there'], 'context': 'Your the best but their are allso good !', 'offset': 18, 'errorLength': 5, 'category': 'CONFUSED_WORDS', 'ruleIssueType': 'misspelling'})
3rd Issue:
matches[2]
and we get:
Match({'ruleId': 'MORFOLOGIK_RULE_EN_US', 'message': 'Possible spelling mistake found.', 'replacements': ['also', 'all so'], 'context': 'Your the best but their are allso good !', 'offset': 28, 'errorLength': 5, 'category': 'TYPOS', 'ruleIssueType': 'misspelling'})
4th Issue:
matches[3]
and we get:
Match({'ruleId': 'WHITESPACE_RULE', 'message': 'Possible typo: you repeated a whitespace', 'replacements': [' '], 'context': 'Your the best but their are allso good!', 'offset': 33, 'errorLength': 2, 'category': 'TYPOGRAPHY', 'ruleIssueType': 'whitespace'})
If you are looking for a more detailed example you can have a look at the related post of Predictive Hacks
Step 1
pip install Caribe
Step 2
import Caribe as cb
sentence="I is playing football"
output=cb.caribe_corrector(sentence)
print(output)