How do I compare these two strings in python?
Question:
In crawling RSS
feed, I do not want duplicate items added to my list. The problem is that some duplicates are not detected by my if title not in mylist
line because they are slightly different. Nonetheless, these two news items are basically the same. Take a look at this two.
"Kom igjen, norsk ungdom, de eldre trenger oss!"
and
"Kom igjen norsk ungdom, de eldre trenger oss"
As you see, the first one has comma after Kom igjen
and the second one doesn’t and has an exclamation mark at the end.
Since there is no other unique id that makes individual items unique, I do not know how to detect duplicates like the one above.
Answers:
You can use str.translate
method before you add your news to your list to remover punctuations :
>>> s1.translate(None, string.punctuation)
'Kom igjen norsk ungdom de eldre trenger oss'
In that case you’ll compare your texts based on theirs alphabets.
In python 3 you can do :
>>> s1.translate(dict.fromkeys(map(ord,string.punctuation),None))
'Kom igjen norsk ungdom de eldre trenger oss'
Python has a SequenceMatcher build-in:
from difflib import SequenceMatcher
SequenceMatcher(None, "Hello you!", "Hello you").ratio()
0.9473684210526315
SequenceMatcher(None, "Apple", "Orange").ratio()
0.18181818181818182
So you can loop over all and compare the ratio against some threshold.
In crawling RSS
feed, I do not want duplicate items added to my list. The problem is that some duplicates are not detected by my if title not in mylist
line because they are slightly different. Nonetheless, these two news items are basically the same. Take a look at this two.
"Kom igjen, norsk ungdom, de eldre trenger oss!"
and
"Kom igjen norsk ungdom, de eldre trenger oss"
As you see, the first one has comma after Kom igjen
and the second one doesn’t and has an exclamation mark at the end.
Since there is no other unique id that makes individual items unique, I do not know how to detect duplicates like the one above.
You can use str.translate
method before you add your news to your list to remover punctuations :
>>> s1.translate(None, string.punctuation)
'Kom igjen norsk ungdom de eldre trenger oss'
In that case you’ll compare your texts based on theirs alphabets.
In python 3 you can do :
>>> s1.translate(dict.fromkeys(map(ord,string.punctuation),None))
'Kom igjen norsk ungdom de eldre trenger oss'
Python has a SequenceMatcher build-in:
from difflib import SequenceMatcher
SequenceMatcher(None, "Hello you!", "Hello you").ratio()
0.9473684210526315
SequenceMatcher(None, "Apple", "Orange").ratio()
0.18181818181818182
So you can loop over all and compare the ratio against some threshold.