Is there a list of characters that look similar to English letters?
Question:
I’m having a crack at profanity filtering for a web forum written in Python.
As part of that, I’m attempting to write a function that takes a word, and returns all possible mock spellings of that word that use visually similar characters in place of specific letters (e.g. s†å©køv€rƒ|øw).
I expect I’ll have to expand this list over time to cover people’s creativity, but is there a list floating around anywhere on the internet that I could use as a starting point?
Answers:
This is probably both vastly more deep than you need, yet not wide enough to cover your use case, but the Unicode consortium have had to deal with attacks against internationalised domain names and came up with this list of homographs (characters with the same or similar rendering):
http://www.unicode.org/Public/security/latest/confusables.txt
Might make a starting point at least.
http://en.wikipedia.org/wiki/Letterlike_Symbols
It’s much much much less comprehensive but is more comprehensible.
I don’t have solution per se, but I have some ideas.
@collapsar’s approach in the comments sounds good to me in principle, but I think you’d want to use an off-the-shelf OCR library rather than try to analyze the images yourself. To make the images, I’d use a font like something in the DejaVu family, because it has good coverage of relatively obscure Unicode characters.
Another easy way to get data is to look at the decompositions of “precomposed” characters like “à”; if a character can be decomposed into one or more combining chapters followed by a base character that looks like an English letter, it probably looks like an English letter itself.
Nothing beats lots of data for a problem like this. You could collect a lot of good examples of character substitutions people have made by scraping the right web forums. Then you can use this procedure to learn new ones: first, find “words” containing mostly characters you can identify, along with some you can’t. Make a regex from the word, converting everything you can to regular letters and replacing everything else with “.”. Then match your regex against a dictionary, and if you get only one match, you have some very good candidates for what the unknown characters are supposed to represent. (I would not actually use a regex for searching a dictionary, but you get the idea.)
Instead of mining forums, you may be able to use Google’s n-gram corpus (http://storage.googleapis.com/books/ngrams/books/datasetsv2.html) instead, but I’m not able to check right now if it contains the kind of pseudo-words you need.
I created a python class to do exactly this, based on Robin’s unicode link for “confusables”
https://github.com/wanderingstan/Confusables
For example, “Hello” would get expanded into the following set of regexp character classes:
[HＨℋℌℍ Η ⲎНᎻᕼꓧ ⱧҢĦӉӇ]
[e℮ｅℯⅇ ꬲеҽɇҿ]
[l|∣⏽￨1۱ IＩⅠℐℑ Ɩｌⅼℓ ǀΙ ⲒІӀⵏᛁꓲ łɭƗƚɫŀĿᒷ ⒈⒓㏫㋋㍤⒔㏬㍥⒕㏭㍦⒖㏮㍧⒗㏯㍨⒘㏰㍩⒙㏱㍪⒚㏲㍫ljIJ‖∥Ⅱǁ ⒒Ⅲ ㏪㋊㍣Ю⒑㏩㋉㍢ʪ₶ⅣⅨɮʫ㏠㋀㍙]
[l|∣⏽￨1۱ IＩⅠℐℑ Ɩｌⅼℓ ǀΙ ⲒІӀⵏᛁꓲ łɭƗƚɫŀĿᒷ ⒈⒓㏫㋋㍤⒔㏬㍥⒕㏭㍦⒖㏮㍧⒗㏯㍨⒘㏰㍩⒙㏱㍪⒚㏲㍫ljIJ‖∥Ⅱǁ ⒒Ⅲ ㏪㋊㍣Ю⒑㏩㋉㍢ʪ₶ⅣⅨɮʫ㏠㋀㍙]
[oంಂംං०੦૦௦౦೦൦๐໐၀۵ｏℴ ᴏᴑꬽο σ ⲟоჿօഠဝ øꬾɵꝋөѳꮎꮻꭴơœɶ∞ꝏꚙൟတ]
This regexp will match against ” ℮ 1೦”
Another source with a few creative options not in the Unicode list:
https://www.i2symbol.com/abc-123
Click a symbol to see the matches.
Mrtop5 Jaliyah424 ReeseDaBest hi Brielle780 TTV_.Drippy Ninja M and subscribe h s j e and man DD Osama from epic games Fortnite creative codes super man iShowspeed Typical Gamer
I’m having a crack at profanity filtering for a web forum written in Python.
As part of that, I’m attempting to write a function that takes a word, and returns all possible mock spellings of that word that use visually similar characters in place of specific letters (e.g. s†å©køv€rƒ|øw).
I expect I’ll have to expand this list over time to cover people’s creativity, but is there a list floating around anywhere on the internet that I could use as a starting point?
This is probably both vastly more deep than you need, yet not wide enough to cover your use case, but the Unicode consortium have had to deal with attacks against internationalised domain names and came up with this list of homographs (characters with the same or similar rendering):
http://www.unicode.org/Public/security/latest/confusables.txt
Might make a starting point at least.
http://en.wikipedia.org/wiki/Letterlike_Symbols
It’s much much much less comprehensive but is more comprehensible.
I don’t have solution per se, but I have some ideas.
@collapsar’s approach in the comments sounds good to me in principle, but I think you’d want to use an off-the-shelf OCR library rather than try to analyze the images yourself. To make the images, I’d use a font like something in the DejaVu family, because it has good coverage of relatively obscure Unicode characters.
Another easy way to get data is to look at the decompositions of “precomposed” characters like “à”; if a character can be decomposed into one or more combining chapters followed by a base character that looks like an English letter, it probably looks like an English letter itself.
Nothing beats lots of data for a problem like this. You could collect a lot of good examples of character substitutions people have made by scraping the right web forums. Then you can use this procedure to learn new ones: first, find “words” containing mostly characters you can identify, along with some you can’t. Make a regex from the word, converting everything you can to regular letters and replacing everything else with “.”. Then match your regex against a dictionary, and if you get only one match, you have some very good candidates for what the unknown characters are supposed to represent. (I would not actually use a regex for searching a dictionary, but you get the idea.)
Instead of mining forums, you may be able to use Google’s n-gram corpus (http://storage.googleapis.com/books/ngrams/books/datasetsv2.html) instead, but I’m not able to check right now if it contains the kind of pseudo-words you need.
I created a python class to do exactly this, based on Robin’s unicode link for “confusables”
https://github.com/wanderingstan/Confusables
For example, “Hello” would get expanded into the following set of regexp character classes:
[HＨℋℌℍ Η ⲎНᎻᕼꓧ ⱧҢĦӉӇ]
[e℮ｅℯⅇ ꬲеҽɇҿ]
[l|∣⏽￨1۱ IＩⅠℐℑ Ɩｌⅼℓ ǀΙ ⲒІӀⵏᛁꓲ łɭƗƚɫŀĿᒷ ⒈⒓㏫㋋㍤⒔㏬㍥⒕㏭㍦⒖㏮㍧⒗㏯㍨⒘㏰㍩⒙㏱㍪⒚㏲㍫ljIJ‖∥Ⅱǁ ⒒Ⅲ ㏪㋊㍣Ю⒑㏩㋉㍢ʪ₶ⅣⅨɮʫ㏠㋀㍙]
[l|∣⏽￨1۱ IＩⅠℐℑ Ɩｌⅼℓ ǀΙ ⲒІӀⵏᛁꓲ łɭƗƚɫŀĿᒷ ⒈⒓㏫㋋㍤⒔㏬㍥⒕㏭㍦⒖㏮㍧⒗㏯㍨⒘㏰㍩⒙㏱㍪⒚㏲㍫ljIJ‖∥Ⅱǁ ⒒Ⅲ ㏪㋊㍣Ю⒑㏩㋉㍢ʪ₶ⅣⅨɮʫ㏠㋀㍙]
[oంಂംං०੦૦௦౦೦൦๐໐၀۵ｏℴ ᴏᴑꬽο σ ⲟоჿօഠဝ øꬾɵꝋөѳꮎꮻꭴơœɶ∞ꝏꚙൟတ]
This regexp will match against ” ℮ 1೦”
Another source with a few creative options not in the Unicode list:
https://www.i2symbol.com/abc-123
Click a symbol to see the matches.
Mrtop5 Jaliyah424 ReeseDaBest hi Brielle780 TTV_.Drippy Ninja M and subscribe h s j e and man DD Osama from epic games Fortnite creative codes super man iShowspeed Typical Gamer