Python Pytesseract not detecting strings on image
Question:
Hi I have a python code with tesseract, the goal is to detect strings from screenshot. Code:
import pytesseract
import cv2
import pyautogui
import numpy as np
pytesseract.pytesseract.tesseract_cmd = r'C:Program FilesTesseract-OCRtesseract.exe'
image = pyautogui.screenshot()
image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)
cv2.imwrite("imagesgameScreenshot.png", image)
img = cv2.imread('imagesgameScreenshot.png')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
print(pytesseract.image_to_string(img))
cv2.imshow('Result', img)
cv2.waitKey(0)
I want it to print out, detect string like "Enemy, Enemy, Enemy", (don’t ask what for okay :D)
But instead it prints out some garbage. Ex:
[|=Li]
2
fyi
TT fi)
938/100
10720
The image i display as a result on the end looks like this: 
How do I go about fixing this?
Answers:
Tesseract works on black and white image. By default on image of black text on white background. The first stage of tesseract is to binarize text, if it is not already binarized. You have to help it to do so. Either binarize yourself. Or, at least, providing and image with text as black as possible, and rest as white as possible.
For example, here, your text seems to be perfect red (255,0,0) (it appears blue in your example, because you mix up RGB2BGR somewhere. But sky color makes obvious that it is red in reality).
So, try to find a formula that makes red color 0, and other color 255. With some smooth transition in between.
For example (just my first shot, it is certainly improvable. Plus, there are many smarter method, providing your own binarization that you could use)
im2=(255-img[:,:,2].clip(232,255) + img[:,:,0].clip(0,20) + img[:,:,1].clip(0,20))*3
Puts in black=0 pixels (255,0,0), and in white=255 those that are not (255,0,0), with some variation in between (for example, (250, 5, 5) is (5+5+5)*3=45, so quite dark, but not black)
And, on this picture
pytesseract.image_to_string(im2)
#" nn nn nnEnemynEnemyn—n8 of Enemyn‘a -n- .n» ' >n-. in“a ®, ownn nx0c"
Not perfect, far from that. But at least, you see that you have your "Enemy Enemy Enemy" among some noise.
It will never be perfect anyway. There is no miracle.
My point is just to show that to start working, you need a black/white image, with black text over white background.
You could certainly improve the way to build that black&white image to exclude more noise.
to_data
Besides all this, image_to_string is made for good old linear, top to bottom, left to right, linear text. This is not your case here.
For this kind of images, with scattered pieces of text, I would use image_to_data instead. That gives a list of text, their coordinate, confidence factor, and even some hierarchical organization (in pages, blocks, lines,…).
print(pytesseract.image_to_data(im2))
shows
level page_num block_num par_num line_num word_num left top width height conf text
1 1 0 0 0 0 0 0 1533 890 -1
2 1 1 0 0 0 69 43 4 57 -1
3 1 1 1 0 0 69 43 4 57 -1
4 1 1 1 1 0 69 43 4 57 -1
5 1 1 1 1 1 69 43 4 57 95
2 1 2 0 0 0 10 39 391 66 -1
3 1 2 1 0 0 10 39 391 66 -1
4 1 2 1 1 0 10 39 391 66 -1
5 1 2 1 1 1 10 39 391 66 95
2 1 3 0 0 0 1440 276 62 172 -1
3 1 3 1 0 0 1440 276 62 172 -1
4 1 3 1 1 0 1440 276 62 172 -1
5 1 3 1 1 1 1440 276 62 172 95
2 1 4 0 0 0 16 255 694 288 -1
3 1 4 1 0 0 16 255 694 288 -1
4 1 4 1 1 0 470 255 60 15 -1
5 1 4 1 1 1 470 255 60 15 95 Enemy
4 1 4 1 2 0 286 352 94 23 -1
5 1 4 1 2 1 286 352 94 23 95 Enemy
4 1 4 1 3 0 295 381 75 9 -1
5 1 4 1 3 1 295 381 75 9 58 —
4 1 4 1 4 0 22 400 688 30 -1
5 1 4 1 4 1 88 409 9 10 0 8
5 1 4 1 4 2 285 401 22 29 26 of
5 1 4 1 4 3 647 400 63 16 67 Enemy
4 1 4 1 5 0 49 420 271 15 -1
5 1 4 1 5 1 49 423 16 12 13 ‘a
5 1 4 1 5 2 306 420 14 11 0 -
4 1 4 1 6 0 105 451 198 14 -1
5 1 4 1 6 1 105 451 6 6 15 -
5 1 4 1 6 2 289 456 14 9 5 .
4 1 4 1 7 0 170 490 47 8 -1
5 1 4 1 7 1 170 490 4 5 30 »
5 1 4 1 7 2 194 475 6 34 0 '
5 1 4 1 7 3 212 490 5 8 23 >
4 1 4 1 8 0 19 509 38 13 -1
5 1 4 1 8 1 19 509 38 13 0 -.
5 1 4 1 8 2 187 504 14 8 15 i
4 1 4 1 9 0 16 534 232 9 -1
5 1 4 1 9 1 16 535 12 8 19 “a
5 1 4 1 9 2 196 534 4 6 0 ®,
5 1 4 1 9 3 228 516 18 34 42 ow
2 1 5 0 0 0 392 487 1141 403 -1
3 1 5 1 0 0 392 487 1141 403 -1
4 1 5 1 1 0 392 487 1141 403 -1
5 1 5 1 1 1 392 487 1141 403 95
See that lines with conf -1 are empty lines used only for hiearchy structure. And lines with conf<50% are trash. So let’s parse (with some split) those data, and filter out the lines with less than 50% confidence factor.
# Split lines (separated by n), and in each of them fields, separated by t
# Ignore first (header) and last (empty) lines
lines=[s.split('t') for s in pytesseract.image_to_data(im2).split('n')[1:-1]]
# Keeps only the one that have confidence factor>50
# and that are not just blank
[l[-1] for l in lines if int(l[-2])>50 and l[-1].strip()]
And result is… tadaa
['Enemy', 'Enemy', '—', 'Enemy']
Not, that bad, if I may say myself. Considering the very artisanal "first shoot" black&whitization.
Note that the — has a confidence factor 58, when the worst "Enemy" has 67, so I could have chosen a threshold of 60 instead of 50. But I don’t want to cheat and adjust thresholds retroactively 😀
Also, note that I kept only text here, but each "Enemy" comes with coordinates. You requested that we don’t ask why you need to find "Enemy, Enemy, Enemy". so I didn’t ask. But for most usage I can think of, being able to know the position is a plus, I guess.
Hi I have a python code with tesseract, the goal is to detect strings from screenshot. Code:
import pytesseract
import cv2
import pyautogui
import numpy as np
pytesseract.pytesseract.tesseract_cmd = r'C:Program FilesTesseract-OCRtesseract.exe'
image = pyautogui.screenshot()
image = cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR)
cv2.imwrite("imagesgameScreenshot.png", image)
img = cv2.imread('imagesgameScreenshot.png')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
print(pytesseract.image_to_string(img))
cv2.imshow('Result', img)
cv2.waitKey(0)
I want it to print out, detect string like "Enemy, Enemy, Enemy", (don’t ask what for okay :D)
But instead it prints out some garbage. Ex:
[|=Li]
2
fyi
TT fi)
938/100
10720
The image i display as a result on the end looks like this:
How do I go about fixing this?
Tesseract works on black and white image. By default on image of black text on white background. The first stage of tesseract is to binarize text, if it is not already binarized. You have to help it to do so. Either binarize yourself. Or, at least, providing and image with text as black as possible, and rest as white as possible.
For example, here, your text seems to be perfect red (255,0,0) (it appears blue in your example, because you mix up RGB2BGR somewhere. But sky color makes obvious that it is red in reality).
So, try to find a formula that makes red color 0, and other color 255. With some smooth transition in between.
For example (just my first shot, it is certainly improvable. Plus, there are many smarter method, providing your own binarization that you could use)
im2=(255-img[:,:,2].clip(232,255) + img[:,:,0].clip(0,20) + img[:,:,1].clip(0,20))*3
Puts in black=0 pixels (255,0,0), and in white=255 those that are not (255,0,0), with some variation in between (for example, (250, 5, 5) is (5+5+5)*3=45, so quite dark, but not black)
And, on this picture
pytesseract.image_to_string(im2)
#" nn nn nnEnemynEnemyn—n8 of Enemyn‘a -n- .n» ' >n-. in“a ®, ownn nx0c"
Not perfect, far from that. But at least, you see that you have your "Enemy Enemy Enemy" among some noise.
It will never be perfect anyway. There is no miracle.
My point is just to show that to start working, you need a black/white image, with black text over white background.
You could certainly improve the way to build that black&white image to exclude more noise.
to_data
Besides all this, image_to_string is made for good old linear, top to bottom, left to right, linear text. This is not your case here.
For this kind of images, with scattered pieces of text, I would use image_to_data instead. That gives a list of text, their coordinate, confidence factor, and even some hierarchical organization (in pages, blocks, lines,…).
print(pytesseract.image_to_data(im2))
shows
level page_num block_num par_num line_num word_num left top width height conf text
1 1 0 0 0 0 0 0 1533 890 -1
2 1 1 0 0 0 69 43 4 57 -1
3 1 1 1 0 0 69 43 4 57 -1
4 1 1 1 1 0 69 43 4 57 -1
5 1 1 1 1 1 69 43 4 57 95
2 1 2 0 0 0 10 39 391 66 -1
3 1 2 1 0 0 10 39 391 66 -1
4 1 2 1 1 0 10 39 391 66 -1
5 1 2 1 1 1 10 39 391 66 95
2 1 3 0 0 0 1440 276 62 172 -1
3 1 3 1 0 0 1440 276 62 172 -1
4 1 3 1 1 0 1440 276 62 172 -1
5 1 3 1 1 1 1440 276 62 172 95
2 1 4 0 0 0 16 255 694 288 -1
3 1 4 1 0 0 16 255 694 288 -1
4 1 4 1 1 0 470 255 60 15 -1
5 1 4 1 1 1 470 255 60 15 95 Enemy
4 1 4 1 2 0 286 352 94 23 -1
5 1 4 1 2 1 286 352 94 23 95 Enemy
4 1 4 1 3 0 295 381 75 9 -1
5 1 4 1 3 1 295 381 75 9 58 —
4 1 4 1 4 0 22 400 688 30 -1
5 1 4 1 4 1 88 409 9 10 0 8
5 1 4 1 4 2 285 401 22 29 26 of
5 1 4 1 4 3 647 400 63 16 67 Enemy
4 1 4 1 5 0 49 420 271 15 -1
5 1 4 1 5 1 49 423 16 12 13 ‘a
5 1 4 1 5 2 306 420 14 11 0 -
4 1 4 1 6 0 105 451 198 14 -1
5 1 4 1 6 1 105 451 6 6 15 -
5 1 4 1 6 2 289 456 14 9 5 .
4 1 4 1 7 0 170 490 47 8 -1
5 1 4 1 7 1 170 490 4 5 30 »
5 1 4 1 7 2 194 475 6 34 0 '
5 1 4 1 7 3 212 490 5 8 23 >
4 1 4 1 8 0 19 509 38 13 -1
5 1 4 1 8 1 19 509 38 13 0 -.
5 1 4 1 8 2 187 504 14 8 15 i
4 1 4 1 9 0 16 534 232 9 -1
5 1 4 1 9 1 16 535 12 8 19 “a
5 1 4 1 9 2 196 534 4 6 0 ®,
5 1 4 1 9 3 228 516 18 34 42 ow
2 1 5 0 0 0 392 487 1141 403 -1
3 1 5 1 0 0 392 487 1141 403 -1
4 1 5 1 1 0 392 487 1141 403 -1
5 1 5 1 1 1 392 487 1141 403 95
See that lines with conf -1 are empty lines used only for hiearchy structure. And lines with conf<50% are trash. So let’s parse (with some split) those data, and filter out the lines with less than 50% confidence factor.
# Split lines (separated by n), and in each of them fields, separated by t
# Ignore first (header) and last (empty) lines
lines=[s.split('t') for s in pytesseract.image_to_data(im2).split('n')[1:-1]]
# Keeps only the one that have confidence factor>50
# and that are not just blank
[l[-1] for l in lines if int(l[-2])>50 and l[-1].strip()]
And result is… tadaa
['Enemy', 'Enemy', '—', 'Enemy']
Not, that bad, if I may say myself. Considering the very artisanal "first shoot" black&whitization.
Note that the — has a confidence factor 58, when the worst "Enemy" has 67, so I could have chosen a threshold of 60 instead of 50. But I don’t want to cheat and adjust thresholds retroactively 😀
Also, note that I kept only text here, but each "Enemy" comes with coordinates. You requested that we don’t ask why you need to find "Enemy, Enemy, Enemy". so I didn’t ask. But for most usage I can think of, being able to know the position is a plus, I guess.