This was a small experiment on how to detect icons from a screenshot. I used OpenCV and Matplotlib to explore and experiment around. OpenCV for template matching and Matplotlib for drawing the figures. The task was as straightforward as this post 😀
The problem was to detect the incognito mask from Firefox starting page. Figure 1 shows the icon that the program was looking for. The icon is a 2 dimensional image with 3 channels. The icon was in PNG-format.
Figure 1. The icon.
The search image was scanned to find the coordinates for the missing icon. The search image is shown in figure 2.
It is really easy for a human to detect the icon. Yet, it is hard to describe what steps were taken during the detection.
Computers have it the other way around. It is hard to detect anything, but simple to say what the program did during the detection. The program in my case went through these steps:
- Load the search image
- Load the icon (or template)
- Slide the icon over the image (2 dimensional convolution)
- Find the highest match (“Template Match” below)
- Extract coordinates for the highest match
- Continue with your application. In this case I drew a white rectangle around it “Detected Area”.
In figure 3 you can see the results for the detection. Lighter area in the “Template Match” subplot in figure 3 indicates a higher match with the icon (step 3). You can clearly see a lighter area at the top-left corner of the search image. It means that at the icon had the highest match around there. “Detected Area” subplot in figure 3 shows the drawn rectangle (step 6).
Detecting icons from screenshots is suspiciously easy since the icon can find an exact match from the search image. Notice that the icon is not only a close match, it fits with the search image pixel by pixel. Many real-life applications with natural images do not have this luxury.