matchTemplate() 缺少检测并给出误报，我该怎么办？

Question

我正在尝试使用 opencv 来检测视频游戏中的对象。我已经将图像抓取为 png 并对其进行了修剪，以便背景是透明的，但是，它仅在阈值水平 ~ 0.6 处检测到它。

针图：

干草堆图片：

这是使用 0.8 的阈值完成的。注意检测中误报的数量以及真阴性的数量。

这是Python代码。

import cv2 as cv
import numpy as np


class Vision:

    # properties
    needle_img = None
    needle_w = 0
    needle_h = 0
    method = None

    # constructor
    def __init__(self, needle_img_path, method=cv.TM_CCOEFF_NORMED):
        # load the image we're trying to match
        self.needle_img = cv.imread(needle_img_path, cv.IMREAD_UNCHANGED)

        # Save the dimensions of the needle image
        self.needle_w = self.needle_img.shape[1]
        self.needle_h = self.needle_img.shape[0]

        # There are 6 methods to choose from:
        # TM_CCOEFF, TM_CCOEFF_NORMED, TM_CCORR, TM_CCORR_NORMED, TM_SQDIFF, TM_SQDIFF_NORMED
        self.method = method

    def find(self, haystack_img, threshold=0.9, debug_mode=None):
        # run the OpenCV algorithm
        base = self.needle_img[:, :, 0:3]
        alpha = self.needle_img[:, :, 3]
        alpha = cv.merge([alpha, alpha, alpha])
        result = cv.matchTemplate(haystack_img, base, self.method,mask=alpha)

        # Get the all the positions from the match result that exceed our threshold
        locations = np.where(result >= threshold)
        locations = list(zip(*locations[::-1]))
        #print(locations)

        # First we need to create the list of [x, y, w, h] rectangles
        rectangles = []
        for loc in locations:
            rect = [int(loc[0]), int(loc[1]), self.needle_w, self.needle_h]
            # Add every box to the list twice in order to retain single (non-overlapping) boxes
            rectangles.append(rect)
            rectangles.append(rect)
        # Apply group rectangles
        # "Relative difference between sides of the rectangles to merge them into a group."
        rectangles, weights = cv.groupRectangles(rectangles, groupThreshold=1, eps=0.5)
        #print(rectangles)

        points = []
        if len(rectangles):
            #print('Found needle.')

            line_color = (0, 255, 0)
            line_type = cv.LINE_4
            marker_color = (255, 0, 255)
            marker_type = cv.MARKER_CROSS

            # Loop over all the rectangles
            for (x, y, w, h) in rectangles:

                # Determine the center position
                center_x = x + int(w/2)
                center_y = y + int(h/2)
                # Save the points
                points.append((center_x, center_y))

                if debug_mode == 'rectangles':
                    # Determine the box position
                    top_left = (x, y)
                    bottom_right = (x + w, y + h)
                    # Draw the box
                    cv.rectangle(haystack_img, top_left, bottom_right, color=line_color,
                                lineType=line_type, thickness=2)
                elif debug_mode == 'points':
                    # Draw the center point
                    cv.drawMarker(haystack_img, (center_x, center_y),
                                color=marker_color, markerType=marker_type,
                                markerSize=40, thickness=2)

        if debug_mode:
            cv.imshow('Matches', haystack_img)
            #cv.waitKey()
            #cv.imwrite('result_click_point.jpg', haystack_img)

        return points

我还使用 wincap 实时捕获屏幕，但我认为根本问题在于图像检测。如果我将针图像作为我希望它检测的精确像素完美图像提供，为什么它不能在高阈值下正确检测它？

Answer 1

您的模板与大海捞针中的实例并不“完美”匹配。左：您的模板。右：干草堆的一部分，我擦除了周围的环境。忽略边缘像素，查看对象内部的像素。

既然我们已经确定这些数据不可能完美匹配，我希望您能理解，您需要给程序一些宽容。

现在来看看误报：

发生这些情况是因为您为此数据选择了

可怕

的匹配模式，TM_CCOEFF_NORMED。

当大海捞针中的实例是针的像素完美副本时，您应该使用

TM_SQDIFF

或

TM_SQDIFF_NORMED

。当实例略有差异但它们通常具有相同的亮度和颜色时也是如此。这是使用

TM_SQDIFF_NORMED

的结果，带有源自针的

mask

参数，并接受 0.2 的差异。实例差异为0.192。