我如何检测特定区域的亮度

Question

我想制作一个机器人来自动玩吉他英雄类型的游戏，但是所有音符都是相同的颜色。这些音符明亮且引人注目，因此我认为找到两点之间的平均亮度将是检测音符的最佳方法。我愿意接受有关如何检测这些注释的新建议，并且我们将不胜感激。我想用 python 来做这个项目，因为它是我所知道的唯一的编码语言之一。然而，检测颜色也可以工作，因为音符会很快，而且决定它是否是音符的时间不会超过一毫秒

我已经尝试用谷歌搜索我的答案，但是似乎没有我应该使用的特定库，并且关于这个主题的信誉良好的来源并不多。

Answer 1

我设置了一些代码，应该可以帮助您开始做您想做的事情。我试图保持依赖关系简单，但它确实需要 mss 来进行屏幕截图，PIL 来进行图像操作，以及 numpy 来处理某些数字。

简而言之：

• 截取屏幕截图并校准按钮所在位置
• 使用这些位置沿着每个彩色按钮绘制线条
• 每帧读取每条线的强度
• 解释强度向量以确定何时发出输入来玩游戏

我没有摇滚乐队，所以我查找了某人玩它的 YouTube 视频来获取一些视频，所以它应该与你的游戏非常接近，信用在底部。

这是模型：

from PIL import Image, ImageDraw
import matplotlib.pyplot as plt
import mss
import numpy as np

def get_screenshot(sct: mss.base.MSSBase, monitor_dict: dict) -> Image:
    sct_img = sct.grab(monitor_dict)
    pil_img = Image.frombytes('RGB', sct_img.size, sct_img.bgra, 'raw', 'BGRX')
    return pil_img

def setup(img: Image, num_lines: int, lines: list[tuple[int, int, int, int]] = None):
    # If lines was not already provided, ask the user to provide the info to generate them
    # This is the calibration
    if lines is None:
        plt.imshow(img)
        plt.title('Select corners of bar, starting with bottom left going clockwise')
        points = plt.ginput(4, timeout=-1)
        plt.close()
        lower_line = int((points[0][1] + points[3][1]) / 2)
        upper_line = int((points[1][1] + points[2][1]) / 2)
        lower_space = (points[3][0] - points[0][0]) / num_lines
        upper_space = (points[2][0] - points[1][0]) / num_lines
        lines = [
            (int(points[0][0] + (i + 0.5) * lower_space), lower_line, int(points[1][0] + (i + 0.5) * upper_space), upper_line)
            for i in range(num_lines)
        ]
    # Draw the image with lines on it to let user verify good calibration
    draw = ImageDraw.Draw(img)
    for line in lines:
        draw.line(line, fill=128, width=3)
    plt.imshow(img)
    plt.title('Verify that lines align with screen, abort if not.')
    plt.show()
    return lines


def get_raster_lines(img, lines_expanded):
    # This function is set up to be pretty amenable to a numba improvement if necessary
    raster_lines = np.empty(shape=lines_expanded.shape[:2], dtype=float)
    for line_num in range(lines_expanded.shape[0]):
        for point_num in range(lines_expanded.shape[1]):
            raster_lines[line_num, point_num] = np.linalg.norm(img[lines_expanded[line_num, point_num][1], lines_expanded[line_num, point_num][0]])
    return raster_lines


def _main():
    # How many guitar lines, seems like 5 based on youtube videos
    num_lines = 5
    # The colours to make plots more understandable
    colour_order = ['green', 'red', 'yellow', 'blue', 'orange']
    # Set up the screen capturing library
    sct = mss.mss()
    monitor = sct.monitors[2]
    monitor['mon'] = 1
    img = get_screenshot(sct, monitor)
    # Run this the first time with lines set to None, then after you get the output, you can put it here
    lines = [(540, 799, 675, 470), (639, 799, 707, 470), (738, 799, 739, 470), (838, 799, 771, 470), (937, 799, 803, 470)]
    lines = setup(img, num_lines, lines)
    # lines = setup(img, num_lines)
    # lines follows: [(x0, y0, x1, y1), ...]
    #   [(bottom left), (top left), (top right), (bottom right)]
    print('lines: ', lines) # print so you can put it above and not have to calibrate every time
    # Generate all the points along the lines
    lines_expanded = np.array([
        [(int(np.interp(yval, (line[3], line[1]), (line[2], line[0]))), yval) for yval in range(line[3], line[1])]
        for line in lines
    ])
    # Wait to start so that you can set up the game or any other initial setup
    input('Press Enter when ready to start')
    print('Starting ...')
    # While true so that it will run for the whole game
    while True:
        try:
            # Take the screenshot
            img = get_screenshot(sct, monitor)
            # Pull out the raster lines
            raster_lines = get_raster_lines(np.array(img), lines_expanded)
            # Process raster_lines to generate your commands
            # In this case just plot to show that the information was captured
            for i in range(num_lines):
                plt.plot(raster_lines[i], color=colour_order[i])
            plt.legend([f'bar {colour_order[i]}' for i in range(num_lines)])
            plt.show()
        except KeyboardInterrupt:
            break
        break # comment this out to continue running

if __name__ == '__main__':
    _main()

当我运行时，我得到：

lines:  [(540, 799, 675, 470), (639, 799, 707, 470), (738, 799, 739, 470), (838, 799, 771, 470), (937, 799, 803, 470)]
Press Enter when ready to start
Starting ...

Process finished with exit code 0

这是我大约使用的校准点：

这是校准图，您可以在其中看到红线笔直延伸到每个彩色关键通道：

这是它生成的光栅线：

我在这里对其进行了注释以显示一些有趣的功能：

这应该足以让你继续前进！

我测试的视频归功于此链接： https://www.youtube.com/watch?v=TIs9x8MfROk