Android NDK/Camera2 从 `ImageReader` 表面/帧处理创建 OpenGL 表面

Question

我正在尝试创建一个从相机流式传输帧的应用程序，使用 Skia 在帧上绘制一个红色框，最后将其显示在屏幕上（预览）。

我已经设法设置 Camera2 并将其连接到显示相机预览的

SurfaceView

，但我不确定如何正确设置中间帧处理部分。

我试图创建一个

ImageReader

（Java）：

// create image reader
imageReader = ImageReader.newInstance(width, height, ImageFormat.YUV_420_888, 3)
imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireLatestImage()
  Log.d(TAG, "Frame: ${image.width} x ${image.height}")
  nativeOnDraw(image)
  image.close()
}, null)

从中创建一个 OpenGL/Skia 表面（C++）：

EGLDisplay eglDisplay = eglGetDisplay(EGL_DEFAULT_DISPLAY);
if (eglDisplay == EGL_NO_DISPLAY) {
    throw std::runtime_error("Failed to get EGL Display! " + std::to_string(glGetError()));
}

EGLint major, minor;
if (!eglInitialize(eglDisplay, &major, &minor)) {
    throw std::runtime_error("Failed to initialize EGL! " + std::to_string(glGetError()));
}

EGLint att[] = {EGL_RENDERABLE_TYPE, EGL_OPENGL_ES2_BIT,
                EGL_SURFACE_TYPE, EGL_WINDOW_BIT,
                EGL_RECORDABLE_ANDROID, 1,
                EGL_ALPHA_SIZE, 8,
                EGL_BLUE_SIZE, 8,
                EGL_GREEN_SIZE, 8,
                EGL_RED_SIZE, 8,
                EGL_NONE};

EGLint numConfigs;
EGLConfig eglConfig;
if (!eglChooseConfig(eglDisplay, att, &eglConfig, 1, &numConfigs) ||
    numConfigs == 0) {
    throw std::runtime_error("Failed to choose a GL Config! " + std::to_string(glGetError()));
}

EGLint contextAttribs[] = {EGL_CONTEXT_CLIENT_VERSION, 2, EGL_NONE};
EGLContext eglContext = eglCreateContext(eglDisplay,
                                         eglConfig,
                                         EGL_NO_CONTEXT,
                                         contextAttribs);
if (eglContext == EGL_NO_CONTEXT) {
    throw std::runtime_error("Failed to create a GL Context! " + std::to_string(glGetError()));
}

EGLSurface eglSurface = eglCreateWindowSurface(eglDisplay, eglConfig, nativeWindow, nullptr);
if (!eglMakeCurrent(eglDisplay, eglSurface, eglSurface, eglContext)) {
    throw std::runtime_error("Failed to set current surface! " + std::to_string(glGetError()));
}

GLint buffer;
glGetIntegerv(GL_FRAMEBUFFER_BINDING, &buffer);
GLint stencil;
glGetIntegerv(GL_STENCIL_BITS, &stencil);
GLint samples;
glGetIntegerv(GL_SAMPLES, &samples);

// Create the Skia backend context
auto backendInterface = GrGLMakeNativeInterface();
auto grContext = GrDirectContext::MakeGL(backendInterface);
if (grContext == nullptr) {
    throw std::runtime_error("Failed to create Skia Context from GL Context! " + std::to_string(glGetError()));
}

auto maxSamples = grContext->maxSurfaceSampleCountForColorType(kRGBA_8888_SkColorType);
if (samples > maxSamples)
    samples = maxSamples;

GrGLFramebufferInfo fbInfo;
fbInfo.fFBOID = buffer;
fbInfo.fFormat = 0x8058; // GR_GL_RGBA8

auto renderTarget = GrBackendRenderTarget(width, height, samples, stencil, fbInfo);

struct RenderContext {
    EGLDisplay display;
    EGLSurface surface;
};
auto ctx = new RenderContext({eglDisplay, eglSurface});

auto surface = SkSurface::MakeFromBackendRenderTarget(
        grContext.get(), renderTarget, kBottomLeft_GrSurfaceOrigin,
        kRGBA_8888_SkColorType, nullptr, nullptr,
        [](void *addr) {
            auto ctx = reinterpret_cast<RenderContext *>(addr);
            eglDestroySurface(ctx->display, ctx->surface);
            delete ctx;
        },
        reinterpret_cast<void *>(ctx));

// `surface` can now be used for drawing

..最后将它传递给相机（Java）：

val captureRequestBuilder = captureSession.device.createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)
captureRequestBuilder.addTarget(imageReader.surface)
val captureRequest = captureRequestBuilder.build()
captureSession.setRepeatingRequest(captureRequest, null, cameraHandler)

但这只会因以下错误而崩溃：

D  eglCreateContext: 0xb40000735bdf0790: maj 3 min 0 rcv 3
D  eglMakeCurrent: 0xb40000735bdf0790: ver 3 0 (tinfo 0x75819f0100) (first time)
I  Re-rendering camera page with active camera. Device: "back (0)" (1280x720 @ 30fps)
D  onSurfaceCreated()
D  onSurfaceChanged()
D  Camera 0 successfully opened
E  [ImageReader-640x480f23m3-1403-0](id:57b00000002,api:1,p:1403,c:1403) connect: already connected (cur=1 req=4)
W  Stream configuration failed due to: endConfigure:648: Camera 0: Unsupported set of inputs/outputs provided
E  Session 0: Failed to create capture session; configuration failed

令我震惊的是

ImageReader ... already connected

错误 - 是否无法从现有的

ImageReader

表面创建 OpenGL/Skia 表面？

是否有另一种方法可以做到这一点，而不需要每帧图像的昂贵的 CPU 缓冲区副本？

Answer 1

我不清楚你在这里想做什么，但你是在使用

Surface

中的

ImageReader.getSurface

来创建

eglSurface

吗？很好，但这意味着您正在设置

GPU -> ImageReader

.

的渲染路径

然后您尝试将相机连接到 ImageReader (

Camera -> ImageReader

)，由于您已经从 GPU 为 ImageReader 提供数据，所以会出错。

If you want

Camera -> GPU

, use a SurfaceTexture to turn camera frames into GL textures (create a

Surface

from it via a constructor).

但也许您需要澄清您要创建的管道是什么。