我试图将使用Camera 2 API从Image收到的ImageReader转换为OpenCV矩阵，并使用CameraBridgeViewBase将其显示在屏幕上，更具体地说是deliverAndDrawFrame函数。读者的ImageFormat是YUV_420_888，据我所知，它有一个Y平面，每个像素都有灰度值，U平面每隔4个像素有一个U / V。但是，当我尝试显示此图像时，看起来好像图像正在重复并旋转90度。下面的代码应该将YUV数据放入OpenCV矩阵（现在只是灰度，而不是rgba）：
/**
* Takes an {@link Image} in the {@link ImageFormat#YUV_420_888} and puts it into a provided {@link Mat} in rgba format.
*
* @param yuvImage {@link Image} in the {@link ImageFormat#YUV_420_888} format.
*/
public static void yuv420888imageToRgbaMat(final Image yuvImage, final Mat rgbaMat) {
final Image.Plane
Yp = yuvImage.getPlanes()[0],
UandVp = yuvImage.getPlanes()[1];
final ByteBuffer
Ybb = Yp .getBuffer(),
UandVbb = UandVp.getBuffer();
Ybb .get(mYdata , 0, 480*640 );
UandVbb.get(mUandVData, 0, 480*640 / 2 - 8);
for (int i = 0; i < 640*480; i++) {
for (int j = 0; j < 4; j++) {
mRawRGBAFrameData[i + 640*480*j] = mYdata[i];
}
mRawRGBAFrameData[i*4 ] = mYdata[i];
mRawRGBAFrameData[i*4+1] = mYdata[i];
mRawRGBAFrameData[i*4+2] = mYdata[i];
mRawRGBAFrameData[i*4+3] = -1;
}
}
这是我的OpenCV框架的代码：
private class CameraFrame implements CvCameraViewFrame {
private Mat mRgba;
@Override
public Mat gray() {
return null;
}
@Override
public Mat rgba() {
mRgbaMat.put(0, 0, mRawRGBAFrameData);
return mRgba;
}
public CameraFrame(final Mat rgba) {
super();
mRgba = rgba;
}
}
接收绘制框架的代码：
private final ImageReader.OnImageAvailableListener mOnImageAvailableListener = new ImageReader.OnImageAvailableListener() {
@Override
public void onImageAvailable(ImageReader reader) {
final Image yuvImage = reader.acquireLatestImage();
yuv420888imageToRgbaMat(yuvImage, mRgbaMat);
deliverAndDrawFrame(mFrame);
yuvImage.close();
}
};
而且，这是制作图像阅读器的代码：
mRgbaMat = new Mat(mFrameHeight, mFrameWidth, CvType.CV_8UC4);
mFrame = new CameraFrame(mRgbaMat);
mImageReader = ImageReader.newInstance(mFrameWidth, mFrameHeight, ImageFormat.YUV_420_888, 1);
mImageReader.setOnImageAvailableListener(mOnImageAvailableListener, mBackgroundHandler);
AllocateCache();
这是数组的初始化：
protected static byte[] mRawRGBAFrameData = new byte[640*480*4], mYdata = new byte[640*480], mUandVData = new byte[640*480 / 2];
注意：mFrameWidth是480，mFrameHeight是640.奇怪的是，ImageReader和从它收到的Image的高度和宽度都有倒置的尺寸。
这是上面代码的图像：https://i.stack.imgur.com/lcdzf.png
这是与yuv420888imageToRgbaMat https://i.stack.imgur.com/T2MOI.png相反的图像
for (int i = 0; i < 640*480; i++) {
mRawRGBAFrameData[i] = mYdata[i];
}
我们可以看到数据在Y帧中重复，并且由于某种原因，这给出了实际的好看的图像。
对于任何尝试使用Camera 2 API使用OpenCV同样问题的人，我想出了一个解决方案。我发现的第一件事是ByteBuffer中有ImageReader提供填充的事实，所以如果你不考虑它会导致输出失真。我选择做的另一件事是创建我自己的SurfaceView并使用Bitmap而不是使用CameraViewBase绘制它，到目前为止它已经很好了。 OpenCV有一个函数Util.matToBitmap，它接受一个BGR矩阵并将其转换为android Bitmap，因此这很有用。我通过将来自Image.Plane提供的前两个ImageReaders的信息放入一个格式为YUV 420的OpenCV单通道矩阵，并使用Imgproc.cvtColor和Imgproc.COLOR_YUV420p2BGR来获得BGR矩阵。重要的是要知道图像的Y平面具有全像素，但第二UV平面具有映射一到四个Y像素的交错像素，因此UV平面的总长度是Y平面的一半。见here。无论如何，这里有一些代码：
矩阵的初始化
m_BGRMat = new Mat(Constants.VISION_IMAGE_HEIGHT, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC3);
m_Yuv420FrameMat = new Mat(Constants.VISION_IMAGE_HEIGHT * 3 / 2, Constants.VISION_IMAGE_WIDTH, CvType.CV_8UC1);
每一帧：
// Convert image to YUV 420 matrix
ImageUtils.imageToMat(image, m_Yuv420FrameMat, m_RawFrameData, m_RawFrameRowData);
// Convert YUV matrix to BGR matrix
Imgproc.cvtColor(m_Yuv420FrameMat, m_BGRMat, Imgproc.COLOR_YUV420p2BGR);
// Flip width and height then mirror vertically
Core.transpose(m_BGRMat, m_BGRMat);
Core.flip(m_BGRMat, m_BGRMat, 0);
// Draw to Surface View
m_PreviewView.drawImageMat(m_BGRMat);
这是转换为YUV 420矩阵：
/**
* Takes an Android {@link Image} in the {@link ImageFormat#YUV_420_888} format and returns an OpenCV {@link Mat}.
*
* @param image {@link Image} in the {@link ImageFormat#YUV_420_888} format
*/
public static void imageToMat(final Image image, final Mat mat, byte[] data, byte[] rowData) {
ByteBuffer buffer;
int rowStride, pixelStride, width = image.getWidth(), height = image.getHeight(), offset = 0;
Image.Plane[] planes = image.getPlanes();
if (data == null || data.length != width * height) data = new byte[width * height * ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8];
if (rowData == null || rowData.length != planes[0].getRowStride()) rowData = new byte[planes[0].getRowStride()];
for (int i = 0; i < planes.length; i++) {
buffer = planes[i].getBuffer();
rowStride = planes[i].getRowStride();
pixelStride = planes[i].getPixelStride();
int
w = (i == 0) ? width : width / 2,
h = (i == 0) ? height : height / 2;
for (int row = 0; row < h; row++) {
int bytesPerPixel = ImageFormat.getBitsPerPixel(ImageFormat.YUV_420_888) / 8;
if (pixelStride == bytesPerPixel) {
int length = w * bytesPerPixel;
buffer.get(data, offset, length);
// Advance buffer the remainder of the row stride, unless on the last row.
// Otherwise, this will throw an IllegalArgumentException because the buffer
// doesn't include the last padding.
if (h - row != 1)
buffer.position(buffer.position() + rowStride - length);
offset += length;
} else {
// On the last row only read the width of the image minus the pixel stride
// plus one. Otherwise, this will throw a BufferUnderflowException because the
// buffer doesn't include the last padding.
if (h - row == 1)
buffer.get(rowData, 0, width - pixelStride + 1);
else
buffer.get(rowData, 0, rowStride);
for (int col = 0; col < w; col++)
data[offset++] = rowData[col * pixelStride];
}
}
}
mat.put(0, 0, data);
}
最后，画画
/**
* Given an {@link Mat} that represents a BGR image, draw it on the surface canvas.
* use the OpenCV helper function {@link Utils#matToBitmap(Mat, Bitmap)} to create a {@link Bitmap}.
*
* @param bgrMat BGR frame {@link Mat}
*/
public void drawImageMat(final Mat bgrMat) {
if (m_HolderReady) {
// Create bitmap from BGR matrix
Utils.matToBitmap(bgrMat, m_Bitmap);
// Obtain the canvas and draw the bitmap on top of it
final SurfaceHolder holder = getHolder();
final Canvas canvas = holder.lockCanvas();
canvas.drawBitmap(m_Bitmap, null, new Rect(0, 0, m_HolderWidth, m_HolderHeight), null);
holder.unlockCanvasAndPost(canvas);
}
}
这种方式有效，但我想最好的方法是设置一个OpenGL渲染上下文并编写一些简单的着色器来显示矩阵。