使用 FFMPEG 解码碎片化的 HEVC 切片仅解码帧的 1/3

Question

我在解码来自实时 555 rtsp 服务器的 hevc 流时遇到问题，对于一些较新的相机，帧被编码在不同的切片中。根据 NAL 值，它发送 I 帧，然后是 P 帧，然后是未知帧。 我试图遵循这个SO线程的逻辑，试图理解逻辑。

下面是我对镜框类型的分类

int CH265Parser::CheckH265IorP(unsigned char* pData, unsigned long dwLen)
{
    int naltype = H265GetNALType(pData, dwLen);

    int frameType = 0;
    switch (naltype)
    {
    case NAL_VPS:
    case NAL_SPS:
    case NAL_PPS:
    {
        break;
    }
        
    case NAL_BLA_W_LP:
    case NAL_BLA_W_RADL:
    case NAL_BLA_N_LP:
    case NAL_IDR_W_RADL:
    case NAL_IDR_N_LP:
    case NAL_CRA_NUT:
    {
        // I-Frame
        frameType = 1;
        break;
    }
    case NAL_TRAIL_N:
    case NAL_TRAIL_R:
    case NAL_TSA_N:
    case NAL_TSA_R:
    case NAL_STSA_N:
    case NAL_STSA_R:
    case NAL_RADL_N:
    case NAL_RADL_R:
    case NAL_RASL_N:
    case NAL_RASL_R:
    {
        // P-Frame
        frameType = 2;
        break;
    }
    case NAL_AUD:               
    case NAL_SEI_SUFFIX:
    case NAL_SEI_PREFIX:
    {
        break;
    }

    default:
        break;
    }

    return frameType;
}

这就是我如何获得 NAL

int CH265Parser::H265GetNALType(void* pBSBuf, long nBSLen)
{
    if (nBSLen < 5)
        return -1;

    int pos = FindNALBegin((unsigned char*)pBSBuf, nBSLen);
    unsigned char* pBS = (unsigned char*)pBSBuf;
    unsigned long nType = (pBS[pos] >> 1) & 0x3F;
    if (nType >= NAL_TRAIL_N && nType <= NAL_SEI_SUFFIX)
        return nType;
    return -1;
}


long CH265Parser::FindNALBegin(unsigned char* pszBuffer, long nLength)
{
    for (int i = 0; i < nLength - 4; i++)
    {
        if (pszBuffer[i] == 0x00 && pszBuffer[i + 1] == 0x00)
        {
            if (pszBuffer[i + 2] == 0x01)
                return i + 3;
            else if (pszBuffer[i + 2] == 0x00 && pszBuffer[i + 3] == 0x01)
                return i + 4;
        }
    }
    return -1;
}

现在根据信息，这是我的解码器的样子（缓冲区中有额外的信息头）：

auto headSize = sizeof(isap::media::AV_HEADER);
    auto header = reinterpret_cast<const isap::media::AV_HEADER*>(packets_.data());
     std::string frameType = "I";
    if (header->nFrameType == isap::media::frame_type::_FRAME_TYPE_P)
        frameType = "P";
    else if (header->nFrameType == isap::media::frame_type::_FRAME_TYPE_UNKNOWN)
        frameType = "B";
    printf("frame %s\n", frameType.c_str());

    if (frameType == "P" || frameType == "B") {
        packets_ = packets_ + packet;
    }
    else if (frameType == "I") {
        packets_ = packet;
    }


    auto ec = decoder_.send(reinterpret_cast<const uint8_t*>(packets_.data()), static_cast<int32_t>(packets_.size()));
   
    if (ec) {
        
        error_ = "media_client::decode_video send failed: ";
        kt::error_message_to(std::back_inserter(error_), ec);
//        throw stop_signal();
    }
    else {
        packets_ = "";
        for (;;) {
                auto frame = av_frame_alloc();
                ec = decoder_.receive(frame);
                if (ec) {
                    av_frame_free(&frame);
                    if (ec.value() == AVERROR(EAGAIN)) break;

                    error_ = "media_client::decode_video receive failed: ";
                    kt::error_message_to(std::back_inserter(error_), ec);
                    throw stop_signal();
                }

                push_frame_queue(frame);
            }
    }

最终结果

请注意，上面的代码适用于具有较低分辨率或非碎片流的 hevc，并且解码器本身适用于 h264 流。我是否正确连接了切片？我对这个主题很陌生。

hevc 碎片流的输出

frame I
frame I
[hevc @ 0000025A9CD08400] PPS changed between slices.
[hevc @ 0000025A9CD08400] Error parsing NAL unit #3.
frame I
[hevc @ 0000025A9CD08400] PPS changed between slices.
[hevc @ 0000025A9CD08400] Error parsing NAL unit #3.
frame I
frame B
frame P
[hevc @ 0000025A9CD08400] First slice in a frame missing.
frame P
[hevc @ 0000025A9CD08400] First slice in a frame missing.
width: 1920, height: 1080, size: 8294400  // this only returns 1/3 of the frame successfully decoded

hevc 非碎片流的输出

frame I
frame I
[hevc @ 00000185A06EBA40] PPS id out of range: 0
[hevc @ 00000185A06EBA40] Error parsing NAL unit #0.
frame P
[hevc @ 00000185A06EBA40] PPS id out of range: 0
[hevc @ 00000185A06EBA40] Error parsing NAL unit #0.
width: 1920, height: 1080, size: 8294400
frame P

h264 输出，无错误/警告

frame I
frame I
frame P
width: 1280, height: 1280, size: 6553600
frame P
width: 1280, height: 1280, size: 6553600
frame P
width: 1280, height: 1280, size: 6553600
frame P
width: 1280, height: 1280, size: 6553600

任何帮助将不胜感激。

Answer 1

在发送到解码器之前，你必须合并切片，在切片之间添加4个字节并将值设置为0001，并将其发送到解码器。

你可以按照这个伪代码

size_t data_pos = 4;
memmove(rtp_data + data_pos, rtp_data, frameSize);
rtp_data[0]=0;rtp_data[1]=0;rtp_data[2]=0;rtp_data[3]=1;
frameSize += data_pos;

if (firstFU){
                memset(temp_rtp_data, 0, BUFFER_SIZE);
                memcpy(temp_rtp_data, rtp_data, frameSize);
                temp_buffer_size = frameSize;
                firstFU = false;
}
else{
                
                memcpy(temp_rtp_data + temp_buffer_size, rtp_data, frameSize);
                temp_buffer_size += frameSize;
                firstFU = false;
}

如果您发现另一个前缀，即在我的例子中 nal_type 等于 39，请将 temp_rtp_data 发送到解码器，希望它有所帮助。