如何将 RTP（IP 摄像头）流式传输到 React 应用程序设置中

Question

我正在尝试将来自 IP 摄像机的“实时广播”或来自 RTP/RTSP 源的任何其他广播传输到我的 REACT 应用程序。但必须是直播 我目前的设置是：

IP 摄像头 -> (RTP) -> FFmpeg -> (udp) -> 服务器(nodeJs) -> (WebRTC) -> React 应用程序

在目前的情况下，几乎没有延迟，但是这里有些事情我无法避免，我无法理解为什么，这是我的问题：

1)

首先，设置是否正确，这是在 Web 应用程序中流式传输 RTP 视频的唯一方法吗？

2)

是否可以避免重新编码流，RTP传输必然采用H.264，因此我实际上不需要执行以下命令：

    return spawn('ffmpeg', [
    '-re',                              // Read input at its native frame rate Important for live-streaming
    '-probesize', '32',                 // Set probing size to 32 bytes (32 is minimum)
    '-analyzeduration', '1000000',      // An input duration of 1 second
    '-c:v', 'h264',                     // Video codec of input video
    '-i', 'rtp://238.0.0.2:48888',      // Input stream URL
    '-map', '0:v?',                     // Select video from input stream
    '-c:v', 'libx264',                  // Video codec of output stream
    '-preset', 'ultrafast',             // Faster encoding for lower latency
    '-tune', 'zerolatency',             // Optimize for zero latency
    // '-s', '768x480',                    // Adjust the resolution (experiment with values)
    '-f', 'rtp', `rtp://127.0.0.1:${udpPort}` // Output stream URL
]);

正如你可以在这个命令中看到的那样，我重新编码为 libx264，但是如果我将 FFMPEG 设置为参数 '-c:v' :'copy' 而不是 '-c:v', 'libx264' 那么 FFMPEG 会抛出一个错误：它不知道如何编码 h264，只知道什么是 libx264-> 

基本上，我想停止重新编码，因为确实不需要它，因为流已经编码为 H264。有什么可以推荐的吗？

3)

我想过完全放弃 FFMPEG，但是当 WEBRTC 限制为最大 1280 BYTE 时，RTP 数据包的大小达到 1200+ BYTES。 有没有一种方法可以在不损坏视频的情况下管理这些破坏行为，并且可以进入这个世界？我想这里有抖动缓冲区的全部故事 这是我的服务器端代码

（这只是一个测试代码）

import {
    MediaStreamTrack,
    randomPort,
    RTCPeerConnection,
    RTCRtpCodecParameters,
    RtpPacket,
} from 'werift'
import {Server} from "ws";
import {createSocket} from "dgram";
import {spawn} from "child_process";
import LoggerFactory from "./logger/loggerFactory";

//

const log = LoggerFactory.getLogger('ServerMedia')

// Websocket server -> WebRTC
const serverPort = 8888
const server = new Server({port: serverPort});
log.info(`Server Media start om port: ${serverPort}`);

// UDP server -> ffmpeg
const udpPort = 48888
const udp = createSocket("udp4");
// udp.bind(udpPort, () => {
//     udp.addMembership("238.0.0.2");
// })
udp.bind(udpPort)
log.info(`UDP port: ${udpPort}`)


const createFFmpegProcess = () => {
    log.info(`Start ffmpeg process`)
    return spawn('ffmpeg', [
        '-re',                              // Read input at its native frame rate Important for live-streaming
        '-probesize', '32',                 // Set probing size to 32 bytes (32 is minimum)
        '-analyzeduration', '1000000',      // An input duration of 1 second
        '-c:v', 'h264',                     // Video codec of input video
        '-i', 'rtp://238.0.0.2:48888',      // Input stream URL
        '-map', '0:v?',                     // Select video from input stream
        '-c:v', 'libx264',                  // Video codec of output stream
        '-preset', 'ultrafast',             // Faster encoding for lower latency
        '-tune', 'zerolatency',             // Optimize for zero latency
        // '-s', '768x480',                    // Adjust the resolution (experiment with values)
        '-f', 'rtp', `rtp://127.0.0.1:${udpPort}` // Output stream URL
    ]);

}

let ffmpegProcess = createFFmpegProcess();


const attachFFmpegListeners = () => {
    // Capture standard output and print it
    ffmpegProcess.stdout.on('data', (data) => {
        log.info(`FFMPEG process stdout: ${data}`);
    });

    // Capture standard error and print it
    ffmpegProcess.stderr.on('data', (data) => {
        console.error(`ffmpeg stderr: ${data}`);
    });

    // Listen for the exit event
    ffmpegProcess.on('exit', (code, signal) => {
        if (code !== null) {
            log.info(`ffmpeg process exited with code ${code}`);
        } else if (signal !== null) {
            log.info(`ffmpeg process killed with signal ${signal}`);
        }
    });
};


attachFFmpegListeners();


server.on("connection", async (socket) => {
    const payloadType = 96; // It is a numerical value that is assigned to each codec in the SDP offer/answer exchange -> for H264
    // Create a peer connection with the codec parameters set in advance.
    const pc = new RTCPeerConnection({
        codecs: {
            audio: [],
            video: [
                new RTCRtpCodecParameters({
                    mimeType: "video/H264",
                    clockRate: 90000, // 90000 is the default value for H264
                    payloadType: payloadType,
                }),
            ],
        },
    });

    const track = new MediaStreamTrack({kind: "video"});


    udp.on("message", (data) => {
        console.log(data)
        const rtp = RtpPacket.deSerialize(data);
        rtp.header.payloadType = payloadType;
        track.writeRtp(rtp);
    });

    udp.on("error", (err) => {
        console.log(err)

    });

    udp.on("close", () => {
        console.log("close")
    });

    pc.addTransceiver(track, {direction: "sendonly"});

    await pc.setLocalDescription(await pc.createOffer());
    const sdp = JSON.stringify(pc.localDescription);
    socket.send(sdp);

    socket.on("message", (data: any) => {
        if (data.toString() === 'resetFFMPEG') {
            ffmpegProcess.kill('SIGINT');
            log.info(`FFMPEG process killed`)
            setTimeout(() => {
                ffmpegProcess = createFFmpegProcess();
                attachFFmpegListeners();
            }, 5000)
        } else {
            pc.setRemoteDescription(JSON.parse(data));
        }
    });
});

这首：

<!DOCTYPE html>
<html lang="ja">
<head>
    <meta charset="UTF-8" />
    <title>Answer</title>
    <script
            crossorigin
            src="https://unpkg.com/react@16/umd/react.development.js"
    ></script>
    <script
            crossorigin
            src="https://unpkg.com/react-dom@16/umd/react-dom.development.js"
    ></script>
    <script
            crossorigin
            src="https://cdnjs.cloudflare.com/ajax/libs/babel-core/5.8.34/browser.min.js"
    ></script>
    <script src="https://cdn.jsdelivr.net/npm/[email protected]/runtime.min.js"></script>
</head>
<body>
<div class="main">
    <div class="section" id="app1"></div>
</div>
<script type="text/babel">
    let rtc;

    const App = () => {
        const [log, setLog] = React.useState([]);
        const videoRef = React.useRef();
        const socket = new WebSocket("ws://localhost:8888");
        const [peer, setPeer] = React.useState(null); // Add state to keep track of the peer connection

        React.useEffect(() => {
            (async () => {
                await new Promise((r) => (socket.onopen = r));
                console.log("open websocket");

                const handleOffer = async (offer) => {
                    console.log("new offer", offer.sdp);

                    const updatedPeer = new RTCPeerConnection({
                        iceServers: [],
                        sdpSemantics: "unified-plan",
                    });

                    updatedPeer.onicecandidate = ({ candidate }) => {
                        if (!candidate) {
                            const sdp = JSON.stringify(updatedPeer.localDescription);
                            console.log(sdp);
                            socket.send(sdp);
                        }
                    };

                    updatedPeer.oniceconnectionstatechange = () => {
                        console.log(
                            "oniceconnectionstatechange",
                            updatedPeer.iceConnectionState
                        );
                    };

                    updatedPeer.ontrack = (e) => {
                        console.log("ontrack", e);
                        videoRef.current.srcObject = e.streams[0];
                    };

                    await updatedPeer.setRemoteDescription(offer);
                    const answer = await updatedPeer.createAnswer();
                    await updatedPeer.setLocalDescription(answer);

                    setPeer(updatedPeer);
                };

                socket.onmessage = (ev) => {
                    const data = JSON.parse(ev.data);
                    if (data.type === "offer") {
                        handleOffer(data);
                    } else if (data.type === "resetFFMPEG") {
                        // Handle the resetFFMPEG message
                        console.log("FFmpeg reset requested");
                    }
                };
            })();
        }, []); // Added socket as a dependency to the useEffect hook

        const sendRequestToResetFFmpeg = () => {
            socket.send("resetFFMPEG");
        };

        return (
            <div>
                Video: 
                <video ref={videoRef} autoPlay muted />
                <button onClick={() => sendRequestToResetFFmpeg()}>Reset FFMPEG</button>
            </div>
        );
    };

    ReactDOM.render(<App />, document.getElementById("app1"));
</script>
</body>
</html>

Answer 1