尽可能快地从内存缓冲区到磁盘

Question

我想提出一个场景并讨论合适的设计模式来解决它。

考虑一个简单的情况：摄像机在停止之前在内存缓冲区中记录十秒钟。记录结束后，二进制文件描述符将打开，数据将传输到磁盘。

此方法的一个主要限制是录制受到可用 RAM 大小的限制。但是，丢帧可能不是问题。

为了缓解这种情况，一种可能的解决方案是使用专用线程或进程来写入磁盘。在此设置中，生产者内存缓冲区在主线程/进程和写入线程/进程之间共享。然而，这引入了一个新问题：当编写器线程锁定缓冲区时，相机可能无法放置新帧，从而导致潜在的帧丢失。

问题 是否有一种设计模式可以解决第二个场景中突出的问题？

下面是 Python 中两种场景的一些代码示例。

Python 中的第一个场景：

import io
from picamera2 import Picamera2
from picamera2.encoders import Encoder as NullEncoder
from picamera2.outputs import FileOutput

# Init camera
cam = Picamera2()

# Init memory buffer
mem_buff = io.BytesIO()
mem_out = FileOutput(mem_buff)

# Open camera
cam.start()

# Just writes frames without encoding i.e.: BGR888
encoder = NullEncoder()

# Recording time
to_record = 10
print(f"Start recording for {to_record} seconds")
cam.start_recording(encoder, mem_out)
time.sleep(to_record)
cam.stop_recording()
print("Finish recording")
cam.close()

# Begin data transfer to disk
out_fpath = "video.bin"
disk_transfer_start = time.perf_counter()
with open(out_fpath, "wb") as fd:
    fd.write(mem_buff.getvalue())
disk_transfer_el = time.perf_counter() - disk_transfer_start
print(f"Data transfer took {disk_transfer_el} sec")

# Get a sense of how many frames where missing
totbytes = os.path.getsize(out_fpath)
byteel = 2304*1296*3 # (frame_width * frame_height * num_channels)
num_frames = totbytes / byteel
print(f"Video has {num_frames} frames")

第二种情况在 Python 中的可能实现：

import io
from threading import Thread, Event, Lock
from picamera2 import Picamera2
from picamera2.encoders import Encoder as NullEncoder
from picamera2.outputs import FileOutput

def disk_writer(mem_buff: io.BytesIO, bin_fd, write_interval: int, stop_event: Event, lock: Lock):
    while not stop_event.is_set():
        start_loop = time.perf_counter()
        lock.acquire()
        curr_buff_pos = mem_buff.tell()
        lock.release()
        if curr_buff_pos > 0:
            lock.acquire()
            bin_fd.write(mem_buff.getvalue())
            mem_buff.seek(0)
            mem_buff.truncate(0)
            lock.release()
        elapsed = time.perf_counter() - start_loop
        if elapsed < write_interval:
            time.sleep(write_interval - elapsed)
    if mem_buff.tell() > 0:
        bin_fd.write(mem_buff.getvalue())
        mem_buff.seek(0)
        mem_buff.truncate(0)

# Init camera
cam = Picamera2()

# Init memory buffer
mem_buff = io.BytesIO()
mem_out = FileOutput(mem_buff)

# Open camera
cam.start()

# Create writing thread and start
stop_event = Event()
lock = Lock()
write_interval = 5
writer_thread = Thread(target=disk_writer, args=(mem_buff, bin_fd, write_interval, stop_event, lock))
writer_thread.start()

# Just writes frames without encoding i.e.: BGR888
encoder = NullEncoder()

# Recording time
to_record = 10
print(f"Start recording for {to_record} seconds")
cam.start_recording(encoder, mem_out)
time.sleep(to_record)
cam.stop_recording()
print("Finish recording")
stop_event.set()
writer_thread.join()
cam.close()
bin_fd.close()


# Get a sense of how many frames where missing
totbytes = os.path.getsize(out_fpath)
byteel = 2304*1296*3 # (frame_width * frame_height * num_channels)
num_frames = totbytes / byteel
print(f"Video has {num_frames} frames")

Answer 1

我的建议是简单地使用带有 DRAM 缓存的 SSD，而不是 SD 卡。
只需将数据写入磁盘，无需内存缓冲区，然后让缓存处理即可。
这是一个非常简单的解决方案，它应该提供可预测的结果，以便您可以调整将数据写入磁盘的速率，而不会压垮缓存。

如果您只需要使用 SD 卡，您可以通过编辑 fstab 中的选项文件来影响操作系统级磁盘缓存的行为（假设您使用的是 Raspbian）。
查看树莓派官方论坛上的这篇文章，了解如何影响磁盘缓存：
https://forums.raspberrypi.com/viewtopic.php?t=157743#p1026791