TypeError:h5py 对象无法被 pickle

问题描述 投票:0回答:2

我正在尝试运行代码的PyTorch实现,该代码应该适用于SBD数据集

训练标签最初以 .bin 文件形式提供,然后转换为 HDF5 (.h5) 文件。

运行算法时,出现错误:“TypeError: h5py 对象无法被腌制”

我认为该错误源于 torch.utils.data.DataLoader。

知道我是否在这里遗漏了任何概念吗?我读到,pickling 通常不是首选,但截至目前,我的数据集仅采用 HDF5 格式。

供您参考,错误的堆栈跟踪如下:

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\main.py", line 130, in <module>
    main()

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\main.py", line 85, in main
    win_feats5, win_fusion, viz, global_step)

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\train_val\model_play.py", line 31, in train
    for i, (img, target) in enumerate(train_loader):

  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
    return _DataLoaderIter(self)

  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
    w.start()

  File "C:\Anaconda3\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)

  File "C:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)

  File "C:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)

  File "C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)

  File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

  File "C:\Anaconda3\lib\site-packages\h5py\_hl\base.py", line 308, in __getnewargs__
    raise TypeError("h5py objects cannot be pickled")

  TypeError: h5py objects cannot be pickled

我正在使用 Conda 版本 4.8.2、Python 3.7.4、PyTorch 1.0.0 和 Cuda 10.2.89

谢谢,

python pytorch hdf5 h5py
2个回答
3
投票

设置 num_workers=0 为我解决这个问题


0
投票

我更喜欢使用 h5pickle 作为包装器来读取 h5 文件。 详细信息请参阅:https://github.com/DaanVanVugt/h5pickle

© www.soinside.com 2019 - 2024. All rights reserved.