由于 pip 内存使用,奇点构建崩溃了

问题描述 投票:0回答:1

我目前正在尝试从具有奇点 4.1.3 的 .def 文件构建一个非常简单的 .sif 图像。 问题是,无论我尝试使用 pip install 设置什么,我总是会因以下原因而导致构建失败:

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

我尝试设置 TMPDIR 变量,尝试使用 pip --no-cache-dir 选项,但没有任何改变。

我使用的是 Fedora 39,并且有 8GB RAM。当我使用 Ubuntu 时,我从未经历过这样的事情,而且我正在构建可能更重的图像。

奇点的 .def 文件如下:

Bootstrap: library
From: ubuntu:20.04

%post
  export DEBIAN_FRONTEND=noninteractive
  apt-get -qq -y update

  apt-get -qq -y install apt-utils > /dev/null

  apt-get -qq -y install --no-install-recommends \
      python3 \
      python3-tk \
      python3-pip \
      python3-distutils \
      python3-setuptools \

  pip3 install -q langchain faiss-cpu torch sentence_transformers```

在构建图像期间,我可以看到(通过 htop),一旦构建进入“pip install”部分,RAM 使用量就会稳定增加到 5.9GB/6.7GB,此时构建会因错误而被终止:

ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

FATAL:   While performing build: while running engine: exit status 1

命令的完整输出是:

WARNING: 'nodev' mount option set on /tmp, it could be a source of failure during build process
INFO:    Starting build...
INFO:    Using cached image
INFO:    Verifying bootstrap image /root/.singularity/cache/library/sha256.cfb23cc09dd3b4570110dd6f13886fe95cc31ef2d81f7abab6796751f5700fa0
INFO:    Running post scriptlet
+ export DEBIAN_FRONTEND=noninteractive
+ apt-get -qq -y update
+ apt-get -qq -y install apt-utils
debconf: delaying package configuration, since apt-utils is not installed
+ apt-get -qq -y install --no-install-recommends python3 python3-tk python3-pip python3-distutils python3-setuptools
Extracting templates from packages: 100%
Preconfiguring packages ...
Selecting previously unselected package libssl1.1:amd64.
(Reading database ... 4162 files and directories currently installed.)
Preparing to unpack .../libssl1.1_1.1.1f-1ubuntu2.22_amd64.deb ...
Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.22) ...
Selecting previously unselected package libpython3.8-minimal:amd64.
Preparing to unpack .../libpython3.8-minimal_3.8.10-0ubuntu1~20.04.9_amd64.deb ...
Unpacking libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.9) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../libexpat1_2.2.9-1ubuntu0.6_amd64.deb ...
Unpacking libexpat1:amd64 (2.2.9-1ubuntu0.6) ...
Selecting previously unselected package python3.8-minimal.
Preparing to unpack .../python3.8-minimal_3.8.10-0ubuntu1~20.04.9_amd64.deb ...
Unpacking python3.8-minimal (3.8.10-0ubuntu1~20.04.9) ...
Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.22) ...
Setting up libpython3.8-minimal:amd64 (3.8.10-0ubuntu1~20.04.9) ...
Setting up libexpat1:amd64 (2.2.9-1ubuntu0.6) ...
Setting up python3.8-minimal (3.8.10-0ubuntu1~20.04.9) ...
Selecting previously unselected package python3-minimal.
(Reading database ... 4463 files and directories currently installed.)
Preparing to unpack .../0-python3-minimal_3.8.2-0ubuntu2_amd64.deb ...
Unpacking python3-minimal (3.8.2-0ubuntu2) ...
Selecting previously unselected package mime-support.
Preparing to unpack .../1-mime-support_3.64ubuntu1_all.deb ...
Unpacking mime-support (3.64ubuntu1) ...
Selecting previously unselected package libmpdec2:amd64.
Preparing to unpack .../2-libmpdec2_2.4.2-3_amd64.deb ...
Unpacking libmpdec2:amd64 (2.4.2-3) ...
Selecting previously unselected package readline-common.
Preparing to unpack .../3-readline-common_8.0-4_all.deb ...
Unpacking readline-common (8.0-4) ...
Selecting previously unselected package libreadline8:amd64.
Preparing to unpack .../4-libreadline8_8.0-4_amd64.deb ...
Unpacking libreadline8:amd64 (8.0-4) ...
Selecting previously unselected package libsqlite3-0:amd64.
Preparing to unpack .../5-libsqlite3-0_3.31.1-4ubuntu0.6_amd64.deb ...
Unpacking libsqlite3-0:amd64 (3.31.1-4ubuntu0.6) ...
Selecting previously unselected package libpython3.8-stdlib:amd64.
Preparing to unpack .../6-libpython3.8-stdlib_3.8.10-0ubuntu1~20.04.9_amd64.deb ...
Unpacking libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.9) ...
Selecting previously unselected package python3.8.
Preparing to unpack .../7-python3.8_3.8.10-0ubuntu1~20.04.9_amd64.deb ...
Unpacking python3.8 (3.8.10-0ubuntu1~20.04.9) ...
Selecting previously unselected package libpython3-stdlib:amd64.
Preparing to unpack .../8-libpython3-stdlib_3.8.2-0ubuntu2_amd64.deb ...
Unpacking libpython3-stdlib:amd64 (3.8.2-0ubuntu2) ...
Setting up python3-minimal (3.8.2-0ubuntu2) ...
Selecting previously unselected package python3.
(Reading database ... 4893 files and directories currently installed.)
Preparing to unpack .../00-python3_3.8.2-0ubuntu2_amd64.deb ...
Unpacking python3 (3.8.2-0ubuntu2) ...
Selecting previously unselected package openssl.
Preparing to unpack .../01-openssl_1.1.1f-1ubuntu2.22_amd64.deb ...
Unpacking openssl (1.1.1f-1ubuntu2.22) ...
Selecting previously unselected package ca-certificates.
Preparing to unpack .../02-ca-certificates_20230311ubuntu0.20.04.1_all.deb ...
Unpacking ca-certificates (20230311ubuntu0.20.04.1) ...
Selecting previously unselected package libbsd0:amd64.
Preparing to unpack .../03-libbsd0_0.10.0-1_amd64.deb ...
Unpacking libbsd0:amd64 (0.10.0-1) ...
Selecting previously unselected package python3-pkg-resources.
Preparing to unpack .../04-python3-pkg-resources_45.2.0-1ubuntu0.1_all.deb ...
Unpacking python3-pkg-resources (45.2.0-1ubuntu0.1) ...
Selecting previously unselected package tzdata.
Preparing to unpack .../05-tzdata_2024a-0ubuntu0.20.04_all.deb ...
Unpacking tzdata (2024a-0ubuntu0.20.04) ...
Selecting previously unselected package ucf.
Preparing to unpack .../06-ucf_3.0038+nmu1_all.deb ...
Moving old data out of the way
Unpacking ucf (3.0038+nmu1) ...
Selecting previously unselected package libpng16-16:amd64.
Preparing to unpack .../07-libpng16-16_1.6.37-2_amd64.deb ...
Unpacking libpng16-16:amd64 (1.6.37-2) ...
Selecting previously unselected package libxau6:amd64.
Preparing to unpack .../08-libxau6_1%3a1.0.9-0ubuntu1_amd64.deb ...
Unpacking libxau6:amd64 (1:1.0.9-0ubuntu1) ...
Selecting previously unselected package libxdmcp6:amd64.
Preparing to unpack .../09-libxdmcp6_1%3a1.1.3-0ubuntu1_amd64.deb ...
Unpacking libxdmcp6:amd64 (1:1.1.3-0ubuntu1) ...
Selecting previously unselected package libxcb1:amd64.
Preparing to unpack .../10-libxcb1_1.14-2_amd64.deb ...
Unpacking libxcb1:amd64 (1.14-2) ...
Selecting previously unselected package libx11-data.
Preparing to unpack .../11-libx11-data_2%3a1.6.9-2ubuntu1.6_all.deb ...
Unpacking libx11-data (2:1.6.9-2ubuntu1.6) ...
Selecting previously unselected package libx11-6:amd64.
Preparing to unpack .../12-libx11-6_2%3a1.6.9-2ubuntu1.6_amd64.deb ...
Unpacking libx11-6:amd64 (2:1.6.9-2ubuntu1.6) ...
Selecting previously unselected package libxext6:amd64.
Preparing to unpack .../13-libxext6_2%3a1.3.4-0ubuntu1_amd64.deb ...
Unpacking libxext6:amd64 (2:1.3.4-0ubuntu1) ...
Selecting previously unselected package libtcl8.6:amd64.
Preparing to unpack .../14-libtcl8.6_8.6.10+dfsg-1_amd64.deb ...
Unpacking libtcl8.6:amd64 (8.6.10+dfsg-1) ...
Selecting previously unselected package libfreetype6:amd64.
Preparing to unpack .../15-libfreetype6_2.10.1-2ubuntu0.3_amd64.deb ...
Unpacking libfreetype6:amd64 (2.10.1-2ubuntu0.3) ...
Selecting previously unselected package fonts-dejavu-core.
Preparing to unpack .../16-fonts-dejavu-core_2.37-1_all.deb ...
Unpacking fonts-dejavu-core (2.37-1) ...
Selecting previously unselected package fontconfig-config.
Preparing to unpack .../17-fontconfig-config_2.13.1-2ubuntu3_all.deb ...
Unpacking fontconfig-config (2.13.1-2ubuntu3) ...
Selecting previously unselected package libfontconfig1:amd64.
Preparing to unpack .../18-libfontconfig1_2.13.1-2ubuntu3_amd64.deb ...
Unpacking libfontconfig1:amd64 (2.13.1-2ubuntu3) ...
Selecting previously unselected package libxrender1:amd64.
Preparing to unpack .../19-libxrender1_1%3a0.9.10-1_amd64.deb ...
Unpacking libxrender1:amd64 (1:0.9.10-1) ...
Selecting previously unselected package libxft2:amd64.
Preparing to unpack .../20-libxft2_2.3.3-0ubuntu1_amd64.deb ...
Unpacking libxft2:amd64 (2.3.3-0ubuntu1) ...
Selecting previously unselected package x11-common.
Preparing to unpack .../21-x11-common_1%3a7.7+19ubuntu14_all.deb ...
dpkg-query: no packages found matching nux-tools
Unpacking x11-common (1:7.7+19ubuntu14) ...
Selecting previously unselected package libxss1:amd64.
Preparing to unpack .../22-libxss1_1%3a1.2.3-1_amd64.deb ...
Unpacking libxss1:amd64 (1:1.2.3-1) ...
Selecting previously unselected package libtk8.6:amd64.
Preparing to unpack .../23-libtk8.6_8.6.10-1_amd64.deb ...
Unpacking libtk8.6:amd64 (8.6.10-1) ...
Selecting previously unselected package tk8.6-blt2.5.
Preparing to unpack .../24-tk8.6-blt2.5_2.5.3+dfsg-4_amd64.deb ...
Unpacking tk8.6-blt2.5 (2.5.3+dfsg-4) ...
Selecting previously unselected package blt.
Preparing to unpack .../25-blt_2.5.3+dfsg-4_amd64.deb ...
Unpacking blt (2.5.3+dfsg-4) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../26-python-pip-whl_20.0.2-5ubuntu1.10_all.deb ...
Unpacking python-pip-whl (20.0.2-5ubuntu1.10) ...
Selecting previously unselected package python3-lib2to3.
Preparing to unpack .../27-python3-lib2to3_3.8.10-0ubuntu1~20.04_all.deb ...
Unpacking python3-lib2to3 (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package python3-distutils.
Preparing to unpack .../28-python3-distutils_3.8.10-0ubuntu1~20.04_all.deb ...
Unpacking python3-distutils (3.8.10-0ubuntu1~20.04) ...
Selecting previously unselected package python3-setuptools.
Preparing to unpack .../29-python3-setuptools_45.2.0-1ubuntu0.1_all.deb ...
Unpacking python3-setuptools (45.2.0-1ubuntu0.1) ...
Selecting previously unselected package python3-wheel.
Preparing to unpack .../30-python3-wheel_0.34.2-1ubuntu0.1_all.deb ...
Unpacking python3-wheel (0.34.2-1ubuntu0.1) ...
Selecting previously unselected package python3-pip.
Preparing to unpack .../31-python3-pip_20.0.2-5ubuntu1.10_all.deb ...
Unpacking python3-pip (20.0.2-5ubuntu1.10) ...
Selecting previously unselected package python3-tk:amd64.
Preparing to unpack .../32-python3-tk_3.8.10-0ubuntu1~20.04_amd64.deb ...
Unpacking python3-tk:amd64 (3.8.10-0ubuntu1~20.04) ...
Setting up libxau6:amd64 (1:1.0.9-0ubuntu1) ...
Setting up mime-support (3.64ubuntu1) ...
Setting up libsqlite3-0:amd64 (3.31.1-4ubuntu0.6) ...
Setting up x11-common (1:7.7+19ubuntu14) ...
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
Setting up tzdata (2024a-0ubuntu0.20.04) ...

Current default time zone: 'Etc/UTC'
Local time is now:      Sun May 12 22:25:53 UTC 2024.
Universal Time is now:  Sun May 12 22:25:53 UTC 2024.
Run 'dpkg-reconfigure tzdata' if you wish to change it.

Setting up libx11-data (2:1.6.9-2ubuntu1.6) ...
Setting up libpng16-16:amd64 (1.6.37-2) ...
Setting up libtcl8.6:amd64 (8.6.10+dfsg-1) ...
Setting up fonts-dejavu-core (2.37-1) ...
Setting up ucf (3.0038+nmu1) ...
Setting up libmpdec2:amd64 (2.4.2-3) ...
Setting up openssl (1.1.1f-1ubuntu2.22) ...
Setting up libbsd0:amd64 (0.10.0-1) ...
Setting up readline-common (8.0-4) ...
Setting up libxdmcp6:amd64 (1:1.1.3-0ubuntu1) ...
Setting up libxcb1:amd64 (1.14-2) ...
Setting up fontconfig-config (2.13.1-2ubuntu3) ...
Setting up libreadline8:amd64 (8.0-4) ...
Setting up ca-certificates (20230311ubuntu0.20.04.1) ...
Updating certificates in /etc/ssl/certs...
137 added, 0 removed; done.
Setting up libfreetype6:amd64 (2.10.1-2ubuntu0.3) ...
Setting up libx11-6:amd64 (2:1.6.9-2ubuntu1.6) ...
Setting up python-pip-whl (20.0.2-5ubuntu1.10) ...
Setting up libfontconfig1:amd64 (2.13.1-2ubuntu3) ...
Setting up libpython3.8-stdlib:amd64 (3.8.10-0ubuntu1~20.04.9) ...
Setting up python3.8 (3.8.10-0ubuntu1~20.04.9) ...
Setting up libpython3-stdlib:amd64 (3.8.2-0ubuntu2) ...
Setting up libxrender1:amd64 (1:0.9.10-1) ...
Setting up libxext6:amd64 (2:1.3.4-0ubuntu1) ...
Setting up python3 (3.8.2-0ubuntu2) ...
Setting up python3-wheel (0.34.2-1ubuntu0.1) ...
Setting up libxss1:amd64 (1:1.2.3-1) ...
Setting up python3-lib2to3 (3.8.10-0ubuntu1~20.04) ...
Setting up python3-pkg-resources (45.2.0-1ubuntu0.1) ...
Setting up python3-distutils (3.8.10-0ubuntu1~20.04) ...
Setting up libxft2:amd64 (2.3.3-0ubuntu1) ...
Setting up python3-setuptools (45.2.0-1ubuntu0.1) ...
Setting up libtk8.6:amd64 (8.6.10-1) ...
Setting up python3-pip (20.0.2-5ubuntu1.10) ...
Setting up tk8.6-blt2.5 (2.5.3+dfsg-4) ...
Setting up blt (2.5.3+dfsg-4) ...
Setting up python3-tk:amd64 (3.8.10-0ubuntu1~20.04) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for ca-certificates (20230311ubuntu0.20.04.1) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
+ pip3 install -q --no-cache-dir langchain faiss-cpu torch sentence_transformers
ERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device

FATAL:   While performing build: while running engine: exit status 1

构建实例化为

sudo singularity build faiss_env.sif faiss_index_env.def

到目前为止我尝试过的:

  • 尝试放大我机器上的
    /tmp
    。预计它不会崩溃,但即使使用了更多内存(20GB 中的 5GB),构建仍然失败并出现相同的错误,并且 RAM 使用仍然疯狂
  • 尝试在使用
    TMPDIR
    到图像内的新目录时指定
    pip install ...
    变量,但出现了同样的问题
  • 尝试使用
    --no-cache-dir
    选项,但
    pip install ...
    仍然没有运气

如果有人可以向我伸出援助之手,我将不胜感激。

pip torch singularity-container
1个回答
0
投票

问题出在 Fedora 39 的策略中,它将

/tmp
文件夹挂载到 RAM 中。禁用它解决了问题。 请仅在阅读并理解您在做什么后才将其禁用。 为此,请运行以下命令:

systemctl mask tmp.mount

然后重新启动。

请参阅我打开的 github 问题fedora 页面以供参考。

© www.soinside.com 2019 - 2024. All rights reserved.