我正在Travis CI上构建Rust二进制文件(liblonlat_bng.dylib
),将其拉入Cython扩展(与Cython源.c / .pyx相同的目录),并测试它,也在Travis CI上(在不同的回购和建立)。但是,Python包的测试失败了,我不知道为什么:
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/travis/build/urschrei/convertbng/test/test_convertbng.py", line 15, in <module>
from convertbng.cutil import convert_bng as cconvert_bng
ImportError: dlopen(/Users/travis/build/urschrei/convertbng/convertbng/cutil.so, 2): Library not loaded: /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib
Referenced from: /Users/travis/build/urschrei/convertbng/convertbng/cutil.so
Reason: image not found
这是Travis软件包的OSX构建输出:
Installing collected packages: convertbng
Running setup.py develop for convertbng
Running command /Users/travis/build/urschrei/convertbng/venv/bin/python2.7 -c "import setuptools, tokenize;__file__='/Users/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
running develop
running egg_info
writing requirements to convertbng.egg-info/requires.txt
writing convertbng.egg-info/PKG-INFO
writing top-level names to convertbng.egg-info/top_level.txt
writing dependency_links to convertbng.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'convertbng.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'convertbng.egg-info/SOURCES.txt'
running build_ext
building 'convertbng.cutil' extension
creating build
creating build/temp.macosx-10.11-x86_64-2.7
creating build/temp.macosx-10.11-x86_64-2.7/convertbng
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -Iconvertbng -I/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c convertbng/cutil.c -o build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -O3
[unused function warnings]
creating build/lib.macosx-10.11-x86_64-2.7
creating build/lib.macosx-10.11-x86_64-2.7/convertbng
clang -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so
copying build/lib.macosx-10.11-x86_64-2.7/convertbng/cutil.so -> convertbng
Creating /Users/travis/build/urschrei/convertbng/venv/lib/python2.7/site-packages/convertbng.egg-link (link to .)
Adding convertbng 0.4.14 to easy-install.pth file
Installed /Users/travis/build/urschrei/convertbng
Successfully installed convertbng
这是Linux Travis输出,它成功地将dylib定位在相对路径上。请注意$ORIGIN
的-R
参数,该参数不能在OSX上使用:
Installing collected packages: convertbng
Running setup.py develop for convertbng
Running command /usr/bin/python -c "import setuptools, tokenize;__file__='/home/travis/build/urschrei/convertbng/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
running develop
running egg_info
writing requirements to convertbng.egg-info/requires.txt
writing convertbng.egg-info/PKG-INFO
writing top-level names to convertbng.egg-info/top_level.txt
writing dependency_links to convertbng.egg-info/dependency_links.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'convertbng.egg-info/SOURCES.txt'
writing manifest file 'convertbng.egg-info/SOURCES.txt'
running build_ext
building 'convertbng.cutil' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/convertbng
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I. -Iconvertbng -I/usr/include/python2.7 -c convertbng/cutil.c -o build/temp.linux-x86_64-2.7/convertbng/cutil.o -O3
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/convertbng
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/convertbng/cutil.o -L. -Lconvertbng -Wl,-R$ORIGIN -llonlat_bng -o build/lib.linux-x86_64-2.7/convertbng/cutil.so
copying build/lib.linux-x86_64-2.7/convertbng/cutil.so -> convertbng
Creating /usr/local/lib/python2.7/dist-packages/convertbng.egg-link (link to .)
Adding convertbng 0.4.14 to easy-install.pth file
Installed /home/travis/build/urschrei/convertbng
Successfully installed convertbng
其他详情:
setup.py build_ext --inplace
,则测试通过setup.py build_ext --inplace
,则测试失败并出现与Travis相同的错误。我对这是怎么回事感到非常困惑。我应该在二进制文件中看到什么吗? otool -l
输出略有不同。例如:
本地:
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 01:00:02 1970
current version 1226.10.1
特拉维斯:
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 00:00:02 1970
current version 1225.1.1
链接到传递Linux的工作:https://travis-ci.org/urschrei/convertbng/jobs/136730347 链接到失败的OSX工作:https://travis-ci.org/urschrei/convertbng/jobs/136730348
更新:这肯定是因为库名。如果我使用install_name_tool
将dylib位置更改为@loader_path/liblonlat_bng.dylib
:
install_name_tool -change /Users/travis/build/urschrei/lonlat_bng/target/x86_64-apple-darwin/release/liblonlat_bng.dylib @loader_path/liblonlat_bng.dylib convertbng/cutil.so
Travis构建的可执行文件将在我的本地计算机上运行。然而,特拉维斯似乎有一个破坏的install_name_tool
安装,我没有设法从cutil.so
传递正确的调用setup.py
。我已经尝试将extra_link_args
设置为生成的['-Wl,-rpath,'+'@loader_path/liblonlat_bng.dylib']
/usr/bin/clang -bundle -undefined dynamic_lookup -arch i386 -arch x86_64 -g build/temp.macosx-10.6-intel-2.7/convertbng/cutil.o -L. -Lconvertbng -llonlat_bng -o build/lib.macosx-10.6-intel-2.7/convertbng/cutil.so -Wl,-rpath,@loader_path/liblonlat_bng.dylib
但如果我这样做,我的测试仍然会失败。
更新2:在建造货物时,我可以在链接阶段修复dylib的install_name
:
RUSTFLAGS="-C link-args=-Wl,-install_name,@rpath/liblonlat_bng.dylib" cargo build --release
这很有效。但我怀疑我的setup.py
Extension
设置错误:
# only append the runtime dir on Linux
rdirs = []
ldirs = []
if sys.platform != 'darwin':
# from http://stackoverflow.com/a/10252190/416626
# the $ORIGIN trick is not perfect, though
rdirs = ['$ORIGIN']
if sys.platform == 'darwin':
ldirs = ['-Wl,-rpath,'+'@loader_path/liblonlat_bng.dylib']
extensions = Extension("convertbng.cutil",
sources=["convertbng/cutil" + suffix],
libraries=["lonlat_bng"],
include_dirs=['.', 'convertbng'],
library_dirs=['.', 'convertbng'],
runtime_library_dirs=rdirs,
extra_compile_args=["-O3"],
extra_link_args=ldirs
)
有两种方法可以在不使用install_name_tool
的情况下解决这个问题。
假设您的dylib与Cython扩展名在同一目录中(在本例中为cutil.so
)
rpath
支持编译Rust二进制文件,并在setup.py
Extension
实例中正确设置rpath:
最干净的方法是在你的rpath=true
中设置[profile.release]
under cargo.toml
。 See here。
你也可以调用RUSTFLAGS="-C rpath" cargo build --release
。cargo rustc --release -- -C link-args=-Wl,-install_name,@rpath/libname.dylib
在Rust dylib中启用rpath
支持后,编辑你的setup.py
:
在你的extra_link_args=["-Wl,-rpath", "-Wl,@loader_path/"]
实例中设置(或添加)Extension
。
如果你编译你的Cython扩展(例如使用setup.py build_ext --inplace
),然后在生成的otool -l
上运行.so
,你就会知道它是有用的:
@rpath/liblonlat_bng.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
如果你希望你的Rust dylib相对于你的Cython扩展,你需要相应调整@rpath/
中的@loader_path/
或setup.py
路径。
注意:上述内容仅适用于OS X上的相对路径。在Linux上,您可以在extra_link_args=["-Wl,-rpath", "-Wl,$ORIGIN"]
setup.py
实例中设置Extension
(如果您的Rust dylib在其他地方相对于您的扩展名,则相应地调整它。)
我有同样的问题,我相信它是Distutils中的一个错误。我已经发出拉动请求来解决这个问题,我也提出了一个解决方法。见https://github.com/python/cpython/pull/12418
你能在setup.py
中尝试以下方法吗?
from Cython.Distutils.build_ext import new_build_ext as build_ext
# alternative:
# from distutils.command import build_ext
class my_build_ext(build_ext):
"""Workaround for rpath bug in distutils for OSX."""
def finalize_options(self):
super().finalize_options()
# Special treatment of rpath in case of OSX, to work around python
# distutils bug 36353. This constructs proper rpath arguments for clang.
# See https://bugs.python.org/issue36353
if sys.platform[:6] == "darwin":
for path in self.rpath:
for ext in self.extensions:
ext.extra_link_args.append("-Wl,-rpath," + path)
self.rpath[:] = []
setup(
cmdclass={'build_ext': my_build_ext}
# ...
)