我正在通过 pybind11 为一个优化库编写一组 Python 包装器,该库的繁重代码是用 C++ 编写的。
要包装的 C++ 代码的抽象类层次结构当前看起来像这样 (multivariate.h):
typedef std::function<double(const double*)> multivariate;
struct multivariate_problem {
// objective
multivariate _f;
int _n;
// bound constraints
double *_lower;
double *_upper;
multivariate_problem(const multivariate f, const int n, double *lower,
double *upper):
_f(f), _n(n), _lower(lower), _upper(upper)) {
}
};
struct multivariate_solution {
... // some code here for return a solution (seems to work ok)
};
class MultivariateOptimizer {
public:
virtual ~MultivariateOptimizer() {
}
// initialize the optimizer
virtual void init(const multivariate_problem &f, const double *guess) = 0;
// perform one step of iteration
virtual void iterate() = 0;
// retrieve current solution
virtual multivariate_solution solution() = 0;
// this essentially calls init(), performs a number of iterate() calls, returns solution()
virtual multivariate_solution optimize(const multivariate_problem &f,
const double *guess) = 0;
};
现在,我的 pybind 代码看起来像这样(multivariate_py.cpp):
// wrap the function expressions
typedef std::function<double(const py::array_t<double>&)> py_multivariate;
// wrap the multivariable optimizer
void build_multivariate(py::module_ &m) {
// wrap the solution object
py::class_<multivariate_solution> solution(m, "MultivariateSolution");
...
// wrap the solver
py::class_<MultivariateOptimizer> solver(m, "MultivariateSearch");
// corresponds to MultivariateSearch::optimize()
solver.def("optimize",
[](MultivariateOptimizer &self, py_multivariate py_f,
py::array_t<double> &py_lower,
py::array_t<double> &py_upper,
py::array_t<double> &py_guess) {
const int n = py_lower.size();
double *lower = static_cast<double*>(py_lower.request().ptr);
double *upper = static_cast<double*>(py_upper.request().ptr);
const multivariate &f = [&py_f, &n](const double *x) -> double {
const auto &py_x = py::array_t<double>(n, x);
return py_f(py_x);
};
const multivariate_problem &prob { f, n, lower, upper };
double *guess = static_cast<double*>(py_guess.request().ptr);
return self.optimize(prob, guess);
}, "f"_a, "lower"_a, "upper"_a, "guess"_a,
py::call_guard<py::scoped_ostream_redirect,
py::scoped_estream_redirect>());
// corresponds to MultivariateSearch::init()
solver.def("initialize",
[](MultivariateOptimizer &self, py_multivariate py_f,
py::array_t<double> &py_lower,
py::array_t<double> &py_upper,
py::array_t<double> &py_guess) {
const int n = py_lower.size();
double *lower = static_cast<double*>(py_lower.request().ptr);
double *upper = static_cast<double*>(py_upper.request().ptr);
const multivariate &f = [&py_f, &n](const double *x) -> double {
const auto &py_x = py::array_t<double>(n, x);
return py_f(py_x);
};
const multivariate_problem &prob { f, n, lower, upper };
double *guess = static_cast<double*>(py_guess.request().ptr);
return self.init(prob, guess);
}, "f"_a, "lower"_a, "upper"_a, "guess"_a,
py::call_guard<py::scoped_ostream_redirect,
py::scoped_estream_redirect>());
// corresponds to MultivariateSearch::iterate()
solver.def("iterate", &MultivariateOptimizer::iterate);
// corresponds to MultivariateSearch::solution()
solver.def("solution", &MultivariateOptimizer::solution);
// put algorithm-specific bindings here
build_acd(m);
build_amalgam(m);
build_basin_hopping(m);
...
如您所见,我在内部将 python 函数包装为 C++ 函数,然后将该函数包装在 multivariate_problem 对象中以传递到我的 C++ 后端。
然后由 pybind 使用以下入口点(mylibrary.cpp)调用:
namespace py = pybind11;
void build_multivariate(py::module_&);
PYBIND11_MODULE(mypackagename, m) {
build_multivariate(m);
...
}
pybind11 将通过
typical pip install .
命令通过 setup.py 编译此文件,不会出现错误。事实上,大多数代码对于 initialize()
和 optimize()
调用都可以正常工作。例如,以下 Python 代码将正常运行(假设 mylibrary
替换为 setup.py 中我的包的名称,并且 MySolverName
是 MultivariateSearch
的特定实例:
import numpy as np
from mylibrary import MySolverName
# function to optimize
def fx(x):
total = 0.0
for x1, x2 in zip(x[:-1], x[1:]):
total += 100 * (x2 - x1 ** 2) ** 2 + (1 - x1) ** 2
return total
n = 10 # dimension of problem
alg = MySolverName(some_args_to_pass...)
sol = alg.optimize(fx,
lower=-10 * np.ones(n),
upper=10 * np.ones(n),
guess=np.random.uniform(low=-10., high=10., size=n))
print(sol)
但是,这就是我目前陷入困境的地方。我还希望用户能够选择以交互方式运行求解器,这就是
initialize
和 iterate
函数发挥作用的地方。但是,上面提供的绑定适用于 optimize
,但不适用于 iterate
。
为了说明用例,以下代码不会运行:
import numpy as np
from mylibrary import MySolverName
# function to optimize
def fx(x):
... return result here for np.ndarray x
n = 10 # dimension of problem
alg = MySolverName(some_args_to_pass...)
alg.initialize(f=fx,
lower=-10 * np.ones(n),
upper=10 * np.ones(n),
guess=np.random.uniform(low=-10., high=10., size=n))
alg.iterate()
print(alg.solution())
执行
iterate
命令时,脚本会挂起,然后在一段时间后终止,通常不会出现错误,并且不会打印最后一行。在极少数情况下,它会在 Python 端产生崩溃,即“维度不能为负”,但没有关于这与哪个内存相关的指导。如果没有 iterate()
行,代码将按预期成功打印最后一行。
当我在 C++
std::cout
函数中 iterate()
时,它总是挂在对函数 fx
的任何调用之前,并且在调用后不打印任何内容,因此我将问题范围缩小到了 Python 函数无法正确坚持以 iterate()
作为入口点。然而,optimize()
在C++代码内部调用iterate()
,并且前面的示例运行成功,所以错误相当奇怪。
我尝试浏览 pybind 的文档,但我找不到一个可以准确实现我上面想要做的事情的示例。我尝试过其他解决方案,例如
keep_alive<>
,但无济于事。
你能帮我修改 pybind 包装器,以便内部存储的 Python 函数将保留在 C++ 对象中,并且即使在 iterate() 调用之间再次将控制权恢复到 Python 解释器后也能正确执行吗? 感谢您的帮助!
受到 pptaszni 评论的启发,我想我已经通过删除 pybind 包装函数本身内的所有悬空引用来解决这个问题:
solver.def("initialize",
[](MultivariateOptimizer &self, py_multivariate &py_f,
py::array_t<double> &py_lower,
py::array_t<double> &py_upper,
py::array_t<double> &py_guess) {
const int n = py_lower.size();
double *lower = static_cast<double*>(py_lower.request().ptr);
double *upper = static_cast<double*>(py_upper.request().ptr);
multivariate f = [py_f, n](const double *x) -> double {
const auto py_x = py::array_t<double>(n, x);
return py_f(py_x);
};
multivariate_problem prob { f, n, lower, upper };
double *guess = static_cast<double*>(py_guess.request().ptr);
return self.init(prob, guess);
}, "f"_a, "lower"_a, "upper"_a, "guess"_a,
py::call_guard<py::scoped_ostream_redirect,
py::scoped_estream_redirect>());
我不确定这是否是正确或最有效的处理方法,但它有效并且永远不会崩溃!