计算pyspark数据框中的地理距离

Question

我的数据框：

DF = spark.createDataFrame([[114.038696,  22.5315,  114.047302, 22.531799], [ 114.027901, 22.5228, 114.026299, 22.5238], [ 114.026299, 22.5238,114.024597,22.5271], [114.024597,  22.5271,114.024696,22.527201]], list('ABCD'))
DF.show()
+----------+-------+----------+---------+
|         A|      B|         C|        D|
+----------+-------+----------+---------+
|114.038696|22.5315|114.047302|22.531799|
|114.027901|22.5228|114.026299|  22.5238|
|114.026299|22.5238|114.024597|  22.5271|
|114.024597|22.5271|114.024696|22.527201|
+----------+-------+----------+---------+

(A, B)

&

(C, D)

是两点的坐标；

column A

和

column C

是经度；

column B

和

column D

是纬度；

我想计算两点之间的地理距离。

我尝试：

from geopy.distance import geodesic
DF = DF.withColumn('Lengths/m', geodesic((['B'],['A']), (['D'],['C'])).m)

然后我收到错误：

类型错误：float() 参数必须是字符串或数字，而不是“列表”

我应该采取哪些不同措施才能成功计算地理距离？

Answer 1

您需要定义一个自定义的用户定义函数：

from geopy.distance import geodesic
import pyspark.sql.functions as F

@F.udf(returnType=FloatType())
def geodesic_udf(a, b):
    return geodesic(a, b).m


DF = DF.withColumn('Lengths/m', geodesic_udf(F.array("B", "A"), F.array("D", "C")))

DF.show()
#+----------+-------+----------+---------+---------+
#|A         |B      |C         |D        |Lengths/m|
#+----------+-------+----------+---------+---------+
#|114.038696|22.5315|114.047302|22.531799|885.94244|
#|114.027901|22.5228|114.026299|22.5238  |198.55937|
#|114.026299|22.5238|114.024597|22.5271  |405.21692|
#|114.024597|22.5271|114.024696|22.527201|15.126849|
#+----------+-------+----------+---------+---------+