如何在 Apache Spark (PySpark) 中将二进制转换为字符串（UUID）而不使用 UDF？

Question

我无法找到一种在不使用 UDF 的情况下将二进制转换为字符串表示形式的方法。有没有一种方法可以使用原生 PySpark 函数而不是 UDF？

from pyspark.sql import DataFrame, SparkSession
import pyspark.sql.functions as F
import uuid
from pyspark.sql.types import Row, StringType

spark_test_instance = (SparkSession
                       .builder
                       .master('local')
                       .getOrCreate())
df: DataFrame = spark_test_instance.createDataFrame([Row()])

df = df.withColumn("id", F.lit(uuid.uuid4().bytes))
df = df.withColumn("length", F.length(df["id"]))

uuidbytes_to_str = F.udf(lambda x: str(uuid.UUID(bytes=bytes(x), version=4)), StringType())

df = df.withColumn("id_str", uuidbytes_to_str(df["id"]))
df = df.withColumn("length_str", F.length(df["id_str"]))

df.printSchema()
df.show(1, truncate=False)

给出：

root
 |-- id: binary (nullable = false)
 |-- length: integer (nullable = false)
 |-- id_str: string (nullable = true)
 |-- length_str: integer (nullable = false)

+-------------------------------------------------+------+------------------------------------+----------+
|id                                               |length|id_str                              |length_str|
+-------------------------------------------------+------+------------------------------------+----------+
|[0A 35 DC 67 13 C8 47 7E B0 80 9F AB 98 CA FA 89]|16    |0a35dc67-13c8-477e-b080-9fab98cafa89|36        |
+-------------------------------------------------+------+------------------------------------+----------+

Answer 1

您可以使用

hex

来获取

id_str

:

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
import uuid

spark = SparkSession.builder.getOrCreate()

data = [(uuid.uuid4().bytes,)]

df = spark.createDataFrame(data, ["id"])

df = df.withColumn("id_str", F.lower(F.hex("id")))

df.show(truncate=False)

# +-------------------------------------------------+--------------------------------+
# |id                                               |id_str                          |
# +-------------------------------------------------+--------------------------------+
# |[8C 76 42 18 BD CA 47 A1 9C D7 9D 74 0C 3C A4 76]|8c764218bdca47a19cd79d740c3ca476|
# +-------------------------------------------------+--------------------------------+

---

更新：

您还可以使用

regexp_replace

来获取准确的

id_str

，格式与您共享的格式相同，如下所示：

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
import uuid

spark = SparkSession.builder.getOrCreate()

data = [(uuid.uuid4().bytes,)]

df = spark.createDataFrame(data, ["id"])

df = df.withColumn(
    "id_str", 
    F.regexp_replace(
        F.lower(F.hex("id")), 
        "(.{8})(.{4})(.{4})(.{4})(.{12})", 
        "$1-$2-$3-$4-$5"
    )
)

df.show(truncate=False)

# +-------------------------------------------------+------------------------------------+
# |id                                               |id_str                              |
# +-------------------------------------------------+------------------------------------+
# |[F8 25 99 3E 2D 7A 40 E4 A1 24 C0 28 B9 30 F6 03]|f825993e-2d7a-40e4-a124-c028b930f603|
# +-------------------------------------------------+------------------------------------+