Zeppelin 0.11.1 与 Spark 3.3 - Spark 解释器因 NoSuchFileException 失败

Question

通过 Zeppelin 提交 Spark 作业时收到

InterpreterException

：

org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Fail to launch interpreter process:
ERROR ClientEndpoint: Exception from cluster was: java.nio.file.NoSuchFileException: /zzz/ppp/lll/Zeppelin/zeppelin-0.11.1-bin-all/interpreter/spark/spark-interpreter-0.11.1.jar

文件位于该位置。我更改了所有权并授予其读写访问权限，但最终仍然得到相同的结果。 我检查了 Spark Master，在工作人员中它指出

ava.nio.file.NoSuchFileException: /zzz/ppp/lll/Zeppelin/zeppelin-0.11.1-bin-all/interpreter/spark/spark-interpreter-0.11.1.jar

因此请求正在向 Spark 集群上的 Worker 发出，但随后出现错误。

Answer 1

建议按照以下顺序执行。

1. 在本地安装并解压 Spark 二进制文件

# Install
wget https://archive.apache.org/dist/spark/spark-3.5.2/spark-3.5.2-bin-hadoop3.tgz

# Extract
tar -xvf spark-3.5.2-bin-hadoop3.tgz

2. 创建 Docker Compose 文件 (docker-compose-with-spark.yml)

services:
  zeppelin:
    hostname: zeppelin
    container_name: zeppelin
    image: docker.io/apache/zeppelin:0.11.2
    ports:
      - "8080:8080"
    environment:
      ZEPPELIN_PORT: 8080
      ZEPPELIN_MEM: -Xmx1024m -XX:MaxMetaspaceSize=512m
      SPARK_HOME: /opt/spark
    volumes:
      - ./spark-3.5.2-bin-hadoop3:/opt/spark

  spark-master:
    hostname: spark-master
    container_name: spark-master
    image: docker.io/bitnami/spark:3.5.2
    ports:
      - "18080:8080"
      - "7077:7077"
    environment:
      SPARK_MODE: master
      SPARK_RPC_AUTHENTICATION_ENABLED: no
      SPARK_RPC_ENCRYPTION_ENABLED: no
      SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED: no
      SPARK_SSL_ENABLED: no
      SPARK_USER: spark
      SPARK_MASTER_PORT: 7077
      SPARK_MASTER_WEBUI_PORT: 8080

  spark-worker:
    hostname: spark-worker
    container_name: spark-worker
    image: docker.io/bitnami/spark:3.5.2
    ports:
      - "18081:8081"
    environment:
      SPARK_MODE: worker
      SPARK_MASTER_URL: spark://spark-master:7077
      SPARK_WORKER_MEMORY: 2G
      SPARK_WORKER_CORES: 2
      SPARK_RPC_AUTHENTICATION_ENABLED: no
      SPARK_RPC_ENCRYPTION_ENABLED: no
      SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED: no
      SPARK_SSL_ENABLED: no
      SPARK_USER: spark
      SPARK_WORKER_WEBUI_PORT: 8081
    depends_on:
      - spark-master

3. 运行 Docker Compose

docker compose -f docker-compose-with-spark.yml up

4. 创建并运行 Zeppelin 笔记本

%spark.conf

SPARK_HOME /opt/spark
spark.master spark://spark-master:7077

%spark

val sdf = spark.createDataFrame(Seq((0, "park", 13, 70, "Korea"), (1, "xing", 14, 80, "China"), (2, "john", 15, 90, "USA"))).toDF("id", "name", "age", "score", "country")
sdf.printSchema
sdf.show()

结果

root
 |-- id: integer (nullable = false)
 |-- name: string (nullable = true)
 |-- age: integer (nullable = false)
 |-- score: integer (nullable = false)
 |-- country: string (nullable = true)
+---+----+---+-----+-------+
| id|name|age|score|country|
+---+----+---+-----+-------+
|  0|park| 13|   70|  Korea|
|  1|xing| 14|   80|  China|
|  2|john| 15|   90|    USA|
+---+----+---+-----+-------+