java.net.MalformedURLException：无协议：将arg传递给spark-submit作业时为null

Question

我有一个小的spark scala工作，需要几个参数，即http_endpoint

import scalaj.http.{Http, HttpOptions}

1 object Main extends App {
2
3   val master = args(0)
4   val http_endpoint = args(1)
5   ...
6   ...
7   
8   def sendRequest = ( text: String ) => {
9      println("#####################: " + http_endpoint)
10     val result = Http(http_endpoint)
11    ...
12    ...
13  }

然后我将其作为火花提交作业发送给纱线。

spark-submit \
--master yarn \
my-job-example-assembly-1.0.0-SNAPSHOT.jar \
yarn \
"http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app"

错误：

#####################: null
Caused by: java.net.MalformedURLException: no protocol: null
    at java.net.URL.<init>(URL.java:611)
    at java.net.URL.<init>(URL.java:508)
    at java.net.URL.<init>(URL.java:457)
    at scalaj.http.HttpRequest.exec(Http.scala:343)
    at scalaj.http.HttpRequest.asString(Http.scala:492)

我什至在发送的URL周围都使用了引号，但失败。当我在行号10的代码中对此进行硬编码时，它可以工作。知道作为arg传递时我如何仍能使它正常工作吗？

Answer 1

[尝试使用--conf spark.driver.args="<args>"传递参数

• 在您的代码中，使用spark.sparkContext.getConf.get()获取参数。

---

spark-submit --master yarn --conf spark.driver.args="yarn http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app" my-job-example-assembly-1.0.0-SNAPSHOT.jar 

Example:

spark-shell --conf spark.driver.args="yarn http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app"

val arg0=spark.sparkContext.getConf.get("spark.driver.args").split("\\s+")(0)
//String = yarn
val arg1=spark.sparkContext.getConf.get("spark.driver.args").split("\\s+")(1)
//String = http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app