Spark 2.2的SBT编译失败

Question

我刚刚开始在HDP 2.6上使用Spark 2.2，当我试图进行sbt编译时，我遇到了问题。

错误

[info] 更新了文件 homemaria_devstructuredstreamingprojectbuild.properties: set sbt.version to 1.3.0 [info] 从 homemaria_devstructuredstreamingproject 加载项目定义 [info] 取回 [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts 的工件。获取工件时出错。 [錯誤] https:/repo1.maven.orgmaven2comsquareupokhtt3okhttp-urlconnection3.7.0okhttp-urlconnection-3.7.0.jar。资料来源：下载错误。下载时遇到java.net.UnknownHostException: repo1.maven.org (repo1.maven.org)。https:/repo1.maven.orgmaven2comsquareupokhtt3okhttp-urlconnection3.7.0okhttp-urlconnection-3.7.0.jar。

build.sbt文件如下

buid.sbt

scalaVersion := "2.11.8"

resolvers ++= Seq(
 "Conjars" at "http://conjars.org/repo",
 "Hortonworks Releases" at "http://repo.hortonworks.com/content/groups/public"
)

publishMavenStyle := true

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.2.0.2.6.3.0-235",
  "org.apache.spark" %% "spark-sql" % "2.2.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-spark2" % "4.7.0.2.6.3.0-235",
  "org.apache.phoenix" % "phoenix-core" % "4.7.0.2.6.3.0-235",
  "org.apache.kafka" % "kafka-clients" % "0.10.1.2.6.3.0-235",
  "org.apache.spark" %% "spark-streaming" % "2.0.2" % "provided",
  "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.0.2",
  "org.apache.spark" %% "spark-sql-kafka-0-10" % "2.0.2" % "provided",
  "com.typesafe" % "config" % "1.3.1",
  "com.typesafe.play" %% "play-json" % "2.7.2",
  "com.solarmosaic.client" %% "mail-client" % "0.1.0",
  "org.json4s" %% "json4s-jackson" % "3.2.10",
  "org.apache.logging.log4j" % "log4j-api-scala_2.11" % "11.0",
  "com.databricks" %% "spark-avro" % "3.2.0",
  "org.elasticsearch" %% "elasticsearch-spark-20" % "5.0.0-alpha5",
  "io.spray" %%  "spray-json" % "1.3.3"
)

retrieveManaged := true

fork in run := true

Answer 1

看起来coursier试图从repo1.maven.org获取依赖关系，但被阻止了。Scala-Metals的人给出了解释 此处. 基本上，你必须设置一个全局Coursier配置指向你的企业代理服务器，通过设置一个 mirror.properties 文件，看起来像这样。

central.from=https://repo1.maven.org/maven2
central.to=http://mycorporaterepo.com:8080/nexus/content/groups/public

根据你的操作系统，它将是：

• Windows系统: C:Users/AppData/Roaming/Coursier/config/mirror.properties。
• Linux：~.configcoursiermirror.properties。
• MacOS：~LibraryPreferencesCoursiermirror.properties。

你也可能需要设置SBT使用代理来下载依赖关系。为此，你需要编辑这个文件。

~/.sbt/repositories

将其设置为以下内容

[repositories]
  local
  maven-central: http://mycorporaterepo.com:8080/nexus/content/groups/public

这两个设置的组合应该可以满足你将SBT指向正确位置的所有要求。