如何使用StandaloneAhcWSClient发出大量GET请求

Question

如何在Scala中使用StandaloneAhcWSClient进行大量简单的get请求？ （这是与Play2框架捆绑在一起的默认http客户端）。在我的情况下，我得到了~100K GET请求到外部API。 Future.traverse()不削减它，是否有更好的方法，也许是一种处理网址列表的方式？

这是我现在的代码：https://scastie.scala-lang.org/HgrIyR23TmG12j3MzMCxUw

它可以在列表中使用一定数量的URL，但是有大量的URL，但有一个例外java.lang.IllegalStateException: failed to create a child event loop

Answer 1

这是我最终得到的：

import play.api.libs.json.JsValue
import akka.actor.ActorSystem

import scala.concurrent.{Await, Future}
import scala.concurrent.duration.Duration
import scala.concurrent.ExecutionContext.Implicits.global
import play.api.libs.ws.StandaloneWSClient
import akka.stream.ActorMaterializer
import play.api.libs.ws.ahc.StandaloneAhcWSClient
import play.api.libs.ws.JsonBodyReadables._
import play.api.libs.json._

implicit val system: ActorSystem = ActorSystem()
system.registerOnTermination {
  System.exit(0)
}
implicit val materializer: ActorMaterializer = ActorMaterializer()

def getAllRecApiResponses(urls: List[String])(
  implicit actorSystem: ActorSystem,
  materializer: ActorMaterializer): List[JsValue] = {
  implicit val wsClient: StandaloneWSClient = StandaloneAhcWSClient()

  val res: Future[List[JsValue]] = Future.traverse(urls)(urlString => {
    wsClient.url(urlString).get().map(_.body[JsValue]).recover {
      case ex: Exception => {
        println( s"Url call returned exception for url $urlString: $ex" )
        JsNull
      }
    }
  }) andThen { case _ => wsClient.close() }

  Await.result(res, Duration.Inf)
}

val result = getAllRecApiResponses(List.fill(10)("https://jsonplaceholder.typicode.com/todos/1"))

result foreach println

使用以下build.sbt：

scalaVersion := "2.11.12"

val liftVersion = "2.6"

libraryDependencies ++= Seq(
  "com.typesafe.play" %% "play-ahc-ws-standalone" % "2.0.3",
  "com.typesafe.play" %% "play-ws-standalone-json" % "2.0.3",
  "com.typesafe.play" %% "play-json" % "2.7.2"
)