从传递文本文件加载列表SQL查询中星火SQL

Question

我已阅读从使用,作为分隔符的文本文件帐户：

val csv = spark.read.text("src/main/resources/in/insight/account_issues.txt")

//implicits
import spark.sqlContext.implicits._

val string_account = csv.map(_.getString(0)).collect.toList.toString()
//print(string_account)

val query = s"""(SELECT
               |    ACCOUNT_NUMBER,
               |    CASE WHEN STMT.CRF_TYPE='CREDIT' THEN STMT.AMOUNT_LCY
               |        ELSE NULL
               |    END as 'CreditAmount',
               |    CASE WHEN STMT.CRF_TYPE='DEBIT' THEN STMT.AMOUNT_LCY
               |        ELSE  NULL
               |    END as 'DebitAmount',
               |    STMT.BOOKING_DATE,
               |    STMT.VALUE_DATE,
               |    CRF_TYPE
               |FROM [InsightLanding].[dbo].[v_STMT_ENTRY] AS STMT
               |    LEFT JOIN [InsightWarehouse].[dbo].[v_Account] AS A ON a.AccountNum = STMT.ACCOUNT_NUMBER
               |
               |WHERE STMT.MIS_DATE='$BusinessDate'
               | AND STMT.ACCOUNT_NUMBER IN ($string_account) ) tmp """.stripMargin

val responseWithSelectedColumns = spark
  .read
  .format("jdbc")
  .option("url", url)
  .option("driver", driver)
  .option("dbtable", query)
  .load()

我不能得到的作品，而不是得到错误：

: 'List' is not a recognized built-in function name

什么是从我的代码错了吗？

Answer 1

当您创建string_account您在列表中使用toString()。这会给你一个字符串List(...)，例如：

scala> List(1,2,3).toString()
res0: String = List(1, 2, 3)

要改用什么是mkString(",")：

scala> List(1,2,3).mkString(",")
res1: String = "1,2,3"

在这种情况下，这将是：

val string_account = csv.map(_.getString(0)).collect.toList.mkString(",")

注意：您可以轻松地添加括号而不是string_account如果想与mkString("(", ",", ")")让他们在SQL查询。