java.lang.RuntimeException: Unsupported literal type class [Ljava.lang.String; [Ljava.lang.String;@2fa1f38b

刘超 22天前 ⋅ 156 阅读   编辑

一、描述

  执行如下spark代码,报如下错误

scala> import spark.implicits._
import spark.implicits._

scala> var df = Seq(
     |   (1, "a3603080413440", 23),
     |   (2, "a2973202622656", 45)
     | ).toDF("id", "adId", "priceInLi")
df: org.apache.spark.sql.DataFrame = [id: int, adId: string ... 1 more field]
scala> df = df.withColumn("priceInLi",when($"adId".isin(Array[String]("a3603080413440")),lit(0)).otherwise($"priceInLi"))
java.lang.RuntimeException: Unsupported literal type class [Ljava.lang.String; [Ljava.lang.String;@2fa1f38b
  at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:57)
  at org.apache.spark.sql.functions$.lit(functions.scala:101)
  at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:773)
  at org.apache.spark.sql.Column$$anonfun$isin$1.apply(Column.scala:773)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
  at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
  at scala.collection.AbstractTraversable.map(Traversable.scala:104)
  at org.apache.spark.sql.Column.isin(Column.scala:773)
  ... 52 elided

二、解决方法

  将数组中每个元素作为参数(:_*)传入isin方法即可,如下

scala> df = df.withColumn("priceInLi",when($"adId".isin(Array[String]("a3603080413440"):_*),0D).otherwise($"priceInLi"))
df: org.apache.spark.sql.DataFrame = [id: int, adId: string ... 1 more field]

scala> df.show(false)
+---+--------------+---------+
|id |adId          |priceInLi|
+---+--------------+---------+
|1  |a3603080413440|0.0      |
|2  |a2973202622656|45.0     |
+---+--------------+---------+

三、参考文章

  1、Scala可变长度参数与:_*使用


注意:本文归作者所有,未经作者允许,不得转载

全部评论: 0

    我有话说: