java.sql.BatchUpdateException: Unknown column 'trafficType' in 'field list'

刘超 27天前 ⋅ 177 阅读   编辑

一、描述

  跑spark2.0.1任务,报如下错误

Exception in thread "main" 20/12/23 02:08:00 INFO TaskSetManager: Finished task 94.0 in stage 138.0 (TID 21971) in 99 ms on n44-12.fn.ams.osa (661/9000)
org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 138.0 failed 4 times, most recent failure: Lost task 3.3 in stage 138.0 (TID 21922, n07-13.fn.ams.osa): java.sql.BatchUpdateException: Unknown column 'trafficType' in 'field list'
at com.mysql.jdbc.PreparedStatement.executeBatchedInserts(PreparedStatement.java:1809)
at com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1441)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.savePartition(JdbcUtils.scala:227)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:300)
at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$saveTable$1.apply(JdbcUtils.scala:299)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$28.apply(RDD.scala:902)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1899)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
at org.apache.spark.scheduler.Task.run(Task.scala:86)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

二、分析

  1、数据库与spark dataframe都有trafficType列,按里说,不应该报这个错误
  2、打印出spark执行的insert语句,但捕获不了异常
  3、同事打的包,跑没问题,但为啥我打的包,就有问题,郁闷啊
  4、使用Luyten反编译这两jar包(同事打的是a包,我自己打的是b包),对比这两jar,发现如下两处差异
  a、a包多如下内容
 
  b、a包也多如下内容
 

  看来打的包有问题,这是idea 2020.1.4打的,错哪啦

  5、拷贝同事的.idea(这里改名为idea_new),自己的.idea(这里改名为idea),比较后发现如下差异
 

  6、
  

  

  


注意:本文归作者所有,未经作者允许,不得转载

全部评论: 0

    我有话说: