作用于application的spark升级

刘超 10月前 ⋅ 4835 阅读   编辑

  我司大数据平台由运维维护,平台不仅仅供我们部门使用,其他部门也在用。所以一般不可能为某一部门升级某一组件。但因为一些组件比较老,所以bug比较多,组件升级反而成了解决这些问题的最好方法,下面尝试一下另类的升级

  一般组件升级是作用于平台,想尝试一下只作用于某个application。下面以spark2.0.2升级到spark-2.4.4-bin-hadoop2.7为例

一、在跳板机上安装spark-2.4.4-bin-hadoop2.7

  安装后效果如下

 

二、在spark-submit配置

  1、首先想到的就是配置SPARK-HOME,如下

/usr/bin/spark-submit --master yarn --deploy-mode cluster--queue $HADOOP_QUEUE_NAME --classcom.opera.adx.job.Stat --conf spark.default.parallelism=160 --conf spark.yarn.executor.memoryOverhead=8192 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.app.name=adx_${project}_${day} --conf spark.storage.memoryFraction=0.6 --files=./hive-site.xml --conf spark.shuffle.memoryFraction=0.2 --conf spark.network.timeout=360 --conf spark.yarn.appMasterEnv.SPARK_HOME=/home/sdev/software/spark-2.4.4-bin-hadoop2.7 --conf spark.executorEnv.SPARK_HOME=/home/sdev/software/spark-2.4.4-bin-hadoop2.7 --executor-cores 4 --num-executors 6 --executor-memory 6G --driver-memory 4G --jars $jars./adx_stat_2.11-0.2-SNAPSHOT.jar adx_request${day}rangeofficial --job_names=statGpsDensity --begin_day=20191201

  任务跑完后,查看日志(yarn logs -applicationId application_1574825830961_9836),SPARK-HOME的确变了,如下

 

  但是加载还是2.0.2.2.5.6.0,如下

...
225050692   40 -r-xr-xr-x   1 yarn     hadoop      40753 Jun 26  2017 ./__spark_libs__/spark-unsafe_2.11-2.0.2.2.5.6.0-40.jar
225050788  388 -r-xr-xr-x   1 yarn     hadoop     395195 Jun 26  2017 ./__spark_libs__/javolution-5.5.1.jar
225050725   20 -r-xr-xr-x   1 yarn     hadoop      19827 Jun 26  2017 ./__spark_libs__/opencsv-2.3.jar
225050978  720 -r-xr-xr-x   1 yarn     hadoop     736658 Jun 26  2017 ./__spark_libs__/httpclient-4.5.2.jar
225050925  188 -r-xr-xr-x   1 yarn     hadoop     188671 Jun 26  2017 ./__spark_libs__/commons-beanutils-1.7.0.jar
225050899 2732 -r-xr-xr-x   1 yarn     hadoop    2796935 Jun 26  2017 ./__spark_libs__/parquet-hadoop-bundle-1.6.0.jar
225050775 11464 -r-xr-xr-x   1 yarn     hadoop   11737355 Jun 26  2017 ./__spark_libs__/spark-core_2.11-2.0.2.2.5.6.0-40.jar
225050884 11232 -r-xr-xr-x   1 yarn     hadoop   11499183 Jun 26  2017 ./__spark_libs__/hive-exec-1.2.1.spark2.hdp.1.jar
225050970  220 -r-xr-xr-x   1 yarn     hadoop     223573 Jun 26  2017 ./__spark_libs__/chill_2.11-0.8.0.jar
225050830   56 -r-xr-xr-x   1 yarn     hadoop      54072 Jun 26  2017 ./__spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.2.5.6.0-40.jar
...

  2、从日志中可以看到spark-core_2.11-2.0.2.2.5.6.0-40.jar前面有个__spark_libs__,__spark_libs__是啥

  查看spark2源码,__spark_libs__可以通过spark.yarn.archive指定,默认/hdp/apps/${hdp.version}/spark2/spark2-hdp-yarn-archive.tar.gz;也可以通过spark.yarn.jars指定,不过优先取spark.yarn.archive

  3、下载spark2-hdp-yarn-archive.tar.gz(hadoop fs -get /hdp/apps/2.5.6.0-40/spark2/spark2-hdp-yarn-archive.tar.gz),这里都是spark-2.0.2/jars下的jar包,所以将spark-2.4.4-bin-hadoop2.7/jars下的jar打包,然后上传到hdfs

cd ~/software/spark-2.4.4-bin-hadoop2.7/jars/
tar -czvf ~/liujichao/spark244-hdp-yarn-archive.tar.gz *
hadoop fs -put spark244-hdp-yarn-archive.tar.gz /user/sdev/spark2

  4、设置spark.yarn.archive,再次执行程序,如下

/usr/bin/spark-submit --master yarn --deploy-mode cluster --queue $HADOOP_QUEUE_NAME --classcom.opera.adx.job.Stat --conf spark.default.parallelism=160 --conf spark.yarn.executor.memoryOverhead=8192 --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.app.name=adx_${project}_${day}  --conf spark.storage.memoryFraction=0.6 --files=./hive-site.xml --conf spark.shuffle.memoryFraction=0.2 --conf spark.network.timeout=360 --conf spark.yarn.archive=hdfs://opera/user/sdev/spark2/spark244-hdp-yarn-archive.tar.gz --executor-cores 4 --num-executors 6 --executor-memory 6G --driver-memory 4G --jars $jars./adx_stat_2.11-0.2-SNAPSHOT.jar adx_request${day}rangeofficial --job_names=statGpsDensity --begin_day=20191201

5、任务跑完后,查看日志(yarn logs -applicationId application_1574825830961_10569),加载的的确是spark-core_2.11-2.4.4.jar等jar包(由于<spark安装目录>/jars文件夹中包含227个jar,下面就不一一列出了。这样配置后,该application会使用这227个jar包),如下

...
68091933  256 -r-xr-xr-x   1 yarn     hadoop     258370 Aug 27 21:30 ./__spark_libs__/calcite-avatica-1.2.0-incubating.jar
68092100  692 -r-xr-xr-x   1 yarn     hadoop     708187 Aug 27 21:30 ./__spark_libs__/spark-graphx_2.11-2.4.4.jar
68092048    4 -r-xr-xr-x   1 yarn     hadoop       3817 Aug 27 21:30 ./__spark_libs__/kubernetes-model-common-4.1.2.jar
68091942   40 -r-xr-xr-x   1 yarn     hadoop      37871 Aug 27 21:30 ./__spark_libs__/commons-compiler-3.0.9.jar
68092119 9888 -r-xr-xr-x   1 yarn     hadoop   10121868 Aug 27 21:30 ./__spark_libs__/spire_2.11-0.13.0.jar
68091921   80 -r-xr-xr-x   1 yarn     hadoop      79912 Aug 27 21:30 ./__spark_libs__/api-util-1.0.0-M20.jar
68092099 13212 -r-xr-xr-x   1 yarn     hadoop   13527109 Aug 27 21:30 ./__spark_libs__/spark-core_2.11-2.4.4.jar
68091940  280 -r-xr-xr-x   1 yarn     hadoop     284184 Aug 27 21:30 ./__spark_libs__/commons-codec-1.10.jar
...

  升级后,这种方式操作hive ,有如下问题:

  1、(已解决)org.apache.spark.sql.AnalysisException: The format of the existing table r_day_stat_gps_density is `HiveFileFormat`. It doesn't match the specified format `ParquetFileFormat`.

  2、(已解决)java.lang.ClassNotFoundException: org.apache.tez.dag.api.SessionNotRunning

  3、(未解决)org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration

  4、(未解决)ERROR ApplicationMaster: User class threw exception: java.lang.reflect.InvocationTargetExceptionorg.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration

  5、(未解决)ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() on RPC id 6213350919299289336 java.io.StreamCorruptedException: invalid stream header: 0001000B

  总结:hive引擎是mr没问题,是tez就有问题;编译环境中的spark与执行环境的spark版本需要保持一致


注意:本文归作者所有,未经作者允许,不得转载

全部评论: 0

    我有话说: