ERROR TaskSetManager: Total size of serialized results of 23695 tasks (1024.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

刘超 12天前 ⋅ 137 阅读   编辑

一、描述

  跑spark程序报如下错误

20/07/16 04:03:27 ERROR TaskSetManager: Total size of serialized results of 23695 tasks (1024.1 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)

二、分析

  1、锁定了是spark.driver.maxResultSize引起的,该参数控制worker送回driver的数据大小,一旦操过该限制,driver会终止执行

三、解决方法

  

四、参考文章

  1、https://stackoverflow.com/questions/47996396/total-size-of-serialized-results-of-16-tasks-1048-5-mb-is-bigger-than-spark-dr

  2、https://github.com/awslabs/athena-glue-service-logs/issues/14


注意:本文归作者所有,未经作者允许,不得转载

全部评论: 0

    我有话说: