hive报错Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTas

一、背景hive导入Hbase的时候采用新建外部表生成HFile,然后bulkload 导入hbase的方式报错出现在hive表数据insert到hive外部表的时候insertoverwriteinto外部表二、问题分析在hive日志中查看报错查看hive_server对应的日志文件目录我的是/var/log/hive然后查看对应节点的log日志找到报错内容主要为Task: attempt_1774000014323_0019_m_000000_0 - exited:java.lang.IllegalArgumentException: Cantreadpartitionsfileat org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:117)at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)at org.apache.hadoop.mapred.MapTask$OldOutputCollector.init(MapTask.java:605)at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)Caused by: java.io.FileNotFoundException: File does not exist: /user/add/hb_range_keys.seq at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1499)at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1492)at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1507)at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1863)at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:304)at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:89)...10more一句话描述为hive执行的sql中set mapreduce.totalorderpartitioner.path/user/add/hb_range_keys.seq;设置后提示File dose not exists /user/add/hb_range_keys.seq三、解决1对于数据量比较小的场景设置 SETmapreduce.job.rduces1;取消/注释掉 SETmapreduce.totalorderpartitioner.path/user/app1/hb_range_keys.seq;2大数据量的场景自动分区不用手动生成文件– 删掉SETmapreduce.totalorderpartitioner.path/user/add/hb_range_keys.seq;SETmapreduce.job.reduces1;改为使用如下参数-- 并行度根据你的集群设置4~20 都可以SETmapreduce.job.reduces8;-- 关键关闭有序分区使用 HBase 默认的 Hash 分区不会报错SEThbase.hfile.output.format.classorg.apache.hadoop.hbase.mapreduce.HFileOutputFormat2;SEThive.hbase.bulk.output.format.classorg.apache.hadoop.hbase.mapreduce.HFileOutputFormat2;