devbox
秋刀鱼在做梦
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

CDH 6.3Unable to instantiate SparkSession with Hive support because Hive classes are not found.解决办法

作者：秋刀鱼在做梦 | 2024-07-23 11:51:44

赞

踩

unable to instantiate sparksession with hive support because hive classes ar

系统环境

CentOS 7.7.1908
CDH 6.3.1

问题现象

执行spark-submit，具体命令如下：

spark-submit --class xxx  --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue  /tmp/xxx.jar

  • 1

console输出如下：

2020-06-25 17:59:56 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-25 17:59:56 INFO  RMProxy:133 - Connecting to ResourceManager at master01-prod.cdh/10.0.101.86:8032
2020-06-25 17:59:56 INFO  Client:57 - Requesting a new application from cluster with 3 NodeManagers
2020-06-25 17:59:56 INFO  Configuration:2662 - resource-types.xml not found
2020-06-25 17:59:56 INFO  ResourceUtils:419 - Unable to find 'resource-types.xml'.
2020-06-25 17:59:56 INFO  Client:57 - Verifying our application has not requested more than the maximum memory capability of the cluster (13458 MB per container)
2020-06-25 17:59:56 INFO  Client:57 - Will allocate AM container, with 4505 MB memory including 409 MB overhead
2020-06-25 17:59:56 INFO  Client:57 - Setting up container launch context for our AM
2020-06-25 17:59:56 INFO  Client:57 - Setting up the launch environment for our AM container
2020-06-25 17:59:56 INFO  Client:57 - Preparing resources for our AM container
2020-06-25 17:59:56 WARN  Client:69 - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
2020-06-25 17:59:57 INFO  Client:57 - Uploading resource file:/tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83/__spark_libs__3873912406489385752.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/__spark_libs__3873912406489385752.zip
2020-06-25 17:59:58 INFO  Client:57 - Uploading resource file:/tmp/dopEtlDev.jar -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/dopEtlDev.jar
2020-06-25 17:59:58 INFO  Client:57 - Uploading resource file:/tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83/__spark_conf__8206675857340680660.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0063/__spark_conf__.zip
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing view acls to: root
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing modify acls to: root
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing view acls groups to:
2020-06-25 17:59:58 INFO  SecurityManager:57 - Changing modify acls groups to:
2020-06-25 17:59:58 INFO  SecurityManager:57 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
2020-06-25 17:59:58 INFO  HiveConf:188 - Found configuration file jar:file:/tmp/dopEtlDev.jar!/hive-site.xml
2020-06-25 17:59:58 INFO  YARNHadoopDelegationTokenManager:57 - Attempting to load user's ticket cache.
2020-06-25 17:59:58 INFO  Client:57 - Submitting application application_1591413191159_0063 to ResourceManager
2020-06-25 17:59:58 INFO  YarnClientImpl:310 - Submitted application application_1591413191159_0063
2020-06-25 17:59:59 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 17:59:59 INFO  Client:57 -
	 client token: N/A
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.thequeue
	 start time: 1593079198932
	 final status: UNDEFINED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0063/
	 user: root
2020-06-25 18:00:00 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:01 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:02 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:03 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:04 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:05 INFO  Client:57 - Application report for application_1591413191159_0063 (state: ACCEPTED)
2020-06-25 18:00:06 INFO  Client:57 - Application report for application_1591413191159_0063 (state: FAILED)
2020-06-25 18:00:06 INFO  Client:57 -
	 client token: N/A
	 diagnostics: Application application_1591413191159_0063 failed 2 times due to AM Container for appattempt_1591413191159_0063_000002 exited with  exitCode: 13
Failing this attempt.Diagnostics: [2020-06-25 18:00:06.105]Exception from container-launch.
Container id: container_1591413191159_0063_02_000001
Exit code: 13

[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


For more detailed output, check the application tracking page: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063 Then click on links to logs of each attempt.
. Failing the application.
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.thequeue
	 start time: 1593079198932
	 final status: FAILED
	 tracking URL: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063
	 user: root
2020-06-25 18:00:06 ERROR Client:73 - Application diagnostics message: Application application_1591413191159_0063 failed 2 times due to AM Container for appattempt_1591413191159_0063_000002 exited with  exitCode: 13
Failing this attempt.Diagnostics: [2020-06-25 18:00:06.105]Exception from container-launch.
Container id: container_1591413191159_0063_02_000001
Exit code: 13

[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


[2020-06-25 18:00:06.106]Container exited with a non-zero exit code 13. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data4/yarn/nm/usercache/root/filecache/57/__spark_libs__3873912406489385752.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


For more detailed output, check the application tracking page: http://master01-prod.cdh:8088/cluster/app/application_1591413191159_0063 Then click on links to logs of each attempt.
. Failing the application.
Exception in thread "main" org.apache.spark.SparkException: Application application_1591413191159_0063 finished with failed status
	at org.apache.spark.deploy.yarn.Client.run(Client.scala:1158)
	at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1606)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
2020-06-25 18:00:06 INFO  ShutdownHookManager:57 - Shutdown hook called
2020-06-25 18:00:06 INFO  ShutdownHookManager:57 - Deleting directory /tmp/spark-15956e0f-d264-4042-8968-1a3ba0b696a9
2020-06-25 18:00:06 INFO  ShutdownHookManager:57 - Deleting directory /tmp/spark-44972a1d-fb2d-443b-b6c8-d94ae7d4dc83


  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83
  • 84
  • 85
  • 86
  • 87
  • 88
  • 89
  • 90
  • 91
  • 92
  • 93
  • 94
  • 95
  • 96
  • 97
  • 98
  • 99
  • 100
  • 101
  • 102
  • 103
  • 104
  • 105
  • 106
  • 107
  • 108
  • 109
  • 110
  • 111
  • 112
  • 113
  • 114
  • 115
  • 116
  • 117
  • 118

解决方案

1.查看yarn日志
访问：http://master01-prod.cdh:19888/jobhistory/logs//slave01-prod.cdh:8041/container_1591413191159_0057_02_000001/container_1591413191159_0057_02_000001/root

具体日志如下：

Log Type: prelaunch.err

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 0


Log Type: prelaunch.out

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 70

Setting up env variables
Setting up job resources
Launching container

Log Type: stderr

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 544

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data2/yarn/nm/usercache/root/filecache/42/__spark_libs__2999614218847688045.zip/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Log Type: stdout

Log Upload Time: Thu Jun 25 16:40:56 +0800 2020

Log Length: 5141

Showing 4096 bytes of 5141 total. Click here for the full log.

-25 16:40:55 INFO  ApplicationMaster:57 - Waiting for spark context initialization...
arg is not exist and use System currentTimeMillis!
2020-06-25 16:40:55 ERROR ApplicationMaster:94 - User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
	at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
2020-06-25 16:40:55 INFO  ApplicationMaster:57 - Final app status: FAILED, exitCode: 13, (reason: User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
	at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
)
2020-06-25 16:40:55 ERROR ApplicationMaster:94 - Uncaught exception: 
org.apache.spark.SparkException: Exception thrown in awaitResult: 
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:226)
	at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:447)
	at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:275)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:805)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:804)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
	at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:804)
	at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)
Caused by: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
	at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:878)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value$.main(dm_scfp_num_type_value.scala:34)
	at com.mskj.dopdev.scfp.dm_scfp_num_type_value.main(dm_scfp_num_type_value.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673)
2020-06-25 16:40:55 INFO  ApplicationMaster:57 - Deleting staging directory hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0057
2020-06-25 16:40:55 WARN  NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-06-25 16:40:55 INFO  ShutdownHookManager:57 - Shutdown hook called

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80
  • 81
  • 82
  • 83

重点关注：reason: User class threw exception: java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found
初步分析是缺少hive相关的jar

2.检查hive相关jar包
找一台可以正常submit的服务器，对比分析发现spark所需要的jar包都在对应位置，因此排除缺少jar包问题

3.检查spark配置
正常服务器的spark配置：
在这里插入图片描述

异常服务器的spark配置：
在这里插入图片描述
这下真相大白了，spark-conf的配置不同

4.unlink spark-conf ，创建正确的软链接

5.验证问题是否解决

[root@master02-prod alternatives]# spark-submit --class xxxx  --master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 --queue thequeue  /tmp/xxxx.jar
20/06/25 18:06:55 INFO client.RMProxy: Connecting to ResourceManager at master01-prod.cdh/10.0.101.86:8032
20/06/25 18:06:55 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
20/06/25 18:06:55 INFO conf.Configuration: resource-types.xml not found
20/06/25 18:06:55 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/06/25 18:06:55 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (13458 MB per container)
20/06/25 18:06:55 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
20/06/25 18:06:55 INFO yarn.Client: Setting up container launch context for our AM
20/06/25 18:06:55 INFO yarn.Client: Setting up the launch environment for our AM container
20/06/25 18:06:55 INFO yarn.Client: Preparing resources for our AM container
20/06/25 18:06:55 INFO yarn.Client: Uploading resource file:/tmp/dopEtlDev.jar -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0064/dopEtlDev.jar
20/06/25 18:06:55 INFO yarn.Client: Uploading resource file:/tmp/spark-410edde9-c2d8-4fd4-9a12-1f7473cc89f5/__spark_conf__1869574925881857964.zip -> hdfs://master01-prod.cdh:8020/user/root/.sparkStaging/application_1591413191159_0064/__spark_conf__.zip
20/06/25 18:06:55 INFO spark.SecurityManager: Changing view acls to: root
20/06/25 18:06:55 INFO spark.SecurityManager: Changing modify acls to: root
20/06/25 18:06:55 INFO spark.SecurityManager: Changing view acls groups to:
20/06/25 18:06:55 INFO spark.SecurityManager: Changing modify acls groups to:
20/06/25 18:06:55 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
20/06/25 18:06:56 INFO conf.HiveConf: Found configuration file file:/etc/hive/conf.cloudera.hive/hive-site.xml
20/06/25 18:06:56 INFO security.YARNHadoopDelegationTokenManager: Attempting to load user's ticket cache.
20/06/25 18:06:56 INFO yarn.Client: Submitting application application_1591413191159_0064 to ResourceManager
20/06/25 18:06:56 INFO impl.YarnClientImpl: Submitted application application_1591413191159_0064
20/06/25 18:06:57 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:06:57 INFO yarn.Client:
	 client token: N/A
	 diagnostics: AM container is launched, waiting for AM container to Register with RM
	 ApplicationMaster host: N/A
	 ApplicationMaster RPC port: -1
	 queue: root.thequeue
	 start time: 1593079616105
	 final status: UNDEFINED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
	 user: root
20/06/25 18:06:58 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:06:59 INFO yarn.Client: Application report for application_1591413191159_0064 (state: ACCEPTED)
20/06/25 18:07:00 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:00 INFO yarn.Client:
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: slave03-prod.cdh
	 ApplicationMaster RPC port: 40442
	 queue: root.thequeue
	 start time: 1593079616105
	 final status: UNDEFINED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
	 user: root
20/06/25 18:07:01 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:02 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:03 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:04 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:05 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:06 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:07 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:08 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:09 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:10 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:11 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:12 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:13 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:14 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:15 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:16 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:17 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:18 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:19 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:20 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:21 INFO yarn.Client: Application report for application_1591413191159_0064 (state: RUNNING)
20/06/25 18:07:22 INFO yarn.Client: Application report for application_1591413191159_0064 (state: FINISHED)
20/06/25 18:07:22 INFO yarn.Client:
	 client token: N/A
	 diagnostics: N/A
	 ApplicationMaster host: slave03-prod.cdh
	 ApplicationMaster RPC port: 40442
	 queue: root.thequeue
	 start time: 1593079616105
	 final status: SUCCEEDED
	 tracking URL: http://master01-prod.cdh:8088/proxy/application_1591413191159_0064/
	 user: root
20/06/25 18:07:22 INFO util.ShutdownHookManager: Shutdown hook called
20/06/25 18:07:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-410edde9-c2d8-4fd4-9a12-1f7473cc89f5
20/06/25 18:07:22 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-e847fe28-3bd5-4c74-bd1b-2b8fc7e5c4e6

  • 1
  • 2
  • 3
  • 4
  • 5
  • 6
  • 7
  • 8
  • 9
  • 10
  • 11
  • 12
  • 13
  • 14
  • 15
  • 16
  • 17
  • 18
  • 19
  • 20
  • 21
  • 22
  • 23
  • 24
  • 25
  • 26
  • 27
  • 28
  • 29
  • 30
  • 31
  • 32
  • 33
  • 34
  • 35
  • 36
  • 37
  • 38
  • 39
  • 40
  • 41
  • 42
  • 43
  • 44
  • 45
  • 46
  • 47
  • 48
  • 49
  • 50
  • 51
  • 52
  • 53
  • 54
  • 55
  • 56
  • 57
  • 58
  • 59
  • 60
  • 61
  • 62
  • 63
  • 64
  • 65
  • 66
  • 67
  • 68
  • 69
  • 70
  • 71
  • 72
  • 73
  • 74
  • 75
  • 76
  • 77
  • 78
  • 79
  • 80

参考资料

  1. https://java.wekeepcoding.com/article/11916051/Unable+to+instantiate+SparkSession+with+Hive+support+because+Hive+classes+are+not+found
  2. https://blog.csdn.net/weixin_43552807/article/details/105209467?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522159307823319725250158297%2522%252C%2522scm%2522%253A%252220140713.130102334…%2522%257D&request_id=159307823319725250158297&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2allsobaiduend~default-1-105209467.ecpm_v1_rank_business_v1&utm_term=Unable+to+instantiate+SparkSes
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/秋刀鱼在做梦/article/detail/869718
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号