Hive Metastore 和 Hiveserver 2的性能上限_hive.server2.thrift.max.worker.threads

metastore 可以建立 10000 个连接，并且每个连接都可以发出请求。
hive-server2 仅能建立 500 个连接，受以下参数的控制。

  <property>
   <name>hive.server2.thrift.max.worker.threads</name>
   <value>500</value>
   <description>Maximum number of Thrift worker threads</description>
 </property>
1
2
3
4
5

但是这些连接不能同时发出请求，受以下两个参数的控制。
hive.server2.async.exec.wait.queue.size是等待队列的长度，HiveServer2 收到请求后，先放到等待队列里。如果队列已满，则抛出异常。
hive.server2.async.exec.threads 设置计算线程的数量。计算线程从等待队列中取请求，进行处理。这些请求不一定都提交到集群上，如 show databases。

<property>
   <name>hive.server2.async.exec.threads</name>
   <value>100</value>
   <description>Number of threads in the async thread pool for HiveServer2</description>
 </property>
 <property>
   <name>hive.server2.async.exec.wait.queue.size</name>
   <value>100</value>
   <description>
     Size of the wait queue for async thread pool in HiveServer2.
     After hitting this limit, the async thread pool will reject new requests.
   </description>
 </property>
1
2
3
4
5
6
7
8
9
10
11
12
13

如果抛出以下错误，则建议调大 hive.server2.async.exec.wait.queue.size。

rg.apache.hive.service.cli.HiveSQLException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6ef9d564 rejected from java.util.concurrent.ThreadPoolExecutor@6e2c02d2[Running, pool size = 100, active threads = 100, queued tasks = 100, completed tasks = 234]
	at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:300)
	at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:286)
	at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:324)
	at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:265)
	at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:497)
	at com.baidu.hive.jdbc.MultiThreadStatementTest.execute(MultiThreadStatementTest.java:82)
	at com.baidu.hive.jdbc.MultiThreadStatementTest.lambda$parallelExecute$0(MultiThreadStatementTest.java:56)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

1
2
3
4
5
6
7
8
9
10
11
12
13
14