Hive 源码_对于filterholder(java.lang.class

hive 编译

• issue

Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.5.1.1:run (default) on project hive-standalone-metastore: Error resolving artifact: com.google.protobuf:protoc:2.5.0: The following artifacts could not be resolved: com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0 (absent): Could not transfer artifact com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0 from/to maven-default-http-blocker (http://0.0.0.0/)

可以增加编译选项

 -Dos.arch=x86_64

1
2

可参见
https://qileq.com/article/202202230001/

• issue

[ERROR] Failed to execute goal on project hive-service: Could not resolve dependencies for project org.apache.hive:hive-service:jar:3.1.3: Failed to collect dependencies at org.apache.directory.server:apacheds-server-integ:jar:1.5.6 -> org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: Failed to read artifact descriptor for org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: The following artifacts could not be resolved: org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT (absent): Could not transfer artifact org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [tbds (http://tbdsrepo.oa.com/repository/tbds/, default, disabled),

添加 repo

<repository>
    <id>Codehaus repository</id>
    <name>codehaus-mule-repo</name>
    <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>
    <layout>default</layout>
</repository>
1
2
3
4
5
6

该问题，可以参见 这个 hive issue ： https://issues.apache.org/jira/browse/HIVE-21777

• issue

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project hive-webhcat: Compilation failure
[ERROR] hive/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java:[258,31] 对于FilterHolder(java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>), 找不到合适的构造器
[ERROR]     构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(org.eclipse.jetty.servlet.Source)不适用
[ERROR]       (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为org.eclipse.jetty.servlet.Source)
[ERROR]     构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(java.lang.Class<? extends javax.servlet.Filter>)不适用
[ERROR]       (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为java.lang.Class<? extends javax.servlet.Filter>)
[ERROR]     构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(javax.servlet.Filter)不适用
[ERROR]       (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为javax.servlet.Filter)
[ERROR] 
1
2
3
4
5
6
7
8
9

将 maven 版本 从 3.9.4 降低到 3.2.5 后该问题解决

Hive 写入

即使 设置了 set hive.exec.mode.local.auto=true; 也会回滚，当 input 文件较多时。

Query ID = hadoop_20240109112435_62531b28-9651-4d82-968d-4ff26142326d
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Cannot run job locally: Number of Input Files (= 7) is larger than hive.exec.mode.local.auto.input.files.max(= 4)
1
2
3
4
5
6
7
8
9
10
11

Hive 写数据

动态分区写入数据

CREATE TABLE htable1(a string, b string)  PARTITIONED BY ( p string ) location "file:///home/hadoop/verify/htable1";
INSERT INTO htable1 VALUES('a2', 'b2', 'p1');
1
2

相关代码区域

if (!isTxnTable && ((loadFileType == LoadFileType.REPLACE_ALL) || (oldPart == null && !isAcidIUDoperation))) {
          //for fullAcid tables we don't delete files for commands with OVERWRITE - we create a new
          // base_x.  (there is Insert Overwrite and Load Data Overwrite)
          boolean isAutoPurge = "true".equalsIgnoreCase(tbl.getProperty("auto.purge"));
          boolean needRecycle = !tbl.isTemporary()
                  && ReplChangeManager.isSourceOfReplication(Hive.get().getDatabase(tbl.getDbName()));
          replaceFiles(tbl.getPath(), loadPath, destPath, oldPartPath, getConf(), isSrcLocal,
              isAutoPurge, newFiles, FileUtils.HIDDEN_FILES_PATH_FILTER, needRecycle, isManaged);
        } else {
//          FileSystem fs = tbl.getDataLocation().getFileSystem(conf);
          FileSystem fs = destPath.getFileSystem(conf);
          copyFiles(conf, loadPath, destPath, fs, isSrcLocal, isAcidIUDoperation,
              (loadFileType == LoadFileType.OVERWRITE_EXISTING), newFiles,
              tbl.getNumBuckets() > 0, isFullAcidTable, isManaged);
        }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

可见 动态分区的临时目录.hive-staging_hive，在 table location 里。

file:/home/hadoop/verify/htable1/.hive-staging_hive_2024-01-09_16-03-00_752_1852953438530261083-1/-ext-10000/p=p1
1

指定分区

insert into htable1 partition (p='p1') VALUES('a1', 'b1') ;
1

指定分区的临时目录.hive-staging_hive，既不在原文件系统的分区里，也不在 table location 里的指定分区里。
而是跨系统的 绝对路径里，但是 scheme 换成了 table location 的scheme 的 filesystem。

Copy files between FileSystems

Copy files between FileSystems.FileUtil

  /** Copy files between FileSystems. */
  public static boolean copy(FileSystem srcFS, FileStatus srcStatus,
                             FileSystem dstFS, Path dst,
                             boolean deleteSource,
                             boolean overwrite,
                             Configuration conf) throws IOException {
    Path src = srcStatus.getPath();
    dst = checkDest(src.getName(), dstFS, dst, overwrite);
    if (srcStatus.isDirectory()) {
      checkDependencies(srcFS, src, dstFS, dst);
      if (!dstFS.mkdirs(dst)) {
        return false;
      }
      FileStatus contents[] = srcFS.listStatus(src);
      for (int i = 0; i < contents.length; i++) {
        copy(srcFS, contents[i], dstFS,
             new Path(dst, contents[i].getPath().getName()),
             deleteSource, overwrite, conf);
      }
    } else {
      InputStream in=null;
      OutputStream out = null;
      try {
        in = srcFS.open(src);
        out = dstFS.create(dst, overwrite);
        IOUtils.copyBytes(in, out, conf, true);
      } catch (IOException e) {
        IOUtils.closeStream(out);
        IOUtils.closeStream(in);
        throw e;
      }
    }
    if (deleteSource) {
      return srcFS.delete(src, true);
    } else {
      return true;
    }

  }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39