devbox
Guff_9hys
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

Flink Iceberg 测试_oceanus 测试flink mysql到iceberg性能测试

作者：Guff_9hys | 2024-08-19 08:05:27

赞

踩

oceanus 测试flink mysql到iceberg性能测试

组件版本

组件

版本

Java

1.8.251

Scala

1.12.14

Flink

1.12.5

Iceberg

0.12.0

Hadoop

2.9.2

Hive

2.3.6

将hdfs-site.xml，core-site.xml，hive-site.xml放入resources下

hadoop_catalog

  1. object TestFlinkSQLOptIcebergHadoopCatalog {
  2.   def main(args: Array[String]): Unit = {
  3.     //   val env = StreamExecutionEnvironment.getExecutionEnvironment
  4.     val settings = EnvironmentSettings
  5.       .newInstance()
  6.       .useBlinkPlanner()
  7.       .inBatchMode()
  8.       .build()
  9.  
  10.     val tableEnv = TableEnvironment.create(settings)
  11.  
  12.     val DDL =
  13.       """
  14.         |create catalog hadoop_catalog with (
  15.         | 'type' = 'iceberg',
  16.         | 'catalog-type' = 'hadoop',
  17.         | 'property-version' = '1',
  18.         | 'warehouse' = 'hdfs:///user/hive/warehouse/'
  19.         |)
  20.         |""".stripMargin
  21.     tableEnv.executeSql(DDL)
  22.  
  23.     //    两种写法
  24.     //    tableEnv.executeSql("use catalog hadoop_catalog")
  25.     tableEnv.useCatalog("hadoop_catalog")
  26.  
  27.     tableEnv.executeSql("create database if not exists iceberg_db")
  28.     //    tableEnv.executeSql("use iceberg_db")
  29.     tableEnv.useDatabase("iceberg_db")
  30.  
  31.     tableEnv.executeSql("show databases").print()
  32.     tableEnv.executeSql("show tables").print()
  33.  
  34.     // 1. 创建表
  35.     //    val tableDDL =
  36.     //      """
  37.     //        |create table if not exists iceberg_test_table (
  38.     //        | id bigint comment 'unique id',
  39.     //        | data string
  40.     //        |) comment 'iceberg test table'
  41.     //        | partitioned by (data)
  42.     //        |""".stripMargin
  43.     //    tableEnv.executeSql(tableDDL)
  44.     //    tableEnv.executeSql("show tables").print()
  45.  
  46.     // *** 2. 修改表名，暂不支持 hadoop catalog 更改表名等操作,当前仅支持更新设置属性和删除
  47.     //    tableEnv.executeSql("alter table iceberg_test_table rename to iceberg_test_table2")
  48.     //    tableEnv.executeSql("show tables").print()
  49.  
  50.     // 3. 删除表
  51.     //    tableEnv.executeSql("drop table if exists iceberg_test_table")
  52.     //    tableEnv.executeSql("show tables").print()
  53.  
  54.     // 4. 查询表
  55.     //    tableEnv.executeSql("show tables").print()
  56.  
  57.     // 5. like 根据已有表创建新表
  58.     tableEnv.executeSql("create table iceberg_test_like like iceberg_test_table")
  59.     tableEnv.executeSql("show tables").print()
  60.  
  61.     // 6. 修改表属性
  62.     // flink1.11后支持
  63.     //    tableEnv.executeSql("""alter table test_like set ('write.format.default'='avro')""")
  64.  
  65.     // 7. 写入
  66.     //    tableEnv.executeSql("insert into test_hadoop_table values (1, 'a')")
  67.     //    tableEnv.executeSql("insert overwrite test_hadoop_table values (2, 'a') ")
  68.     //    tableEnv.executeSql("insert overwrite test_table PARTITION(data='b') SELECT 6")
  69.  
  70.     // 8. 读取数据
  71.     //    tableEnv.executeSql("select * from test_hadoop_table").print()
  72.  
  73.     // 9. 写入数据
  74.     //    val insert =
  75.     //      """
  76.     //        |insert into test_like
  77.     //        |select
  78.     //        | id, data
  79.     //        |from test_hadoop_table
  80.     //        |""".stripMargin
  81.     //    tableEnv.executeSql(insert)
  82.   }
  83. }

hive_catalog

  1. object TestFlinkSQLOptIcebergHiveCatalog {
  2.   private var logger: org.slf4j.Logger = _
  3.  
  4.   def main(args: Array[String]): Unit = {
  5.     logger = LoggerFactory.getLogger(this.getClass.getSimpleName)
  6.     Logger.getLogger("org.apache").setLevel(Level.INFO)
  7.     Logger.getLogger("hive.metastore").setLevel(Level.INFO)
  8.     Logger.getLogger("akka").setLevel(Level.INFO)
  9.  
  10.     val tableEnv = FlinkUtils.initStreamTableEnvironment()
  11.  
  12.     //    val env = StreamExecutionEnvironment.getExecutionEnvironment
  13.     //
  14.     //    val settings = EnvironmentSettings
  15.     //      .newInstance()
  16.     //      .useBlinkPlanner()
  17.     //      .inStreamingMode()
  18.     //      .build()
  19.     //
  20.       // streamTable 环境
  21.     //    val tableEnv = StreamTableEnvironment.create(env, settings)
  22.  
  23.     // batchTable 环境
  24.     //    val settings = EnvironmentSettings
  25.     //      .newInstance()
  26.     //      .useBlinkPlanner()
  27.     //      .inBatchMode()
  28.     //      .build()
  29.     //    val tableEnv = TableEnvironment.create(settings)
  30.  
  31.     //        val catalog_name = "hive_catalog"
  32.     //        val database = "iceberg_test_db"
  33.     //        val hiveConf = "F:\\workspace\\realtime-lakehouse\\test\\src\\main\\resources"
  34.     //
  35.     //        val hiveCatalog = new HiveCatalog(
  36.     //          catalog_name,
  37.     //          null,
  38.     //          hiveConf
  39.     //        )
  40.     //        tableEnv.registerCatalog(catalog_name, hiveCatalog)
  41.     //        tableEnv.getConfig.setSqlDialect(SqlDialect.HIVE)
  42.  
  43.     // catalog
  44.     //    val catalogDDL =
  45.     //      """
  46.     //        |create catalog hive_catalog with (
  47.     //        | 'type' = 'iceberg',
  48.     //        | 'catalog-type' = 'hive',
  49.     //        | 'uri' = 'thrift://test-lakehouse:9083',
  50.     //        | 'clients' = '5',
  51.     //        | 'property-version' = '1',
  52.     //        | 'warehouse' = 'hdfs://test-lakehouse:9000/user/hive/warehouse/'
  53.     //        |)
  54.     //        |""".stripMargin
  55.     //    tableEnv.executeSql(catalogDDL)
  56.  
  57.     //    两种写法
  58.     //    //    tableEnv.executeSql("use catalog hive_catalog")
  59.     //    tableEnv.useCatalog("hive_catalog")
  60.     //    tableEnv.executeSql("show catalogs").print()
  61.     //
  62.     //    val databaseDDL = "create database if not exists iceberg_test_db"
  63.     //    tableEnv.executeSql(databaseDDL)
  64.     //
  65.     //    tableEnv.useDatabase("iceberg_test_db")
  66.     //    println(s"current database: ${tableEnv.getCurrentDatabase}")
  67.  
  68.     //    tableEnv.executeSql("show databases").print()
  69.  
  70.     //    println("list catalogs:")
  71.     //    tableEnv.listCatalogs().foreach(println)
  72.     //    tableEnv.listDatabases()
  73.  
  74.     //    1. 创建表
  75.     //    val tableDDL =
  76.     //      """
  77.     //        |create table if not exists iceberg_test_table (
  78.     //        | id bigint comment 'unique id',
  79.     //        | data string,
  80.     //        | primary key (id) not enforced
  81.     //        |) comment 'iceberg test table'
  82.     //        | partitioned by (id)
  83.     //        |""".stripMargin
  84.     //    tableEnv.executeSql(tableDDL)
  85.     //    tableEnv.executeSql("show tables").print()
  86.  
  87.     //  2. 修改表名
  88.     //    tableEnv.executeSql("alter table iceberg_test_table rename to iceberg_test_table2")
  89.     //    tableEnv.executeSql("show tables").print()
  90.     //
  91.  
  92.     // 3. 删除表
  93.     //    tableEnv.executeSql("drop table if exists iceberg_test_table")
  94.     //    tableEnv.executeSql("show tables").print()
  95.  
  96.     // 4. 查询表
  97.     //    tableEnv.executeSql("show tables").print()
  98.  
  99.     // 5. like 根据已有表创建新表
  100.     //    tableEnv.executeSql("create table iceberg_test_like like iceberg_test_table")
  101.     //    tableEnv.executeSql("show tables").print()
  102.  
  103.     // 6. 修改表属性
  104.     // flink1.11后支持
  105.     //    tableEnv.executeSql("alter table iceberg_test_like set ('write.format.default'='avro')")
  106.  
  107.     // 7. 写入
  108.     // tableAPI
  109.     //    val statementSet = tableEnv.createStatementSet()
  110.     //    statementSet.addInsertSql("insert into iceberg_test_table values (1, 'a')")
  111.     //    statementSet.execute()
  112.  
  113.     //    tableEnv.executeSql("insert into iceberg_test_table values (1, 'a'), (2, 'b')")
  114.     //+----------------------+--------------------------------+
  115.     //|                   id |                           data |
  116.     //+----------------------+--------------------------------+
  117.     //|                    1 |                              a |
  118.     //|                    2 |                              b |
  119.     //+----------------------+--------------------------------+
  120.  
  121.     //    tableEnv.executeSql("insert overwrite iceberg_test_table values (111, 'b')")
  122.     //+----------------------+--------------------------------+
  123.     //|                   id |                           data |
  124.     //+----------------------+--------------------------------+
  125.     //|                    1 |                            aaa |
  126.     //|                    2 |                              b |
  127.     //+----------------------+--------------------------------+
  128.  
  129.     //    tableEnv.executeSql("insert overwrite iceberg_test_table partition(data='b') select 888")
  130.     //+----------------------+--------------------------------+
  131.     //|                   id |                           data |
  132.     //+----------------------+--------------------------------+
  133.     //|                    2 |                              b |
  134.     //|                    1 |                            ccc |
  135.     //+----------------------+--------------------------------+
  136.  
  137.     // 8. 读取数据
  138.     //    tableEnv.executeSql("select * from iceberg_test_table").print()
  139.     //    val table = tableEnv.sqlQuery("select * from iceberg_test_table")
  140.     //    table.printSchema()
  141.     //    table.execute().print()
  142.  
  143.     // 9. 写入数据
  144.     //    val insert =
  145.     //      """
  146.     //        |insert into iceberg_test_like
  147.     //        |select
  148.     //        | id, data
  149.     //        |from iceberg_test_table
  150.     //        |""".stripMargin
  151.     //    tableEnv.executeSql(insert)
  152.     //    tableEnv.executeSql("select * from iceberg_test_like").print()
  153.  
  154.     // 10. 流读
  155.     //    val config = tableEnv.getConfig.getConfiguration
  156.     //    config.setBoolean(TableConfigOptions.TABLE_DYNAMIC_TABLE_OPTIONS_ENABLED, true)
  157.     //
  158.     //    // read all the records
  159.     //    val readAllDML =
  160.     //    """
  161.     //      |select * from iceberg_test_table
  162.     //      |/*+ options('streaming'='true', 'monitor-interval'='1s')*/
  163.     //      |""".stripMargin
  164.     //    tableEnv.executeSql(readAllDML).print()
  165.     //
  166.     //    // read incremental data
  167.     //    val readIncrementalDML =
  168.     //      """
  169.     //        |select * from iceberg_test_table
  170.     //        |/*+ options('streaming'='true', 'monitor-interval'='1s', 'start-snapshot-id'='8116368287341314212')*/
  171.     //        |""".stripMargin
  172.     //    tableEnv.executeSql(readIncrementalDML).print()
  173.  
  174.     // 11. cdc
  175.     tableEnv.useCatalog("default_catalog")
  176.     val cdcDDL =
  177.       """
  178.         |create table if not exists iceberg_cdc_source (
  179.         |  id int,
  180.         |  data string,
  181.         |  primary key (id) not enforced
  182.         |) with (
  183.         | 'connector' = 'mysql-cdc',
  184.         | 'hostname' = 'test-lakehouse',
  185.         | 'port' = '3306',
  186.         | 'username' = 'test',
  187.         | 'password' = '123456',
  188.         | 'database-name' = 'test_db',
  189.         | 'table-name' = 'test',
  190.         | 'server-time-zone' = 'Asia/Shanghai'
  191.         |)
  192.       """.stripMargin
  193.     tableEnv.executeSql(cdcDDL)
  194.  
  195.     // output
  196.     //    tableEnv.executeSql("select * from iceberg_cdc_source").print()
  197.  
  198.     //        val printSinkSql =
  199.     //          """
  200.     //            |create table if not exists print_sink (
  201.     //            |  id int,
  202.     //            |  data string,
  203.     //            |  primary key (id) not enforced
  204.     //            |) with (
  205.     //            | 'connector' = 'print'
  206.     //            |)
  207.     //          """.stripMargin
  208.     //        tableEnv.executeSql(printSinkSql)
  209.     //
  210.     //        tableEnv.executeSql("insert into print_sink select * from iceberg_cdc_source")
  211.  
  212.     //   catalog
  213.     val catalogDDL =
  214.       """
  215.         |create catalog hive_catalog with (
  216.         | 'type' = 'iceberg',
  217.         | 'catalog-type' = 'hive',
  218.         | 'uri' = 'thrift://test-lakehouse:9083',
  219.         | 'clients' = '5',
  220.         | 'property-version' = '1',
  221.         | 'warehouse' = 'hdfs://test-lakehouse:9000/user/hive/warehouse/'
  222.         |)
  223.         |""".stripMargin
  224.     tableEnv.executeSql(catalogDDL)
  225.  
  226.     val databaseDDL = "create database if not exists hive_catalog.iceberg_test_db"
  227.     tableEnv.executeSql(databaseDDL)
  228.  
  229.     //    tableEnv.executeSql("drop table if exists hive_catalog.iceberg_test_db.iceberg_cdc_test")
  230.  
  231.     val tableDDL =
  232.       """
  233.         |create table if not exists hive_catalog.iceberg_test_db.iceberg_cdc_test (
  234.         | id bigint comment 'unique id',
  235.         | data string,
  236.         | primary key (id) not enforced
  237.         |) comment 'iceberg test table'
  238.         | partitioned by (id)
  239.         | with(
  240.         |  'iceberg.format.version' = '2',
  241. //            |  'write.metadata.delete-after-commit.enabled' = 'true',
  242. //            |  'write.metadata.previous-versions-max' = '100'
  243.         | )
  244.         |""".stripMargin
  245.     tableEnv.executeSql(tableDDL)
  246.  
  247.     val cdcDML =
  248.       """
  249.         |insert into hive_catalog.iceberg_test_db.iceberg_cdc_test
  250.         |select * from default_catalog.default_database.iceberg_cdc_source
  251.         |""".stripMargin
  252.     tableEnv.executeSql(cdcDML)
  253.  
  254.     // stop cdc after
  255.     // tableEnv.executeSql("select * from iceberg_cdc_test").print()
  256.  
  257.     //    val config = tableEnv.getConfig.getConfiguration
  258.     //    config.setBoolean(TableConfigOptions.TABLE_DYNAMIC_TABLE_OPTIONS_ENABLED, true)
  259.     //    tableEnv.executeSql(
  260.     //      """
  261.     //        |select * from iceberg_cdc_test
  262.     //        |/*+ options('streaming'='true', 'monitor-interval'='1s')*/
  263.     //      """.stripMargin).print()
  264.   }
  265. }

检查数据

建表后

2021-09-24 14:40:31,948 INFO - Successfully committed to table hive_catalog.iceberg_test_db.iceberg_test_table in 2008 ms

+--------------------+

| table name |

+--------------------+

| iceberg_test_table |

+--------------------+

hdfs数据

未写入数据前，只有metastore

metastore信息

  1. {
  2.   "format-version" : 1,
  3.   "table-uuid" : "efbc787a-6eed-46ef-a2a8-c04b8cbcf1c2",
  4.   "location" : "hdfs://test-lakehouse:9000/user/hive/warehouse/iceberg_test_db.db/iceberg_test_table",
  5.   "last-updated-ms" : 1632715958040,
  6.   "last-column-id" : 2,
  7.   "schema" : {
  8.     "type" : "struct",
  9.     "schema-id" : 0,
  10.     "identifier-field-ids" : [ 1 ],
  11.     "fields" : [ {
  12.       "id" : 1,
  13.       "name" : "id",
  14.       "required" : true,
  15.       "type" : "long"
  16.     }, {
  17.       "id" : 2,
  18.       "name" : "data",
  19.       "required" : false,
  20.       "type" : "string"
  21.     } ]
  22.   },
  23.   "current-schema-id" : 0,
  24.   "schemas" : [ {
  25.     "type" : "struct",
  26.     "schema-id" : 0,
  27.     "identifier-field-ids" : [ 1 ],
  28.     "fields" : [ {
  29.       "id" : 1,
  30.       "name" : "id",
  31.       "required" : true,
  32.       "type" : "long"
  33.     }, {
  34.       "id" : 2,
  35.       "name" : "data",
  36.       "required" : false,
  37.       "type" : "string"
  38.     } ]
  39.   } ],
  40.   "partition-spec" : [ {
  41.     "name" : "id",
  42.     "transform" : "identity",
  43.     "source-id" : 1,
  44.     "field-id" : 1000
  45.   } ],
  46.   "default-spec-id" : 0,
  47.   "partition-specs" : [ {
  48.     "spec-id" : 0,
  49.     "fields" : [ {
  50.       "name" : "id",
  51.       "transform" : "identity",
  52.       "source-id" : 1,
  53.       "field-id" : 1000
  54.     } ]
  55.   } ],
  56.   "last-partition-id" : 1000,
  57.   "default-sort-order-id" : 0,
  58.   "sort-orders" : [ {
  59.     "order-id" : 0,
  60.     "fields" : [ ]
  61.   } ],
  62.   "properties" : { },
  63.   "current-snapshot-id" : -1,
  64.   "snapshots" : [ ],
  65.   "snapshot-log" : [ ],
  66.   "metadata-log" : [ ]
  67. }

修改数据后

  1.     ......
  2.  
  3.   "properties" : { },
  4.   "current-snapshot-id" : 3357358225130025285,
  5.   "snapshots" : [ {
  6.     "snapshot-id" : 750183960105471040,
  7.     "timestamp-ms" : 1632715970291,
  8.     "summary" : {
  9.       "operation" : "append",
  10.       "flink.job-id" : "c79435a3ae5097eba8842a1816409be5",
  11.       "flink.max-committed-checkpoint-id" : "9223372036854775807",
  12.       "added-data-files" : "2",
  13.       "added-records" : "2",
  14.       "added-files-size" : "1354",
  15.       "changed-partition-count" : "2",
  16.       "total-records" : "2",
  17.       "total-files-size" : "1354",
  18.       "total-data-files" : "2",
  19.       "total-delete-files" : "0",
  20.       "total-position-deletes" : "0",
  21.       "total-equality-deletes" : "0"
  22.     },
  23.     "manifest-list" : "hdfs://test-lakehouse:9000/user/hive/warehouse/iceberg_test_db.db/iceberg_test_table/metadata/snap-750183960105471040-1-bc72ec07-52b5-4352-9c6b-1db44c8f85e9.avro",
  24.     "schema-id" : 0
  25.   }, {
  26.     "snapshot-id" :
  27.  
  28.        ......
  29.  
  30.     },
  31.     "manifest-list" : "hdfs://test-lakehouse:9000/user/hive/warehouse/iceberg_test_db.db/iceberg_test_table/metadata/snap-3357358225130025285-1-9f5c0553-7a4b-42c7-8199-1f7cff77f3ac.avro",
  32.     "schema-id" : 0
  33.   } ],
  34.   "snapshot-log" : [ {
  35.     "timestamp-ms" : 1632715970291,
  36.     "snapshot-id" : 750183960105471040
  37.   }, {
  38.     
  39.     ......
  40.  
  41.   } ],
  42.  
  43.   "metadata-log" : [ {
  44.     "timestamp-ms" : 1632715958040,
  45.     "metadata-file" : "hdfs://test-lakehouse:9000/user/hive/warehouse/iceberg_test_db.db/iceberg_test_table/metadata/00000-7f31a7d0-6bd9-45a4-82f6-210ea2aa5f10.metadata.json"
  46.   }, {
  47.  
  48.     ......
  49.  
  50.   } ]
  51. }
hdfs dfs -text /.../iceberg_test_db.db/iceberg_test_table/metadata/snap-3357358225130025285-1-9f5c0553-7a4b-42c7-8199-1f7cff77f3ac.avro

{"manifest_path":"hdfs://test-lakehouse:9000/user/hive/warehouse/iceberg_test_db.db/iceberg_test_table/metadata/9f5c0553-7a4b-42c7-8199-1f7cff77f3ac-m1.avro","manifest_length":6030,"partition_spec_id":0,"added_snapshot_id":{"long":3357358225130025285},"added_data_files_count":{"int":1},"existing_data_files_count":{"int":0},"deleted_data_files_count":{"int":0},"partitions":{"array":[{"contains_null":false,"contains_nan":{"boolean":false},"lower_bound":{"bytes":"\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},"upper_bound":{"bytes":"\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000"}}]},"added_rows_count":{"long":1},"existing_rows_count":{"long":0},"deleted_rows_count":{"long":0}} 

hdfs dfs -text /.../iceberg_test_db.db/iceberg_test_table/metadata/9f5c0553-7a4b-42c7-8199-1f7cff77f3ac-m0.avro


{"status":2,"snapshot_id":{"long":3357358225130025285},"data_file":{"file_path":"hdfs://test-lakehouse:9000/user/hive/warehouse/iceberg_test_db.db/iceberg_test_table/data/id=1/00007-0-3ccc043d-9d03-4b5c-8268-55c09827927b-00001.parquet","file_format":"PARQUET","partition":{"id":{"long":1}},"record_count":1,"file_size_in_bytes":691,"block_size_in_bytes":67108864,"column_sizes":{"array":[{"key":1,"value":46},{"key":2,"value":54}]},"value_counts":{"array":[{"key":1,"value":1},{"key":2,"value":1}]},"null_value_counts":{"array":[{"key":1,"value":0},{"key":2,"value":0}]},"nan_value_counts":{"array":[]},"lower_bounds":{"array":[{"key":1,"value":"\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},{"key":2,"value":"aaa"}]},"upper_bounds":{"array":[{"key":1,"value":"\u0001\u0000\u0000\u0000\u0000\u0000\u0000\u0000"},{"key":2,"value":"aaa"}]},"key_metadata":null,"split_offsets":{"array":[4]},"sort_order_id":{"int":0}}}

hiveCatalog hive元数据信息会记录 iceberg 表名及 iceberg 元数据位置 metadata_location

 

修改iceberg表名，只会修改hive元数据信息，iceberg元数据metadata不变，也就是hdfs上表目录名和json信息不会变。

删除表，会删除hive元数据信息和iceberg元数据位置信息，会删除hdfs上目录下的metadata下的元数据，但是不会删目录。

Overwrite，会写入新的parquet文件，不会第一时间清理旧文件。

Hive信息

修改后

overwrite后，之前的snapshots 无法再读取

Found overwrite operation, cannot support incremental data in snapshots (8116368287341314212, 3591005179391590033]

另外，Flink cdc mysql8.x问题

Public Key Retrieval is not allowed

MySQL8.0连接验证机制发生了变化，默认使用caching_sha2_password作为身份验证插件，修改为使用mysql_native_password加密规则来校验身份。

alter user 'test'@'%' identified with mysql_native_password by '123456';

Iceberg v1 CDC 不支持deletions，只能初始化CDC时导入，后续有deletions操作数据进入会报错。v2功能还在开发中，暂未对外开放，所以CDC功能目前并不能使用。

 pom文件

  1.     <properties>
  2.         <!-- project compiler -->
  3.         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  4.         <maven.compiler.source>1.8</maven.compiler.source>
  5.         <maven.compiler.target>1.8</maven.compiler.target>
  6.         <!-- maven compiler-->
  7.         <scala.maven.plugin.version>3.2.2</scala.maven.plugin.version>
  8.         <maven.compiler.plugin.version>3.8.1</maven.compiler.plugin.version>
  9.         <maven.assembly.plugin.version>3.1.1</maven.assembly.plugin.version>
  10.         <!-- sdk -->
  11.         <java.version>1.8</java.version>
  12.         <scala.version>2.12.13</scala.version>
  13.         <scala.binary.version>2.12</scala.binary.version>
  14.         <!-- engine-->
  15.         <hadoop.version>2.9.2</hadoop.version>
  16.         <flink.version>1.12.5</flink.version>
  17.         <iceberg.version>0.12.0</iceberg.version>
  18.         <hive.version>2.3.9</hive.version>
  19.  
  20. <!--         <scope.type>provided</scope.type>-->
  21.         <scope.type>compile</scope.type>
  22.     </properties>
  23.  
  24.     <dependencies>
  25.         <!-- scala -->
  26.         <dependency>
  27.             <groupId>org.scala-lang</groupId>
  28.             <artifactId>scala-library</artifactId>
  29.             <version>${scala.version}</version>
  30.             <scope>${scope.type}</scope>
  31.         </dependency>
  32.  
  33.         <!-- flink Dependency -->
  34.         <dependency>
  35.             <groupId>org.apache.flink</groupId>
  36.             <artifactId>flink-runtime-web_${scala.binary.version}</artifactId>
  37.             <version>${flink.version}</version>
  38.             <scope>${scope.type}</scope>
  39.         </dependency>
  40.         <dependency>
  41.             <groupId>org.apache.flink</groupId>
  42.             <artifactId>flink-core</artifactId>
  43.             <version>${flink.version}</version>
  44.             <scope>${scope.type}</scope>
  45.         </dependency>
  46.         <dependency>
  47.             <groupId>org.apache.flink</groupId>
  48.             <artifactId>flink-scala_${scala.binary.version}</artifactId>
  49.             <version>${flink.version}</version>
  50.             <scope>${scope.type}</scope>
  51.         </dependency>
  52.         <dependency>
  53.             <groupId>org.apache.flink</groupId>
  54.             <artifactId>flink-table-common</artifactId>
  55.             <version>${flink.version}</version>
  56.             <scope>${scope.type}</scope>
  57.         </dependency>
  58.         <dependency>
  59.             <groupId>org.apache.flink</groupId>
  60.             <artifactId>flink-table-api-scala-bridge_${scala.binary.version}</artifactId>
  61.             <version>${flink.version}</version>
  62.             <scope>${scope.type}</scope>
  63.         </dependency>
  64.         <dependency>
  65.             <groupId>org.apache.flink</groupId>
  66.             <artifactId>flink-streaming-scala_${scala.binary.version}</artifactId>
  67.             <version>${flink.version}</version>
  68.             <scope>${scope.type}</scope>
  69.         </dependency>
  70.         <dependency>
  71.             <groupId>org.apache.flink</groupId>
  72.             <artifactId>flink-table-planner-blink_${scala.binary.version}</artifactId>
  73.             <version>${flink.version}</version>
  74.             <scope>${scope.type}</scope>
  75.         </dependency>
  76.         <dependency>
  77.             <groupId>org.apache.flink</groupId>
  78.             <artifactId>flink-clients_${scala.binary.version}</artifactId>
  79.             <version>${flink.version}</version>
  80.             <scope>${scope.type}</scope>
  81.         </dependency>
  82.         <dependency>
  83.             <groupId>org.apache.flink</groupId>
  84.             <artifactId>flink-csv</artifactId>
  85.             <version>${flink.version}</version>
  86.             <scope>${scope.type}</scope>
  87.         </dependency>
  88.         <dependency>
  89.             <groupId>org.apache.flink</groupId>
  90.             <artifactId>flink-json</artifactId>
  91.             <version>${flink.version}</version>
  92.             <scope>${scope.type}</scope>
  93.         </dependency>
  94.         <dependency>
  95.             <groupId>org.apache.flink</groupId>
  96.             <artifactId>flink-orc_${scala.binary.version}</artifactId>
  97.             <version>${flink.version}</version>
  98.             <scope>${scope.type}</scope>
  99.         </dependency>
  100.         <dependency>
  101.             <groupId>org.apache.flink</groupId>
  102.             <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
  103.             <version>${flink.version}</version>
  104.             <scope>${scope.type}</scope>
  105.         </dependency>
  106.         <dependency>
  107.             <groupId>org.apache.flink</groupId>
  108.             <artifactId>flink-sql-connector-kafka_${scala.binary.version}</artifactId>
  109.             <version>${flink.version}</version>
  110.             <scope>${scope.type}</scope>
  111.         </dependency>
  112.         <dependency>
  113.             <groupId>org.apache.flink</groupId>
  114.             <artifactId>flink-statebackend-rocksdb_2.11</artifactId>
  115.             <version>${flink.version}</version>
  116.             <scope>${scope.type}</scope>
  117.         </dependency>
  118.         <dependency>
  119.             <groupId>org.apache.flink</groupId>
  120.             <artifactId>flink-connector-hive_${scala.binary.version}</artifactId>
  121.             <version>${flink.version}</version>
  122.             <scope>${scope.type}</scope>
  123.         </dependency>
  124.         <dependency>
  125.             <groupId>com.alibaba.ververica</groupId>
  126.             <artifactId>flink-sql-connector-mysql-cdc</artifactId>
  127.             <version>1.2.0</version>
  128.             <scope>${scope.type}</scope>
  129.         </dependency>
  130.  
  131.         <!-- iceberg Dependency -->
  132.         <dependency>
  133.             <groupId>org.apache.iceberg</groupId>
  134.             <artifactId>iceberg-flink-runtime</artifactId>
  135.             <version>${iceberg.version}</version>
  136.             <scope>${scope.type}</scope>
  137.         </dependency>
  138.  
  139.         <!-- hadoop Dependency-->
  140.         <dependency>
  141.             <groupId>org.apache.hadoop</groupId>
  142.             <artifactId>hadoop-common</artifactId>
  143.             <version>${hadoop.version}</version>
  144.             <scope>${scope.type}</scope>
  145.         </dependency>
  146.         <dependency>
  147.             <groupId>org.apache.hadoop</groupId>
  148.             <artifactId>hadoop-hdfs</artifactId>
  149.             <version>${hadoop.version}</version>
  150.             <scope>${scope.type}</scope>
  151.         </dependency>
  152.         <dependency>
  153.             <groupId>org.apache.hadoop</groupId>
  154.             <artifactId>hadoop-client</artifactId>
  155.             <version>${hadoop.version}</version>
  156.             <scope>${scope.type}</scope>
  157.         </dependency>
  158.  
  159.         <!-- hive Dependency-->
  160.         <dependency>
  161.             <groupId>org.apache.hive</groupId>
  162.             <artifactId>hive-exec</artifactId>
  163.             <version>${hive.version}</version>
  164.             <scope>${scope.type}</scope>
  165.             <exclusions>
  166.                 <exclusion>
  167.                     <groupId>org.apache.logging.log4j</groupId>
  168.                     <artifactId>log4j-slf4j-impl</artifactId>
  169.                 </exclusion>
  170.                 <exclusion>
  171.                     <groupId>org.apache.hive</groupId>
  172.                     <artifactId>hive-llap-tez</artifactId>
  173.                 </exclusion>
  174.             </exclusions>
  175.         </dependency>
  176.         <dependency>
  177.             <groupId>org.antlr</groupId>
  178.             <artifactId>antlr-runtime</artifactId>
  179.             <version>3.5.2</version>
  180.         </dependency>
  181.  
  182.     </dependencies>
  183.  
  184.     <build>
  185.         <plugins>
  186.             <plugin>
  187.                 <groupId>net.alchim31.maven</groupId>
  188.                 <artifactId>scala-maven-plugin</artifactId>
  189.                 <version>${scala.maven.plugin.version}</version>
  190.                 <executions>
  191.                     <execution>
  192.                         <goals>
  193.                             <!--声明绑定到maven的compile阶段-->
  194.                             <goal>compile</goal>
  195.                         </goals>
  196.                     </execution>
  197.                 </executions>
  198.             </plugin>
  199.             <plugin>
  200.                 <groupId>org.apache.maven.plugins</groupId>
  201.                 <artifactId>maven-assembly-plugin</artifactId>
  202.                 <version>${maven.assembly.plugin.version}</version>
  203.                 <configuration>
  204.                     <descriptorRefs>
  205.                         <descriptorRef>jar-with-dependencies</descriptorRef>
  206.                     </descriptorRefs>
  207.                 </configuration>
  208.                 <executions>
  209.                     <execution>
  210.                         <id>make-assembly</id>
  211.                         <phase>package</phase>
  212.                         <goals>
  213.                             <goal>single</goal>
  214.                         </goals>
  215.                     </execution>
  216.                 </executions>
  217.             </plugin>
  218.         </plugins>
  219.     </build>
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/Guff_9hys/article/detail/1001411
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号