devbox
AI编程探险家
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

使用idea 创建flink项目并测试WindowWordCount_windows环境 flink wordcount

作者：AI编程探险家 | 2024-01-29 17:59:08

赞

踩

windows环境 flink wordcount

使用idea 创建flink项目并测试WindowWordCount

环境

flink 1.17.1

jdk：1.8

scala 2.13.11

创建项目

使用idea来直接创建项目 如下：

设置Archetype 内容如上 

org.apache.flink

flink-quickstart-java

version 为flink的版本

配置pom

基本设置

  1. <dependency>
  2.     <groupId>org.apache.flink</groupId>
  3.     <artifactId>flink-streaming-java</artifactId>
  4.     <version>${flink.version}</version>
  5.     <scope>provided</scope>
  6. </dependency>
  7. <dependency>
  8.     <groupId>org.apache.flink</groupId>
  9.     <artifactId>flink-clients</artifactId>
  10.     <version>${flink.version}</version>
  11.     <scope>provided</scope>
  12. </dependency>

flink核心设置

  1. <!-- flink核心API -->
  2. <dependency>
  3.     <groupId>org.apache.flink</groupId>
  4.     <artifactId>flink-java</artifactId>
  5.     <version>${flink.version}</version>
  6. </dependency>
  7. <dependency>
  8.     <groupId>org.apache.flink</groupId>
  9.     <artifactId>flink-table-api-scala-bridge_${scala.binary.version}</artifactId>
  10.     <version>${flink.version}</version>
  11. </dependency>
  12.  
  13. <dependency>
  14.     <groupId>org.apache.flink</groupId>
  15.     <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
  16.     <version>${flink.version}</version>
  17.     <scope>provided</scope>
  18. </dependency>
  19. <dependency>
  20.     <groupId>org.apache.flink</groupId>
  21.     <artifactId>flink-json</artifactId>
  22.     <version>${flink.version}</version>
  23. </dependency>
  24. <dependency>
  25.     <groupId>org.apache.flink</groupId>
  26.     <artifactId>flink-scala_${scala.binary.version}</artifactId>
  27.     <version>${flink.version}</version>
  28. </dependency>
  29. <dependency>
  30.     <groupId>org.apache.flink</groupId>
  31.     <artifactId>flink-table-api-scala_${scala.binary.version}</artifactId>
  32.     <version>${flink.version}</version>
  33. </dependency>

测试 使用flnk开发一个 WindowWordCount 代码如下

  1. package org.example;
  2. import org.apache.flink.api.common.functions.FlatMapFunction;
  3. import org.apache.flink.api.java.tuple.Tuple2;
  4. import org.apache.flink.streaming.api.datastream.DataStream;
  5. import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
  6. import org.apache.flink.streaming.api.windowing.assigners.TumblingEventTimeWindows;
  7. import org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows;
  8. import org.apache.flink.streaming.api.windowing.time.Time;
  9. import org.apache.flink.util.Collector;
  10. /**
  11. nc -lk 9999
  12. ceshi nhiaho  zhong
  13. ceshi nhiaho  zhong
  14. cehi  zhang san
  15. cehi  zhang san
  16. **/
  17. public class WindowWordCount {
  18.     public static void main(String[] args) throws  Exception{
  19.         StreamExecutionEnvironment env =StreamExecutionEnvironment.getExecutionEnvironment();
  20.         DataStream<Tuple2<String,Integer>> dataStream=env
  21.                 .socketTextStream("172.31.7.10",9999)
  22.                 .flatMap(new Splitter())
  23.                 .keyBy(v->v.f0)
  24.                 .window(TumblingProcessingTimeWindows.of(Time.seconds(5)))
  25.                 .sum(1);
  26.         dataStream.print();
  27.         env.execute("Window WordNount");
  28.  
  29.     }
  30.     public static class Splitter implements FlatMapFunction<String,Tuple2<String,Integer>>
  31.     {
  32.  
  33.         @Override
  34.         public void flatMap(String s, Collector<Tuple2<String, Integer>> outcol) throws Exception {
  35.             for(String word:s.split(" "))
  36.             {
  37.                 outcol.collect(new Tuple2<String,Integer>(word,1));
  38.             }
  39.  
  40.         }
  41.     }
  42. }

在socket 机器上输入

程序输出如下：

MYSQL系列书籍

高可用mysql: https://url41.ctfile.com/f/49289241-959127723-de4738?p=2651 (访问密码: 2651)

MySQL王者晋级之路.pdf: https://url41.ctfile.com/f/49289241-959127432-204284?p=2651 (访问密码: 2651)

MySQL技术内幕InnoDB存储引擎第2版.pdf: https://url41.ctfile.com/f/49289241-959126379-4590a8?p=2651 (访问密码: 2651)

MySQL技术内幕 第4版.pdf: https://url41.ctfile.com/f/49289241-959125506-a5bcec?p=2651 (访问密码: 2651)

MySQL管理之道，性能调优，高可用与监控（第二版）.pdf: https://url41.ctfile.com/f/49289241-959124249-d59f54?p=2651 (访问密码: 2651)

深入浅出MySQL数据库开发、优化与管理维护第2版.pdf: https://url41.ctfile.com/f/49289241-961464090-68bf10?p=2651 (访问密码: 2651)

高性能MySQL.第3版.Baron Schwartz.pdf: https://url41.ctfile.com/f/49289241-961462308-52cc5d?p=2651 (访问密码: 2651)

MYSQL内核：INNODB存储引擎 卷1.pdf: https://url41.ctfile.com/f/49289241-961461357-ee63e3?p=2651 (访问密码: 2651)

MySQL技术内幕InnoDB存储引擎第2版.pdf: https://url41.ctfile.com/f/49289241-959126379-4590a8?p=2651 (访问密码: 2651)

MySQLDBA修炼之道.pdf: https://url41.ctfile.com/f/49289241-961459500-9b201d?p=2651 (访问密码: 2651)

MySQL5.7从入门到精通.pdf: https://url41.ctfile.com/f/49289241-961459329-48cbcf?p=2651 (访问密码: 2651)

高可用mysql.pdf: https://url41.ctfile.com/f/49289241-959127723-de4738?p=2651 (访问密码: 2651)

HIVE电子书

Practical Hive.pdf: https://url41.ctfile.com/f/49289241-959129883-d35ee9?p=2651 (访问密码: 2651)

Hive-Succinctly.pdf: https://url41.ctfile.com/f/49289241-959129709-30f30b?p=2651 (访问密码: 2651)

Apache Hive Essentials.pdf: https://url41.ctfile.com/f/49289241-959129691-b1a4aa?p=2651 (访问密码: 2651)

Apache Hive Cookbook.pdf: https://url41.ctfile.com/f/49289241-959129619-3a8ea6?p=2651 (访问密码: 2651)

hadoop电子书

Practical Hadoop Migration.pdf: https://url41.ctfile.com/f/49289241-959131470-dd3e24?p=2651 (访问密码: 2651)

Hadoop实战-陆嘉恒(高清完整版).pdf: https://url41.ctfile.com/f/49289241-959131365-433ec9?p=2651 (访问密码: 2651)

Hadoop & Spark大数据开发实战.pdf: https://url41.ctfile.com/f/49289241-959131032-ba40ea?p=2651 (访问密码: 2651)

Expert Hadoop Administration.pdf: https://url41.ctfile.com/f/49289241-959130468-ba70cd?p=2651 (访问密码: 2651)

Big Data Forensics - Learning Hadoop Investigations.pdf: https://url41.ctfile.com/f/49289241-959130435-9ab981?p=2651 (访问密码: 2651)

python电子书

python学习手册.pdf: https://url41.ctfile.com/f/49289241-959129403-5b45b1?p=2651 (访问密码: 2651)

Python基础教程-第3版.pdf: https://url41.ctfile.com/f/49289241-959128707-de6ef2?p=2651 (访问密码: 2651)

Python编程：从入门到实践.pdf: https://url41.ctfile.com/f/49289241-959128548-ce965d?p=2651 (访问密码: 2651)

Python Projects for Beginners.pdf: https://url41.ctfile.com/f/49289241-959128461-b53321?p=2651 (访问密码: 2651)

kafka电子书

Learning Apache Kafka, 2nd Edition.pdf: https://url41.ctfile.com/f/49289241-959134953-a14305?p=2651 (访问密码: 2651)

Kafka权威指南.pdf: https://url41.ctfile.com/f/49289241-959134932-295734?p=2651 (访问密码: 2651)

Kafka in Action.pdf: https://url41.ctfile.com/f/49289241-959134116-12111a?p=2651 (访问密码: 2651)

Apache Kafka实战.pdf: https://url41.ctfile.com/f/49289241-959133999-76ef77?p=2651 (访问密码: 2651)

Apache Kafka Cookbook.pdf: https://url41.ctfile.com/f/49289241-959132547-055c36?p=2651 (访问密码: 2651)

spark电子书

Spark最佳实践.pdf: https://url41.ctfile.com/f/49289241-959415393-5829fe?p=2651 (访问密码: 2651)

数据算法--Hadoop-Spark大数据处理技巧.pdf: https://url41.ctfile.com/f/49289241-959415927-5bdddc?p=2651 (访问密码: 2651)

Spark大数据分析实战.pdf: https://url41.ctfile.com/f/49289241-959416377-924161?p=2651 (访问密码: 2651)

Spark 2.0 for Beginners.pdf: https://url41.ctfile.com/f/49289241-959416710-7ea156?p=2651 (访问密码: 2651)

Pro Spark Streaming.pdf: https://url41.ctfile.com/f/49289241-959416866-6116d7?p=2651 (访问密码: 2651)

Spark in Action.pdf: https://url41.ctfile.com/f/49289241-959416986-e759e9?p=2651 (访问密码: 2651)

Learn PySpark.pdf: https://url41.ctfile.com/f/49289241-959417049-ac04a0?p=2651 (访问密码: 2651)

Fast Data Processing with Spark.pdf: https://url41.ctfile.com/f/49289241-959417157-8ec3b0?p=2651 (访问密码: 2651)

Fast Data Processing with Spark, 2nd Edition.pdf: https://url41.ctfile.com/f/49289241-959417211-856d08?p=2651 (访问密码: 2651)

OReilly.Learning.Spark.2015.1.pdf: https://url41.ctfile.com/f/49289241-959417292-90c1bc?p=2651 (访问密码: 2651)

High Performance Spark.pdf: https://url41.ctfile.com/f/49289241-959417439-7e7893?p=2651 (访问密码: 2651)

Machine Learning with PySpark.pdf: https://url41.ctfile.com/f/49289241-959417580-5941b3?p=2651 (访问密码: 2651)

Spark for Python Developers.pdf: https://url41.ctfile.com/f/49289241-959417721-d59fbe?p=2651 (访问密码: 2651)

Spark Cookbook.pdf: https://url41.ctfile.com/f/49289241-959417811-19c75d?p=2651 (访问密码: 2651)

Big Data Analytics with Spark.pdf: https://url41.ctfile.com/f/49289241-959417907-41dbce?p=2651 (访问密码: 2651)

PySpark SQL Recipes.pdf: https://url41.ctfile.com/f/49289241-959417970-c23242?p=2651 (访问密码: 2651)

Advanced Analytics with Spark Patterns for Learning from Data at Scale .pdf: https://url41.ctfile.com/f/49289241-959417997-a5e3f5?p=2651 (访问密码: 2651)

OReilly.Advanced.Analytics.with.Spark.Patterns.for.Learning.from.Data.at.Scale.pdf: https://url41.ctfile.com/f/49289241-959418024-2ff34c?p=2651 (访问密码: 2651)

Big Data Analytics Beyond Hadoop_ Real-Time Applications with Storm, Spark, and More Hadoop Alternatives.pdf: https://url41.ctfile.com/f/49289241-959418042-581fb9?p=2651 (访问密码: 2651)

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/article/detail/44014
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号