spark scala 编写 词频统计案例_linux spark 词频统计

在  home/hzb 下 新建 test.txt 文件

vim test.txt

 

1.从 Linux 本地文件系统加载数据创建 RDD

scala> val lines=sc.textFile("file:///home/hzb/test.txt")
lines: org.apache.spark.rdd.RDD[String] = file:///home/hzb/test.txt MapPartitionsRDD[1] at textFile at <console>:24
 
 

2.转换算子 通过 flatMap 转换算子 把文件内容拆分成一个个单词

scala> val words=lines.flatMap(line=>line.split(" "))
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at flatMap at <console>:25

3.通过 Map 算子 返回一个新的数据集

scala> val wordAndOne=words.map(word=>(word,1))
wordAndOne: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[3] at map at <console>:25

4.通过groupByKey 算子操作文件内容中所有单词进行分组

scala> val wordCount=wordAndOne.reduceByKey((a,b)=>a+b)
wordCount: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:25

5.打印 wordCoount

scala> wordCount.foreach(println)

6.词凭统计结果

(scala,1)
(spark,3)
(itcast,3)
(hadoop,2)
(heima,1)