基于Hadoop平台的电信客服数据的处理与分析③项目开发：搭建基于Hadoop的全分布式集群---任务8：测试Hadoop集群的可用性

任务描述

测试Hadoop集群的可用性

任务指导

1. 在Web UI查看HDFS和YARN状态

2. 测试HDFS和YARN的可用性

任务实现

1. 在Web UI查看HDFS和YARN状态

在【master1】打开Web浏览器访问Hadoop其中HDFS NameNode对应的Web UI地址如下：

http://master1:50070

如下图HDFS自检成功，已退出安全模式代表着可以对HDFS进行读写；集群中有2个【Live Nodes】可用来存储数据

YARN ResourceManager对应的Web UI地址如下：

http://master1:8088

如下图YARN集群启动成功，集群中有两个节点【slave1、slave2】可供用来执行计算任务：

2. 测试HDFS和YARN的可用性

在HDFS创建目录

[root@master1 ~]# hdfs dfs -mkdir /input

将本地的文本文件上传到HDFS

[root@master1 ~]# cd $HADOOP_HOME
[root@master1 hadoop-2.10.1]# hdfs dfs -put README.txt /input

执行Hadoop自带的测试程序WordCount，此程序将统计输入目录下所有文件中的单词数量：

[root@master1 hadoop-2.10.1]# cd $HADOOP_HOME
[root@master1 hadoop-2.10.1]# cd share/hadoop/mapreduce
[root@master1 mapreduce]# yarn jar hadoop-mapreduce-examples-2.10.1.jar wordcount /input /output

执行结果如下：

[root@master1 mapreduce]# yarn jar hadoop-mapreduce-examples-2.10.1.jar wordcount /input /output
23/10/18 09:50:56 INFO client.RMProxy: Connecting to ResourceManager at master1/192.168.3.129:8032
23/10/18 09:50:56 INFO input.FileInputFormat: Total input files to process : 1
23/10/18 09:50:56 INFO mapreduce.JobSubmitter: number of splits:1
23/10/18 09:50:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1697619784960_0001
23/10/18 09:50:57 INFO conf.Configuration: resource-types.xml not found
23/10/18 09:50:57 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
23/10/18 09:50:57 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
23/10/18 09:50:57 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
23/10/18 09:50:57 INFO impl.YarnClientImpl: Submitted application application_1697619784960_0001
23/10/18 09:50:57 INFO mapreduce.Job: The url to track the job: http://master1:8088/proxy/application_1697619784960_0001/
23/10/18 09:50:57 INFO mapreduce.Job: Running job: job_1697619784960_0001
23/10/18 09:51:04 INFO mapreduce.Job: Job job_1697619784960_0001 running in uber mode : true
23/10/18 09:51:04 INFO mapreduce.Job:  map 100% reduce 0%
23/10/18 09:51:06 INFO mapreduce.Job:  map 100% reduce 100%
23/10/18 09:51:06 INFO mapreduce.Job: Job job_1697619784960_0001 completed successfully
23/10/18 09:51:06 INFO mapreduce.Job: Counters: 52
       File System Counters
               FILE: Number of bytes read=3704
               FILE: Number of bytes written=5572
               FILE: Number of read operations=0
               FILE: Number of large read operations=0
               FILE: Number of write operations=0
               HDFS: Number of bytes read=2992
               HDFS: Number of bytes written=439695
               HDFS: Number of read operations=41
               HDFS: Number of large read operations=0
               HDFS: Number of write operations=16
       Job Counters 
               Launched map tasks=1
               Launched reduce tasks=1
               Other local map tasks=1
               Total time spent by all maps in occupied slots (ms)=0
               Total time spent by all reduces in occupied slots (ms)=0
               TOTAL_LAUNCHED_UBERTASKS=2
               NUM_UBER_SUBMAPS=1
               NUM_UBER_SUBREDUCES=1
               Total time spent by all map tasks (ms)=428
               Total time spent by all reduce tasks (ms)=1202
               Total vcore-milliseconds taken by all map tasks=0
               Total vcore-milliseconds taken by all reduce tasks=0
               Total megabyte-milliseconds taken by all map tasks=0
               Total megabyte-milliseconds taken by all reduce tasks=0
       Map-Reduce Framework
               Map input records=31
               Map output records=179
               Map output bytes=2055
               Map output materialized bytes=1836
               Input split bytes=101
               Combine input records=179
               Combine output records=131
               Reduce input groups=131
               Reduce shuffle bytes=1836
               Reduce input records=131
               Reduce output records=131
               Spilled Records=262
               Shuffled Maps =1
               Failed Shuffles=0
               Merged Map outputs=1
               GC time elapsed (ms)=10
               CPU time spent (ms)=1560
               Physical memory (bytes) snapshot=812826624
               Virtual memory (bytes) snapshot=6216478720
               Total committed heap usage (bytes)=562036736
       Shuffle Errors
               BAD_ID=0
               CONNECTION=0
               IO_ERROR=0
               WRONG_LENGTH=0
               WRONG_MAP=0
               WRONG_REDUCE=0
       File Input Format Counters 
               Bytes Read=1366
       File Output Format Counters 
               Bytes Written=4540

查看结果目录

[root@master1 mapreduce]# hdfs dfs -ls /output

回显如下

[root@master1 mapreduce]# hdfs dfs -ls /output
Found 2 items
-rw-r--r--   2 root supergroup          0 2023-10-18 09:51 /output/_SUCCESS
-rw-r--r--   2 root supergroup       1306 2023-10-18 09:51 /output/part-r-00000

其中【_SUCCESS】文件为标识文件，代表任务执行成功，以【part-r-】为前缀的文件为Reduce的输出文件，【part-r-XXXXX】文件个数与MapReduce中的Reduce个数对应，执行如下命令查看结果

[root@master1 mapreduce]# hdfs dfs -cat /output/part-r-*

结果如下：

[root@master1 mapreduce]# hdfs dfs -cat /output/part-r-*
(BIS),  1
(ECCN)  1
(TSU)   1
(see    1
5D002.C.1,      1
740.13) 1
<http://www.wassenaar.org/>     1
Administration  1
Apache  1
BEFORE  1
BIS     1
Bureau  1
Commerce,       1
Commodity       1
Control 1
Core    1
Department      1
ENC     1
Exception       1
Export  2
For     1
Foundation      1
Government      1
Hadoop  1
Hadoop, 1
Industry        1
Jetty   1
License 1
Number  1
Regulations,    1
SSL     1
.
.
.
略

由此证明HDFS和YARN可以正常运行。