当前位置:   article > 正文

基于Hadoop平台的电信客服数据的处理与分析③项目开发:搭建基于Hadoop的全分布式集群---任务8:测试Hadoop集群的可用性

基于Hadoop平台的电信客服数据的处理与分析③项目开发:搭建基于Hadoop的全分布式集群---任务8:测试Hadoop集群的可用性

任务描述

测试Hadoop集群的可用性

任务指导

1. 在Web UI查看HDFS和YARN状态

2. 测试HDFS和YARN的可用性

任务实现

1. 在Web UI查看HDFS和YARN状态

在【master1】打开Web浏览器访问Hadoop其中HDFS NameNode对应的Web UI地址如下:

http://master1:50070

如下图HDFS自检成功,已退出安全模式代表着可以对HDFS进行读写;集群中有2个【Live Nodes】可用来存储数据

YARN ResourceManager对应的Web UI地址如下:

http://master1:8088

如下图YARN集群启动成功,集群中有两个节点【slave1、slave2】可供用来执行计算任务:

2. 测试HDFS和YARN的可用性

在HDFS创建目录

[root@master1 ~]# hdfs dfs -mkdir /input

将本地的文本文件上传到HDFS

  1. [root@master1 ~]# cd $HADOOP_HOME
  2. [root@master1 hadoop-2.10.1]# hdfs dfs -put README.txt /input

执行Hadoop自带的测试程序WordCount,此程序将统计输入目录下所有文件中的单词数量:

  1. [root@master1 hadoop-2.10.1]# cd $HADOOP_HOME
  2. [root@master1 hadoop-2.10.1]# cd share/hadoop/mapreduce
  3. [root@master1 mapreduce]# yarn jar hadoop-mapreduce-examples-2.10.1.jar wordcount /input /output

执行结果如下:

  1. [root@master1 mapreduce]# yarn jar hadoop-mapreduce-examples-2.10.1.jar wordcount /input /output
  2. 23/10/18 09:50:56 INFO client.RMProxy: Connecting to ResourceManager at master1/192.168.3.129:8032
  3. 23/10/18 09:50:56 INFO input.FileInputFormat: Total input files to process : 1
  4. 23/10/18 09:50:56 INFO mapreduce.JobSubmitter: number of splits:1
  5. 23/10/18 09:50:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1697619784960_0001
  6. 23/10/18 09:50:57 INFO conf.Configuration: resource-types.xml not found
  7. 23/10/18 09:50:57 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
  8. 23/10/18 09:50:57 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
  9. 23/10/18 09:50:57 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
  10. 23/10/18 09:50:57 INFO impl.YarnClientImpl: Submitted application application_1697619784960_0001
  11. 23/10/18 09:50:57 INFO mapreduce.Job: The url to track the job: http://master1:8088/proxy/application_1697619784960_0001/
  12. 23/10/18 09:50:57 INFO mapreduce.Job: Running job: job_1697619784960_0001
  13. 23/10/18 09:51:04 INFO mapreduce.Job: Job job_1697619784960_0001 running in uber mode : true
  14. 23/10/18 09:51:04 INFO mapreduce.Job: map 100% reduce 0%
  15. 23/10/18 09:51:06 INFO mapreduce.Job: map 100% reduce 100%
  16. 23/10/18 09:51:06 INFO mapreduce.Job: Job job_1697619784960_0001 completed successfully
  17. 23/10/18 09:51:06 INFO mapreduce.Job: Counters: 52
  18. File System Counters
  19. FILE: Number of bytes read=3704
  20. FILE: Number of bytes written=5572
  21. FILE: Number of read operations=0
  22. FILE: Number of large read operations=0
  23. FILE: Number of write operations=0
  24. HDFS: Number of bytes read=2992
  25. HDFS: Number of bytes written=439695
  26. HDFS: Number of read operations=41
  27. HDFS: Number of large read operations=0
  28. HDFS: Number of write operations=16
  29. Job Counters
  30. Launched map tasks=1
  31. Launched reduce tasks=1
  32. Other local map tasks=1
  33. Total time spent by all maps in occupied slots (ms)=0
  34. Total time spent by all reduces in occupied slots (ms)=0
  35. TOTAL_LAUNCHED_UBERTASKS=2
  36. NUM_UBER_SUBMAPS=1
  37. NUM_UBER_SUBREDUCES=1
  38. Total time spent by all map tasks (ms)=428
  39. Total time spent by all reduce tasks (ms)=1202
  40. Total vcore-milliseconds taken by all map tasks=0
  41. Total vcore-milliseconds taken by all reduce tasks=0
  42. Total megabyte-milliseconds taken by all map tasks=0
  43. Total megabyte-milliseconds taken by all reduce tasks=0
  44. Map-Reduce Framework
  45. Map input records=31
  46. Map output records=179
  47. Map output bytes=2055
  48. Map output materialized bytes=1836
  49. Input split bytes=101
  50. Combine input records=179
  51. Combine output records=131
  52. Reduce input groups=131
  53. Reduce shuffle bytes=1836
  54. Reduce input records=131
  55. Reduce output records=131
  56. Spilled Records=262
  57. Shuffled Maps =1
  58. Failed Shuffles=0
  59. Merged Map outputs=1
  60. GC time elapsed (ms)=10
  61. CPU time spent (ms)=1560
  62. Physical memory (bytes) snapshot=812826624
  63. Virtual memory (bytes) snapshot=6216478720
  64. Total committed heap usage (bytes)=562036736
  65. Shuffle Errors
  66. BAD_ID=0
  67. CONNECTION=0
  68. IO_ERROR=0
  69. WRONG_LENGTH=0
  70. WRONG_MAP=0
  71. WRONG_REDUCE=0
  72. File Input Format Counters
  73. Bytes Read=1366
  74. File Output Format Counters
  75. Bytes Written=4540

查看结果目录

[root@master1 mapreduce]# hdfs dfs -ls /output

回显如下

  1. [root@master1 mapreduce]# hdfs dfs -ls /output
  2. Found 2 items
  3. -rw-r--r-- 2 root supergroup 0 2023-10-18 09:51 /output/_SUCCESS
  4. -rw-r--r-- 2 root supergroup 1306 2023-10-18 09:51 /output/part-r-00000

其中【_SUCCESS】文件为标识文件,代表任务执行成功,以【part-r-】为前缀的文件为Reduce的输出文件,【part-r-XXXXX】文件个数与MapReduce中的Reduce个数对应,执行如下命令查看结果

[root@master1 mapreduce]# hdfs dfs -cat /output/part-r-*

结果如下:

  1. [root@master1 mapreduce]# hdfs dfs -cat /output/part-r-*
  2. (BIS), 1
  3. (ECCN) 1
  4. (TSU) 1
  5. (see 1
  6. 5D002.C.1, 1
  7. 740.13) 1
  8. <http://www.wassenaar.org/> 1
  9. Administration 1
  10. Apache 1
  11. BEFORE 1
  12. BIS 1
  13. Bureau 1
  14. Commerce, 1
  15. Commodity 1
  16. Control 1
  17. Core 1
  18. Department 1
  19. ENC 1
  20. Exception 1
  21. Export 2
  22. For 1
  23. Foundation 1
  24. Government 1
  25. Hadoop 1
  26. Hadoop, 1
  27. Industry 1
  28. Jetty 1
  29. License 1
  30. Number 1
  31. Regulations, 1
  32. SSL 1
  33. .
  34. .
  35. .

由此证明HDFS和YARN可以正常运行。

声明:本文内容由网友自发贡献,不代表【wpsshop博客】立场,版权归原作者所有,本站不承担相应法律责任。如您发现有侵权的内容,请联系我们。转载请注明出处:https://www.wpsshop.cn/w/在线问答5/article/detail/795180
推荐阅读
相关标签
  

闽ICP备14008679号