热门标签
热门文章
- 1一个php快速项目搭建框架源码，带一键CURD等功能_cms curd
- 2“八股文”在程序员面试中的作用及其对实际工作的影响_程序员八股文指的是什么
- 3Vue+ElementUI+SpringMVC实现图片上传和回显_vue可以将mysql保存的图像地址回显页面中么
- 4深度学习框架 —— 分布式训练
- 5mysql util_mysql jdbcUtil
- 6ESP8266APWeb页面控制开关三个灯附详细指导视频和源码_esp8266网页控制
- 7细数AI程序员 Devin、SWE-agent、通义灵码、Devika...，“码农”要失业了吗？_devin、devika,gpt-pilot,chatdev,codeflying对比
- 8cuda11.7+visual studio 2022 编译时出现MSB3721问题_vs cuda error msb3721: 命令 返回代码为2
- 9Linux环境搭建Hadoop及完全分布式集群_linux安装hadoop
- 10华为 UltraPixel: 最强高清修复模型现已开源可用_华为开源的ultrapixel
当前位置: article > 正文
Hadoop大数据应用：Linux 部署 HDFS 分布式集群_linux操作hdfs(1)_hdfs 配置文件fs.defaultfs
作者：空白诗007 | 2024-08-19 13:39:27
赞
踩
hdfs 配置文件fs.defaultfs
https://hadoop.apache.org/docs/r2.7.7/
- 1
查看核心配置文件
https://hadoop.apache.org/docs/r2.7.7/hadoop-project-dist/hadoop-common/core-default.xml
- 1
文件系统配置参数：
数据目录配置参数：
（8）修改核心配置文件
[root@hadoop hadoop]# vim core-site.xml
- 1
- 2
修改前：
修改后：
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop:9000</value>
<description>hdfs file system</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/var/hadoop</value>
</property>
</configuration>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
（9）查看HDFS配置文件
https://hadoop.apache.org/docs/r2.7.7/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
- 1
namenode:
副本数量：
（10）修改HDFS配置文件
[root@hadoop hadoop]# vim hdfs-site.xml
- 1
- 2
修改前：
修改后：
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop:50070</value>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop:50090</value>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
(11) 查看同步
[root@hadoop ~]# rpm -q rsync
- 1
- 2
同步
[root@hadoop ~]# rsync -aXSH --delete /usr/local/hadoop node01:/usr/local/
[root@hadoop ~]# rsync -aXSH --delete /usr/local/hadoop node02:/usr/local/
[root@hadoop ~]# rsync -aXSH --delete /usr/local/hadoop node03:/usr/local/
- 1
- 2
- 3
- 4
（12）初始化hdfs
[root@hadoop ~]# mkdir /var/hadoop
- 1
- 2
（13）查看命令
[root@hadoop hadoop]# ./bin/hdfs Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND where COMMAND is one of: dfs run a filesystem command on the file systems supported in Hadoop. classpath prints the classpath namenode -format format the DFS filesystem secondarynamenode run the DFS secondary namenode namenode run the DFS namenode journalnode run the DFS journalnode zkfc run the ZK Failover Controller daemon datanode run a DFS datanode dfsadmin run a DFS admin client haadmin run a DFS HA admin client fsck run a DFS filesystem checking utility balancer run a cluster balancing utility jmxget get JMX exported values from NameNode or DataNode. mover run a utility to move block replicas across storage types oiv apply the offline fsimage viewer to an fsimage oiv_legacy apply the offline fsimage viewer to an legacy fsimage oev apply the offline edits viewer to an edits file fetchdt fetch a delegation token from the NameNode getconf get config values from configuration groups get the groups which users belong to snapshotDiff diff two snapshots of a directory or diff the current directory contents with a snapshot lsSnapshottableDir list all snapshottable dirs owned by the current user Use -help to see options portmap run a portmap service nfs3 run an NFS version 3 gateway cacheadmin configure the HDFS cache crypto configure HDFS encryption zones storagepolicies list/get/set block storage policies version print the version Most commands print help when invoked w/o parameters.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
（14）格式化hdfs
[root@hadoop hadoop]# ./bin/hdfs namenode -format
- 1
- 2
查看目录
[root@hadoop hadoop]# cd /var/hadoop/
[root@hadoop hadoop]# tree .
.
└── dfs
└── name
└── current
├── fsimage_0000000000000000000
├── fsimage_0000000000000000000.md5
├── seen_txid
└── VERSION
3 directories, 4 files
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
(15) 启动集群
查看目录
[root@hadoop hadoop]# cd ~
[root@hadoop ~]# cd /usr/local/hadoop/
[root@hadoop hadoop]# ls
- 1
- 2
- 3
- 4
启动
[root@hadoop hadoop]# ./sbin/start-dfs.sh
- 1
- 2
查看日志（新生成logs目录）
[root@hadoop hadoop]# cd logs/ ; ll
- 1
- 2
查看jps
[root@hadoop hadoop]# jps
- 1
- 2
datanode节点查看（node01）
datanode节点查看（node02）
datanode节点查看（node03）
(16)查看命令
[root@hadoop hadoop]# ./bin/hdfs dfsadmin Usage: hdfs dfsadmin Note: Administrative commands can only be run as the HDFS superuser. [-report [-live] [-dead] [-decommissioning]] [-safemode <enter | leave | get | wait>] [-saveNamespace] [-rollEdits] [-restoreFailedStorage true|false|check] [-refreshNodes] [-setQuota <quota> <dirname>...<dirname>] [-clrQuota <dirname>...<dirname>] [-setSpaceQuota <quota> [-storageType <storagetype>] <dirname>...<dirname>] [-clrSpaceQuota [-storageType <storagetype>] <dirname>...<dirname>] [-finalizeUpgrade] [-rollingUpgrade [<query|prepare|finalize>]] [-refreshServiceAcl] [-refreshUserToGroupsMappings] [-refreshSuperUserGroupsConfiguration] [-refreshCallQueue] [-refresh <host:ipc_port> <key> [arg1..argn] [-reconfig <datanode|...> <host:ipc_port> <start|status>] [-printTopology] [-refreshNamenodes datanode_host:ipc_port] [-deleteBlockPool datanode_host:ipc_port blockpoolId [force]] [-setBalancerBandwidth <bandwidth in bytes per second>] [-fetchImage <local directory>] [-allowSnapshot <snapshotDir>] [-disallowSnapshot <snapshotDir>] [-shutdownDatanode <datanode_host:ipc_port> [upgrade]] [-getDatanodeInfo <datanode_host:ipc_port>] [-metasave filename] [-triggerBlockReport [-incremental] <datanode_host:ipc_port>] [-help [cmd]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
（17）验证集群
查看报告，发现3个节点
[root@hadoop hadoop]# ./bin/hdfs dfsadmin -report Configured Capacity: 616594919424 (574.25 GB) Present Capacity: 598915952640 (557.78 GB) DFS Remaining: 598915915776 (557.78 GB) DFS Used: 36864 (36 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (3): Name: 192.168.204.53:50010 (node03) Hostname: node03 Decommission Status : Normal Configured Capacity: 205531639808 (191.42 GB) DFS Used: 12288 (12 KB) Non DFS Used: 5620584448 (5.23 GB) DFS Remaining: 199911043072 (186.18 GB) DFS Used%: 0.00% DFS Remaining%: 97.27% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Thu Mar 14 10:30:18 CST 2024 Name: 192.168.204.51:50010 (node01) Hostname: node01 Decommission Status : Normal Configured Capacity: 205531639808 (191.42 GB) DFS Used: 12288 (12 KB) Non DFS Used: 6028849152 (5.61 GB) DFS Remaining: 199502778368 (185.80 GB) DFS Used%: 0.00% DFS Remaining%: 97.07% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Thu Mar 14 10:30:18 CST 2024 Name: 192.168.204.52:50010 (node02) Hostname: node02 Decommission Status : Normal Configured Capacity: 205531639808 (191.42 GB) DFS Used: 12288 (12 KB) Non DFS Used: 6029533184 (5.62 GB) DFS Remaining: 199502094336 (185.80 GB) DFS Used%: 0.00% DFS Remaining%: 97.07% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Thu Mar 14 10:30:18 CST 2024
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
- 56
- 57
- 58
- 59
- 60
- 61
- 62
- 63
- 64
- 65
- 66
- 67
（18）web页面验证
http://192.168.204.50:50070/
- 1
http://192.168.204.50:50090/
- 1
http://192.168.204.51:50075/
- 1
(19)访问系统
目前为空
3.Linux 使用 HDFS 文件系统
（1）查看命令
[root@hadoop hadoop]# ./bin/hadoop Usage: hadoop [--config confdir] [COMMAND | CLASSNAME] CLASSNAME run the class named CLASSNAME or where COMMAND is one of: fs run a generic filesystem user client version print the version jar <jar> run a jar file note: please use "yarn jar" to launch YARN applications, not this command. checknative [-a|-h] check native hadoop and compression libraries availability distcp <srcurl> <desturl> copy file or directories recursively archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive classpath prints the class path needed to get the credential interact with credential providers Hadoop jar and the required libraries daemonlog get/set the log level for each daemon trace view and modify Hadoop tracing settings Most commands print help when invoked w/o parameters.
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
[root@hadoop hadoop]# ./bin/hadoop fs Usage: hadoop fs [generic options] [-appendToFile <localsrc> ... <dst>] [-cat [-ignoreCrc] <src> ...] [-checksum <src> ...] [-chgrp [-R] GROUP PATH...] [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...] [-chown [-R] [OWNER][:[GROUP]] PATH...] [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>] [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-count [-q] [-h] <path> ...] [-cp [-f] [-p | -p[topax]] <src> ... <dst>] [-createSnapshot <snapshotDir> [<snapshotName>]] [-deleteSnapshot <snapshotDir> <snapshotName>] [-df [-h] [<path> ...]] [-du [-s] [-h] <path> ...] [-expunge] [-find <path> ... <expression> ...] [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>] [-getfacl [-R] <path>] [-getfattr [-R] {-n name | -d} [-e en] <path>] [-getmerge [-nl] <src> <localdst>] [-help [cmd ...]] [-ls [-d] [-h] [-R] [<path> ...]] [-mkdir [-p] <path> ...] [-moveFromLocal <localsrc> ... <dst>] [-moveToLocal <src> <localdst>] [-mv <src> ... <dst>] [-put [-f] [-p] [-l] <localsrc> ... <dst>] [-renameSnapshot <snapshotDir> <oldName> <newName>] [-rm [-f] [-r|-R] [-skipTrash] <src> ...] [-rmdir [--ignore-fail-on-non-empty] <dir> ...] [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]] [-setfattr {-n name [-v value] | -x name} <path>] [-setrep [-R] [-w] <rep> <path> ...] [-stat [format] <path> ...] [-tail [-f] <file>] [-test -[defsz] <path>] [-text [-ignoreCrc] <src> ...] [-touchz <path> ...] [-truncate [-w] <length> <path> ...] [-usage [cmd ...]] Generic options supported are -conf <configuration file> specify an application configuration file -D <property=value> use value for given property -fs <local|namenode:port> specify a namenode -jt <local|resourcemanager:port> specify a ResourceManager -files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster -libjars <comma separated list of jars> specify comma separated jar files to include in the classpath. -archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines. The general command line syntax is bin/hadoop command [genericOptions] [commandOptions]
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
- 15
- 16
- 17
- 18
- 19
- 20
- 21
- 22
- 23
- 24
- 25
- 26
- 27
- 28
- 29
- 30
- 31
- 32
- 33
- 34
- 35
- 36
- 37
- 38
- 39
- 40
- 41
- 42
- 43
- 44
- 45
- 46
- 47
- 48
- 49
- 50
- 51
- 52
- 53
- 54
- 55
（2）查看文件目录
[root@hadoop hadoop]# ./bin/hadoop fs -ls /
- 1
- 2
（3）创建文件夹
[root@hadoop hadoop]# ./bin/hadoop fs -mkdir /devops
- 1
- 2
查看
查看web
（4）上传文件
[root@hadoop hadoop]# ./bin/hadoop fs -put *.txt /devops/
- 1
- 2
查看
[root@hadoop hadoop]# ./bin/hadoop fs -ls /devops/
- 1
- 2
查看web
Permission Owner Group Size Last Modified Replication Block Size Name
-rw-r--r-- root supergroup 84.4 KB 2024/3/14 11:05:33 2 128 MB LICENSE.txt
-rw-r--r-- root supergroup 14.63 KB 2024/3/14 11:05:34 2 128 MB NOTICE.txt
-rw-r--r-- root supergroup 1.33 KB 2024/3/14 11:05:34 2 128 MB README.txt
- 1
- 2
- 3
- 4
下载
（5）创建文件
[root@hadoop hadoop]# ./bin/hadoop fs -touchz /tfile
- 1
- 2
查看
[root@hadoop hadoop]# ./bin/hadoop fs -ls /
- 1
- 2
（5）下载文件
[root@hadoop hadoop]# ./bin/hadoop fs -get /tfile /tmp/
- 1
- 2
查看
[root@hadoop hadoop]# ls -l /tmp/ | grep tfile
- 1
- 2
查看web
(6) 查看命令比较
之前的设置
所以查看功能相同
[root@hadoop hadoop]# ./bin/hadoop fs -ls /
[root@hadoop hadoop]# ./bin/hadoop fs -ls hdfs://hadoop:9000/
- 1
- 2
- 3
- 4
另外官网默认是file ，使用的是本地文件目录
[root@hadoop hadoop]# ./bin/hadoop fs -ls file:///
- 1
- 2
二、问题
1.ssh-copy-id 报错
（1）报错
/usr/bin/ssh-copy-id: ERROR: ssh: connect to host hadoop port 22: Connection refused
- 1
- 2
（2）原因分析
主机解析错误。
（3）解决方法
修改前：
修改后：
成功：
2. 如何禁用ssh key 检测
（1）修改配置文件
[root@hadoop .ssh]# vim /etc/ssh/ssh_config
- 1
- 2
添加配置
StrictHostKeyChecking no
- 1
- 2
成功：
3.HDFS有哪些配置文件
（1）配置文件
1）环境配置文件
hadoop-env.sh
2)核心配置文件
core-site.xml
3)HDFS配置文件
hdfs-site.xml
4)节点配置文件
slaves
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
4.hadoop查看版本报错
(1) 报错
（2）原因分析
未申明JAVA环境。
（3）解决方法
申明JAVA环境。
网上学习资料一大堆，但如果学到的知识不成体系，遇到问题时只是浅尝辄止，不再深入研究，那么很难做到真正的技术提升。
一个人可以走的很快，但一群人才能走的更远！不论你是正从事IT行业的老鸟或是对IT行业感兴趣的新人，都欢迎加入我们的的圈子（技术交流、学习资源、职场吐槽、大厂内推、面试辅导），让我们一起学习成长！
声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/空白诗007/article/detail/1002497
推荐阅读
相关标签
