当前位置:网站首页>第2章 HFDS的Shell操作
第2章 HFDS的Shell操作
2022-07-06 09:29:00 【留不住斜阳】
2.1 基本语法
hadoop fs 具体命令
或
hdfs dfs 具体命令
2.2 命令大全
具体命令如下
Usage: hadoop fs [generic options]
[-appendToFile <localsrc> ... <dst>]
[-cat [-ignoreCrc] <src> ...]
[-checksum <src> ...]
[-chgrp [-R] GROUP PATH...]
[-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
[-chown [-R] [OWNER][:[GROUP]] PATH...]
[-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
[-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-count [-q] [-h] [-v] [-x] <path> ...]
[-cp [-f] [-p | -p[topax]] <src> ... <dst>]
[-createSnapshot <snapshotDir> [<snapshotName>]]
[-deleteSnapshot <snapshotDir> <snapshotName>]
[-df [-h] [<path> ...]]
[-du [-s] [-h] [-x] <path> ...]
[-expunge]
[-find <path> ... <expression> ...]
[-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
[-getfacl [-R] <path>]
[-getfattr [-R] {
-n name | -d} [-e en] <path>]
[-getmerge [-nl] <src> <localdst>]
[-help [cmd ...]]
[-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
[-mkdir [-p] <path> ...]
[-moveFromLocal <localsrc> ... <dst>]
[-moveToLocal <src> <localdst>]
[-mv <src> ... <dst>]
[-put [-f] [-p] [-l] <localsrc> ... <dst>]
[-renameSnapshot <snapshotDir> <oldName> <newName>]
[-rm [-f] [-r|-R] [-skipTrash] <src> ...]
[-rmdir [--ignore-fail-on-non-empty] <dir> ...]
[-setfacl [-R] [{
-b|-k} {
-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
[-setfattr {
-n name [-v value] | -x name} <path>]
[-setrep [-R] [-w] <rep> <path> ...]
[-stat [format] <path> ...]
[-tail [-f] <file>]
[-test -[defsz] <path>]
[-text [-ignoreCrc] <src> ...]
[-touchz <path> ...]
[-usage [cmd ...]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|resourcemanager:port> specify a ResourceManager
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
2.3 常用命令
注意:如果提示root权限不能进行写操作,请切换到hdfs用户操作
2.3.1 -help
帮助命令
hadoop fs -help rm
或
hdfs dfs -help rm
注意:hadoop脚本、hdfs脚本,在hadoop安装目录的bin目录下
2.3.2 -ls
显示目录信息
hadoop fs -ls /
或
hdfs dfs –ls /
2.3.3 -mkdir
在hdfs上创建目录
hadoop fs -mkdir -p /teaching/hdfs
或
hdfs dfs -mkdir -p /teaching/hdfs
2.3.4 -moveFromLocal
从本地剪切后粘贴到hdfs
touch test.txt
hadoop fs -moveFromLocal ./test.txt /teaching/hdfs
或
hdfs dfs –moveFromLocal ./test.txt /teaching/hdfs
注意:执行过程为先复制,后删除,如果hdfs用户不具有删除本地文件权限则复制到hdfs能成功,删除本地文件不能成功
2.3.5 -appendToFile
追加一个文件到已经存在的文件末尾
touch test2.txt
vi test2.txt
# 输入
hello hdfs
hadoop fs -appendToFile test2.txt /teaching/hdfs/test.txt
或
hdfds dfs -appendToFile test2.txt /teaching/hdfs/test.txt
2.3.6 -cat
查看文件内容
hadoop fs -cat /teaching/hdfs/test.txt
或
hdfs dfs –cat /teaching/hdfs/test.txt
2.3.7 -tail
显示文件最后1kb内容
hadoop fs -tail /teaching/hdfs/test.txt
或
hdfs dfs -tail /teaching/hdfs/test.txt
2.3.8 -chgrp、-chmod、-chown
修改文件所属组、权限、用户
- 修改HDFS存储的文件/teaching/hdfs/tt.txt文件所属组,默认组:supergroup选项-R递归执行
$ hdfs dfs -ls /teaching/hdfs
-rw-r--r-- 3 hdfs supergroup 11 2019-08-12 14:40 /teaching/hdfs/tt.txt
$ hdfs dfs -chgrp lubin-group /teaching/hdfs/tt.txt
$ hdfs dfs -ls /teaching/hdfs
-rw-r--r-- 3 hdfs lubin-group 11 2019-08-12 14:40 /teaching/hdfs/tt.txt
- 修改HDFS存储的文件/teaching/hdfs/tt.txt文件的权限
$ hdfs dfs -chmod 666 /teaching/hdfs/tt.txt
$ hdfs dfs -ls /teaching/hdfs
-rw-rw-rw- 3 hdfs lubin-group 11 2019-08-12 14:40 /teaching/hdfs/tt.txt
- 修改HDFS存储的文件/teaching/hdfs/tt.txt文件所属的用户和组,lubin1为用户,lubin2为组
$ hdfs dfs -chown lubin1:lubin2 /teaching/hdfs/tt.txt
$ hdfs dfs -ls /teaching/hdfs/tt.txt
-rw-rw-rw- 3 lubin1 lubin2 11 2019-08-12 14:40 /teaching/hdfs/tt.txt
2.3.9 -copyFromLocal
从本地文件系统拷贝文件到hdfs路径
hadoop fs -copyFromLocal README.txt /teaching/hdfs
或
hdfs dfs -copyFromLocal test3.txt /teaching/hdfs
2.3.10 -copyToLocal
从hdfs拷贝到本地
hadoop fs -copyToLocal /teaching/hdfs/test.txt /tmp
或
hdfs dfs –copyToLocal /teaching/hdfs/test.txt /tmp
2.3.11 -cp
从hdfs的一个路径拷贝到hdfs的另一个路径
hadoop fs -cp /teaching/hdfs/test3.txt /teaching/hdfs2
或
hdfs dfs -cp /teaching/hdfs/test3.txt /teaching/hdfs2
2.3.12 -mv
在hdfs目录中移动文件
hadoop fs -mv /teaching/hdfs/tt.txt /teaching/hdfs2
或
hdfs dfs -mv /teaching/hdfs/tt.txt /teaching/hdfs2
2.3.13 -get
等同于copyToLocal,从hdfs下载文件到本地
hadoop fs -get /teaching/hdfs/test3.txt /tmp
或
hdfs dfs -get /teaching/hdfs/test3.txt /tmp
2.3.14 -getmerge
合并下载多个文件(多个文件的内容合并到一个文件中),比如hdfs的目录/aaa/下有多个文件:log.1, log.2,log.3,…
hadoop fs -getmerge /teaching/hdfs/log*.txt /tmp/log.txt
或
hdfs dfs -getmerge /teaching/hdfs/log*.txt /tmp/log.txt
2.3.15 -put
等同于copyFromLocal
hadoop fs -put log1.txt /teaching/hdfs
或
hdfs dfs –put log1.txt /teaching/hdfs
2.3.16 -rm
删除文件或文件夹
hadoop fs -rm /teaching/hdfs2/log1.txt
或
hdfs dfs –rm /teaching/hdfs2/log2.txt
注意:如果用户没有删除的权限,可以使用如下方式删除:su hdfs -c “hadoop fs -rm /test/test.txt”
2.3.17 -rmdir
删除空目录
hadoop fs -rmdir /test
或
hdfs dfs –rmdir /test
注意:上述命令只能删除空目录,删除非空目录用rm命令
2.3.18 -du
统计文件夹的大小信息
hadoop fs -du -s -h /teaching/hdfs
或
hdfs dfs -du -s -h /teaching/hdfs
58 174 /teaching/hdfs
注意:-s显示总(摘要)大小,去掉-s则显示每个匹配的文件大小;
-h以人类可读的方式格式化大小,不是用字节去展示;
第一列显示目录下总文件大小,第二列标示该目录下所有文件在集群上的总存储大小(与副本数有关),第三列是查询的目录名称
2.3.19 -setrep
设置hdfs中文件的副本数量
hadoop fs -setrep 10 /teaching/hdfs/test.txt
或
hdfs dfs –setrep 2 /teaching/hdfs /test.txt
设置的副本数只是记录在NameNode的元数据中,是否真的会有这么多副本,还得看DataNode的数量。因为目前只有3台设备,最多也就3个副本,如果设置为10,只有节点数的增加到10台时,副本数才能达到10。
边栏推荐
- Specify the format time, and fill in zero before the month and days
- Install Jupiter notebook under Anaconda
- Summary of FTP function implemented by qnetworkaccessmanager
- Discussion on QWidget code setting style sheet
- Share an example of running dash application in raspberry pie.
- 力扣:第81场双周赛
- 解决Intel12代酷睿CPU单线程调度问题(二)
- Codeforces Round #800 (Div. 2)AC
- Problem - 1646C. Factorials and Powers of Two - Codeforces
- 拉取分支失败,fatal: ‘origin/xxx‘ is not a commit and a branch ‘xxx‘ cannot be created from it
猜你喜欢
Hbuilder X格式化快捷键设置
Installation and configuration of MariaDB
Codeforces Round #802(Div. 2)A~D
QT有关QCobobox控件的样式设置(圆角、下拉框,向上展开、可编辑、内部布局等)
“鬼鬼祟祟的”新小行星将在本周安全掠过地球:如何观看
新手必会的静态站点生成器——Gridsome
Pull branch failed, fatal: 'origin/xxx' is not a commit and a branch 'xxx' cannot be created from it
Anaconda下安装Jupyter notebook
sublime text 代码格式化操作
Maximum product (greedy)
随机推荐
VMware Tools和open-vm-tools的安装与使用:解决虚拟机不全屏和无法传输文件的问题
Codeforces Round #803 (Div. 2)A~C
Problem - 922D、Robot Vacuum Cleaner - Codeforces
(POJ - 1458) common subsequence (longest common subsequence)
Double specific tyrosine phosphorylation regulated kinase 1A Industry Research Report - market status analysis and development prospect prediction
QT按钮点击切换QLineEdit焦点(含代码)
OneForAll安装使用
2078. Two houses with different colors and the farthest distance
Acwing: Game 58 of the week
Codeforces Round #797 (Div. 3)无F
Codeforces - 1526C1&&C2 - Potions
Read and save zarr files
Advancedinstaller installation package custom action open file
Codeforces Round #803 (Div. 2)A~C
Candy delivery (Mathematics)
日期加1天
(lightoj - 1354) IP checking (Analog)
Advancedinstaller安装包自定义操作打开文件
Discussion on QWidget code setting style sheet
QT实现窗口渐变消失QPropertyAnimation+进度条