hdfs commands - isgaur/AWS-BigData-Solutions GitHub Wiki

  1. help HDFS Shell Command

Syntax of help hdfs Command

$ hadoop fs –help

Help hdfs shell command helps hadoop developers figure out all the available hadoop commands and how to use them.

HDFS Help Command

Variations of the Hadoop fs Help Command

​$ hadoop fs –help ls

Using the help command with a specific command lists the usage information along with the options to use the command.

Help Usage for ls Command

2 ### Usage HDFS Shell Command

$ hadoop fs –usage ls Usage command gives all the options that can be used with a particular hdfs command.

Usage Hadoop Command

3 ### ls HDFS Shell Command

Syntax for ls Hadoop Command -

$ hadoop fs –ls This command will list all the available files and subdirectories under default directory.For instance, in our example the default directory for Cloudera VM is /user/cloudera

Hadoop fs commands - ls

Variations of Hadoop ls Shell Command

$ hadoop fs –ls /

Returns all the available files and subdirectories present under the root directory.

hadoop fs commands ls /

$ hadoop fs –ls –R /user/cloudera

Returns all the available files and recursively lists all the subdirectories under /user/Cloudera

Hadoop ls - R HDFS Command

4 ### mkdir- Used to create a new directory in HDFS at a given location.

Example of HDFS mkdir Command -

$ hadoop fs –mkdir /user/cloudera/dezyre1

The above command will create a new directory named dezyre1 under the location /user/cloudera

mkdir hdfs shell command

Note : Cloudera and other hadoop distribution vendors provide /user/ directory with read/write permission to all users but other directories are available as read-only.Thus, to create a folder in the root directory, users require superuser permission as shown below -

$ sudo –u hdfs hadoop fs –mkdir /dezyre

This command will create a new directory named dezyre under the / (root directory).

mkdir with Superuser permission

5 ### copyFromLocal Copy a file from local filesytem to HDFS location.

For the following examples, we will use Sample.txt file available in the /home/Cloudera location.

Sample File for Executing Hadoop Commands

Example - $ hadoop fs –copyFromLocal Sample1.txt /user/cloudera/dezyre1

Copy/Upload Sample1.txt available in /home/cloudera (local default) to /user/cloudera/dezyre1 (hdfs path)

CopyFromLocal HDFS Command

  1. put –

This hadoop command uploads a single file or multiple source files from local file system to hadoop distributed file system (HDFS).

Ex - $ hadoop fs –put Sample2.txt /user/cloudera/dezyre1

Copy/Upload Sample2.txt available in /home/cloudera (local default) to /user/cloudera/dezyre1 (hdfs path)

hadoop fs PUT command

  1. moveFromLocal

This hadoop command functions similar to the put command but the source file will be deleted after copying.

Example - $ hadoop fs –moveFromLocal Sample3.txt /user/cloudera/dezyre1

Move Sample3.txt available in /home/cloudera (local default) to /user/cloudera/dezyre1 (hdfs path). Source file will be deleted after moving.

moveFromLocal Hadoop HDFS Command

Output of moveFromLocal Hadoop Command

  1. du

Displays the disk usage for all the files available under a given directory.

Example - $ hadoop fs –du /user/cloudera/dezyre1

HDFS du Command Usage

  1. df

Displas disk usage of current hadoop distributed file system. Example - $ hadoop fs –df

Hadoop HDFS df Command Usage

  1. Expunge

This HDFS command empties the trash by deleting all the files and directories.

Example - $ hadoop fs –expunge

Hadoop HDFS Expunge Command

  1. Cat

This is similar to the cat command in Unix and displays the contents of a file.

Example - $ hadoop fs –cat /user/cloudera/dezyre1/Sample1.txt

Hadoop fs cat Command

  1. cp

Copy files from one HDFS location to another HDFS location.

Example – $ hadoop fs –cp /user/cloudera/dezyre/war_and_peace /user/cloudera/dezyre1/

Cp HDFS Command

Hadoop cp Command

  1. mv

Move files from one HDFS location to another HDFS location.

Example – $ hadoop fs –mv /user/cloudera/dezyre1/Sample1.txt /user/cloudera/dezyre/

ls HDFS Command

ls Hadoop HDFS Command

hadoop fs move command

fs ls hadoop command

After executing fs ls command

  1. rm

Removes the file or directory from the mentioned HDFS location.

Example – $ hadoop fs –rm -r /user/cloudera/dezyre3

Hadoop fs rm Commandrm -r

Example – $ hadoop fs –rm -r /user/cloudera/dezyre3

Deletes or removes the directory and its content from HDFS location in a recursive manner.

Remove directory from HDFS

Example – $ hadoop fs –rm /user/cloudera/dezyre3

Delete or remove the files from HDFS location.

Remove the files or directories from a specific location

  1. tail

This hadoop command will show the last kilobyte of the file to stdout.

Example – $ hadoop fs -tail /user/cloudera/dezyre/war_and_peace

Hadoop HDFS tail Command

Example – $ hadoop fs -tail –f /user/cloudera/dezyre/war_and_peace

Using the tail commands with -f option, shows the last kilobyte of the file from end in a page wise format.

tail Hadoop Command with -f option

  1. copyToLocal

Copies the files to the local filesystem . This is similar to hadoop fs -get command but in this case the destination location msut be a local file reference

Example - $ hadoop fs –copyFromLocal /user/cloudera/dezyre1/Sample1.txt /home/cloudera/hdfs_bkp/

Copy/Download Sample1.txt available in /user/cloudera/dezyre1 (hdfs path) to /home/cloudera/hdfs_bkp/ (local path)

CopyToLocal Hadoop Command

  1. get

Downloads or Copies the files to the local filesystem.

Example - $ hadoop fs –get /user/cloudera/dezyre1/Sample2.txt /home/cloudera/hdfs_bkp/

Copy/Download Sample2.txt available in /user/cloudera/dezyre1 (hdfs path) to /home/cloudera/hdfs_bkp/ (local path)

hadoop fs -get command

  1. touchz

Used to create an emplty file at the specified location.

Example - $ hadoop fs –touchz /user/cloudera/dezyre1/Sample4.txt

It will create a new empty file Sample4.txt in /user/cloudera/dezyre1/ (hdfs path)

hadoop fs -touchz command

  1. setrep

This hadoop fs command is used to set the replication for a specific file.

Example - $ hadoop fs –setrep –w 1 /user/cloudera/dezyre1/Sample1.txt

It will set the replication factor of Sample1.txt to 1

Set the replication factor of the file

  1. chgrp

This hadoop command is basically used to change the group name.

Example - $ sudo –u hdfs hadoop fs –chgrp –R cloudera /dezyre

It will change the /dezyre directory group membership from supergroup to cloudera (To perform this operation superuser permission is required)

Hadoop chgrp Command

chgrp HDFS Hadoop Command

  1. chown

This command lets you change both the owner and group name simulataneously.

Example - $ sudo –u hdfs hadoop fs –chown –R cloudera /dezyre

It will change the /dezyre directory ownership from hdfs user to cloudera user (To perform this operation superuser is permission required)

chown hadoop command

22)### hadoop chmod

Used to change the permissions of a given file/dir.

Example - $ hadoop fs –chmod /dezyre

It will change the /dezyre directory permission to 700 (drwx------).

chmod Hadoop fs command

Note : hadoop chmod 777

To execute this , the user must be the owner of the file or must be a super user. On executing this command, all users will get read,write and execute permission on the file.