04 Pipes and Filters - ryandkuster/EPP_575_RNA_25 GitHub Wiki
Questions
- How can I combine existing commands to produce a desired output?
- How can I show only part of the output?
Objectives
- Explain the advantage of linking commands with pipes and filters.
- Combine sequences of commands to get new output
- Redirect a command’s output to a file.
- Explain what usually happens if a program or pipeline isn’t given any input to process.
Okay so let's begin, we will start with directory -
shell-lesson-data/exercise-data/alkanes
Let's list the files here -
ls
Next, let's run an example command -
wc ethane.pdb
Output -
12 84 622 ethane.pdb
wc is the 'word count' command - It counts the number of lines, words, and characters in files (returning the values in that order from left to right).
If we run the command wc *.pdb, the * in *.pdb matches zero or more characters, so the shell turns *.pdb into a list of all .pdb files in the current directory:
wc *.pdb
Output -
20 156 1158 cubane.pdb
12 84 622 ethane.pdb
9 57 422 methane.pdb
30 246 1828 octane.pdb
21 165 1226 pentane.pdb
15 111 825 propane.pdb
107 819 6081 total
Note that wc *.pdb also shows the total number of all lines in the last line of the output.
If we run wc -l instead of just wc, the output shows only the number of lines per file:
wc -l *.pdb
Output -
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
The -m and -w options can also be used with the wc command to show only the number of characters or the number of words, respectively.
QUESTION - What happens if a command is supposed to process a file, but we don’t give it a filename? For example, what if we type:
wc -l
but don’t type *.pdb (or anything else) after the command? Since it doesn’t have any filenames, wc assumes it is supposed to process input given at the command prompt, so it just sits there and waits for us to give it some data interactively. From the outside, though, all we see is it sitting there, and the command doesn’t appear to do anything.
If you make this kind of mistake, you can escape out of this state by holding down the control key (Ctrl) and pressing the letter C once: Ctrl+C. Then release both keys.
Capturing output from commands
Which of these files contains the most number of lines? It’s an easy question to answer when there are only six files, but what if there were 6000? Our first step toward a solution is to run the command:
wc -l *.pdb > lengths.txt
The greater than symbol, >, tells the shell to redirect the command’s output to a file instead of printing it to the screen. This command prints no screen output, because everything that wc would have printed has gone into the file lengths.txt instead. If the file doesn’t exist prior to issuing the command, the shell will create the file. If the file exists already, it will be silently overwritten, which may lead to data loss. Thus, redirect commands require caution.
ls lengths.txt confirms that the file exists:
ls lengths.txt
We can now send the content of lengths.txt to the screen using cat lengths.txt. The cat command gets its name from ‘concatenate’ i.e. join together, and it prints the contents of files one after another. There’s only one file in this case, so cat just shows us what it contains:
cat lengths.txt
Output -
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
NOTE - We’ll continue to use cat in this lesson, for convenience and consistency, but it has the disadvantage that it always dumps the whole file onto your screen. More useful in practice is the command less (e.g. less lengths.txt). This displays a screenful of the file, and then stops. You can go forward one screenful by pressing the spacebar, or back one by pressing b. Press q to quit.
Filtering output
Next we’ll use the sort command to sort the contents of the lengths.txt file. But first we’ll do an exercise to learn a little about the sort command. Go back up one directory where file numbers.txt exists.
cd ..
and list files here -
ls
Output -
alkanes animal-counts creatures numbers.txt writing
Let's 'concatenate' the contents of numbers.txt file -
cat numbers.txt
If we run sort numbers.txt, the output is -
sort numbers.txt
Output -
10
19
2
22
6
QUESTION - Can you guess what happened here?
Now, if we run sort -n numbers.txt, then output looks like -
2
6
10
19
22
Now, let's go back to the alkanes directory -
cd alkanes
and show or concatenate the contents of the lengths.txt file -
cat lengths.txt
Output -
20 cubane.pdb
12 ethane.pdb
9 methane.pdb
30 octane.pdb
21 pentane.pdb
15 propane.pdb
107 total
We will also use the -n option to specify that the sort is numerical instead of alphanumerical. This does not change the file; instead, it sends the sorted result to the screen:
sort -n lengths.txt
Output -
9 methane.pdb
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
107 total
We can put the sorted list of lines in another temporary file called sorted-lengths.txt by putting > sorted-lengths.txt after the command, just as we used > lengths.txt to put the output of wc into lengths.txt. Once we’ve done that, we can run another command called head to get the first few lines in sorted-lengths.txt:
sort -n lengths.txt > sorted-lengths.txt
$ head -n 1 sorted-lengths.txt
Output -
9 methane.pdb
Using -n 1 with head tells it that we only want the first line of the file; -n 20 would get the first 20, and so on. Since sorted-lengths.txt contains the lengths of our files ordered from least to greatest, the output of head must be the file with the fewest lines.
:grey_exclamation: REDIRECTING TO THE SAME FILE
It’s a very bad idea to try redirecting the output of a command that operates on a file to the same file. For example:
sort -n lengths.txt > lengths.txt
Doing something like this may give you incorrect results and/or delete the contents of lengths.txt.
:bulb: WHAT DOES >> MEAN?
We have seen the use of >, but there is a similar operator >> which works slightly differently. We’ll learn about the differences between these two operators by printing some strings. We can use the echo command to print strings e.g. use this command -
echo The echo command prints text
Output -
The echo command prints text
Now test the commands below to reveal the difference between the two operators:
echo hello > testfile01.txt
and -
echo hello >> testfile02.txt
Passing output to another command
What if we want to use two commands but in a single line of code? We can do something like this -
sort -n lengths.txt | head -n 1
Output -
9 methane.pdb
The vertical bar, |, between the two commands is called a pipe. It tells the shell that we want to use the output of the command on the left as the input to the command on the right.
This has removed the need for the sorted-lengths.txt file.
Combining multiple commands
We can chain multiple commands using pipes. Let's start building those -
wc -l *.pdb | sort -n
Output -
9 methane.pdb
12 ethane.pdb
15 propane.pdb
20 cubane.pdb
21 pentane.pdb
30 octane.pdb
107 total
We can then send that output through another pipe, to head, so that the full pipeline becomes:
wc -l *.pdb | sort -n | head -n 1
:question: In our current directory, how can we find the files which have the least number of lines. Try using pipes to get to the answer.
Now, let's back up one directory work with the file shell-lesson-data/exercise-data/animal-counts/animals.csv
What if we just want the second column from this file? Use command -
cut -d , -f 2 animals.csv
The cut command is used to remove or ‘cut out’ certain sections of each line in the file, and cut expects the lines to be separated into columns by a Tab character. A character used in this way is called a delimiter. In the example above we use the -d option to specify the comma as our delimiter character. We have also used the -f option to specify that we want to extract the second field (column). This gives the following output:
deer
rabbit
raccoon
rabbit
deer
fox
rabbit
bear
:question: The uniq command filters out adjacent matching lines in a file. How could you extend this pipeline (using uniq and another command) to find out what animals the file contains (without any duplicates in their names)?
Okay, so this is the time to apply our fresh skills on some real world biological data. Please download this gff3 (click on 'Download Raw File' button on the right) which contains the annotation information for white oak chromosome 1. Put this file in shell-lesson-data directory on your computer. Try to answer some of the questions below using the skills we just learned.
- How many lines are there in this annotation file?
- What different genomic intervals are annotated in this file?
- How many mRNA, exons, and introns are present in white oak chr 1?
- If you'd want to check if this file really just contains Chr 01 and not any other Chr, how can you do that?
FASTQ FORMAT
@SEQ_ID
GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
+
!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65
A FASTQ file has four line-separated fields per sequence:
- Field 1 begins with a '@' character and is followed by a sequence identifier and an optional description.
- Field 2 is the raw sequence letters.
- Field 3 begins with a '+' character and is optionally followed by the same sequence identifier (and any description) again.
- Field 4 encodes the quality values for the sequence in Field 2, and must contain the same number of symbols as letters in the sequence.