Article No: 3
TOOLS: FILTERS AND UTILITIES
One of the most valuable features of the UNIX System is the rich set of commands it gives you. This chapter surveys a particularly useful set of commands that are often referred to as tools or utilities. These are collection of small, modular commands, each of which performs a specific function, such as sorting a list into order, searching for a word in a file, or joining two files together on a common field. You can use them singly and in combination to carry out many common office tasks.
Most of the tools described in this chapter are what are often referred to as filters. Filters are programs that read standard input, operate on it, and produce the result as standard output. They are not interactive – they do not prompt you or wait for input. For example, when you use the sort command, you tell it where to get its input and where to send its output, and specify any options that you choose; then sort goes off and does the rest. A filter can be used with other commands in a command pipeline as well as directly. Because its output is a simple stream of information, it can itself be used in a pipeline as input to another command. For example, you can use a pipe to connect the output of sort to the uniq command to count multiple instances of the same name list.
Most of the UNIX system tools can take their input from files, pipelines, or from the keyboard. Similarly, they write output to other files, to pipelines, or to the screen. You control the input and output by specifying them on the command line, along with the command name. It is often useful to try a command first using a few lines of input from your keyboard, to test whether it works the way you expected , and then repeat it, taking the input from a file. Similarly, you may wish to send the output to the screen to check it before sending to it a file, you can test the effects of a filter without worrying that you will change the contents of your file.
Most of the tools described in this chapter are designed to work with text and text files. Text files can be created by a text editor such as ed or vi, or with a word processor, if you save the file as text. They are stored in a ASCII format. You can view them directly with the cat command. Typical examples of text files include letters, memos, and documents, as well as lists, phone directories, and other data that can take the form of structured files. Documents created by database and word processing programs are not normally text files, but in most cases you can provide a version in ASCII format on which you can then use the tools described in this chapter.
You can use the commands described in this chapter to search for words in files, to search a file when you have forgotten its name, to sort a list of names or zip codes into ascending or descending order, or to select a part of a database or structured file and send it to another file. By combining tools in pipelines, you can do more complicated tasks such as performing sophisticated frequency analyses or dealing with structured files.
Some of the tools described in this chapter have features that are especially useful in dealing with files containing structured lists. Such files are often used as simple databases. Typically, each line in the file is a separate record containing information about a particular item. The information is often structured in fields. For example, each line in a personnel file may contain record consisting of information about one employee, with fields for name, address, phone number, and so forth. Fields in such files are typically separated by a field separator character, for example, a tab, comma or colon. Typical examples of structured files include telephone lists, recipes, indexes and merchandise inventories.
You will see that many of the commands described in chapter can be used for working with structured files, since they can operate on individual fields or columns as well as on entire lines.
In addition to tools that act as filters, this chapter also describes a number of other tools, including commands for getting time and date information, two numerical calculator programs, and commands for monitoring input and output.
Most of the tools described here can be found in any standard UNIX or Linux system. A few, such as patch, tac and look, come with Linux but not part of the standard UNIX command set.
SEARCHING PATTERNS FROM THE FILES
1. $grep command
Use:
The grep command searches through one or more files containing a target and then prints all of the matching lines it finds.
Syntax:
grep <options> <”pattern”> <filename(s)>
Example:
1. grep “Amit” sybcaa
It will search pattern “Amit” from sybcaa file, and display all the lines containing the pattern “Amit”.
2. grep “Amit” sybcaa bca
It will search pattern “Amit” from sybcaa and bca, and will display all lines containing the pattern.
3. grep “a pen” stationery
The pattern can be phrase-that is, two or more words separated by spaces. If the target contains spaces, however, you have to enclose it in quotes to prevent the shell from treating different words as separate arguments.
It will search “a pen” pattern from file stationery. “a” and “pen” must be together.
Options:
Ignoring the case (-i)
Normally, grep distinguishes between uppercase and lowercase. Sometimes, though, you don’t care about uppercase and lowercase distinctions. For example, you may want to find a word whether or not it is the first in a sentence, or you may want to find all instances of a name in a list in which some entries are uppercase and others are not.
grep –i “Amit” sybcaa
It will search all the patterns like Amit, AMIT, amit and it will display those lines without checking cases.
Deleting Lines (-v)
By default, grep finds all lines that match the target pattern. Sometimes, though, it is useful to find the lines that do not match a particular pattern. You can do this with the –v option, which tells grep to print all lines that do not contain the specified target. This provides a quick way to find entries in a file that are missing a required piece of information.
grep –v “amit” sybcaa
It will display all the lines which does not contains any word “amit” in any line.
Displaying Line Numbers (-n)
It will display the line number containing the pattern.
grep –n “amit” sybcaa9
Counting Lines containing patterns (-c)
It will count the number of lines in which the pattern has occurred.
grep –c “amit” sybcaa
5
In five lines, the pattern “amit” has occurred.
Displaying file names (-l)
It will list all the files in which the patterns have occurred.
grep –l “amit”
f1
f2
f3
……..
F1,f2, and f3 are the files that contains the pattern “amit”.
Matching multiple patterns (-e)
It will match for multiple patterns from the file.
grep –e “amit” –e “pinky” –e “abhay” sybcaa
It will display all the lines that contain the patterns like amit, pinky, and abhay.
Matching for exact pattern (-w)
It will match for exact pattern from the file.
grep –w “pinky” sybcaa
It will display all the lines that contain the exact patterns. It means it will not display the lines like, pink, pin, pinkyy..etc.
Matching for exact pattern (-x)
It will match for entire line from the files.
grep –x “Myself Amit Chaudhary” sybcaa
2. $fgrep
The fgrep command is similar to grep, but with three main differences: You can use it to search for several targets at once, it does not allow you to use regular expressions to search for patterns, and it is faster than grep. When you need to search a large file or several smaller files the difference in speed can be significant.
Use: It is used to match multiple patterns same time.
Syntax: fgrep “<pattern1> <pattern2> <pattern3”> filename
Example:
fgrep “amit
>pinky
>abhay” file1
It will display all the lines that containing patterns like amit, pinky or abhay.
3. $egrep
The egrep command accepts all of the basic regular expressions recognized by grep, as well as several useful extensions to the set. These include use of the plus sign (+) to indicate one or more repetitions of a character, and the question mark (?) to specify zero or one instance of a character.
The egrep command provides most of the basic options of both grep and fgrep. You can tell it to ignore uppercase and lowercase distinctions (-i), print lines that do not contain the target (-v), and take the list of targets from a file (-f).
Use: It is used to match dissimilar patterns with a single expression from file.
Syntax: egrep <”pattern”> <filename(s)>
Example:
egrep “[Aa]mit” sybcca
It will display all the lines that contain pattern either amit or Amit.
Expressions:
Ch+
Matches one or more occurrences of character ch.
E.g., egrep “Ap+ly” f1
It will display those lines that have Apply word, but in Apply word p will be one or more than one times.
Output: Apply, Appply, Aply etc.
Ch?
Matches zero or one occurrences of character ch.
E.g., egrep “Ap?ly” f1
It will display only those lines that has Apply word, but in Apply word p can be one time or zero times.
Output: Aly, Aply.
exp1|exp2
Matches for either exp1 or exp2.
E.g. egrep “Amit|Abhay” f1
It will display all the lines that has pattern either Amit or Abhay.
(x1|x2)x3
Matches for either x1 x3 or x2 x3.
E.g. egrep “(Amit|Abhay) Chaudhary”.
It will display all the lines that has pattern either Amit Chuadhary or Abhay Chaudhary.
4. $look
The look command is a specialized program that finds and prints all of the lines in a file or set of files that begin with a specified string. You can do the same thing with egrep or grep, or with sed or awk, by using the ^ in a regular expression to search for a string at the beginning of a line, but look makes this particular task easier.
Use: Look command is used to search the pattern at the prefix position from the file. Data must be in sorted order.
Syntax: look <“pattern”> <filename>
Example:
look “Gandhinagar” Gujarat
Output:
Gandhinagar is well-known for Green City.
Gandhinagar is the capital city of Gujarat State.
SORTING THE FILES
Many tasks require the use of sorted data, and sorting the information in files and operating on stored files are some of the most common computer tasks. You sort information in a file in order to make it easier to read, to make it easier to compare two lists, or to prepare the information for processing by a command that requires sorted input. You may need to sort information alphabetically or numerically, by lines or according to a particular field or column.
$Sort
Usage: The sort command orders or reorders the lines of a file.
Syntax: sort <filename>
Example:
1. sort f1
It will sort the contents of file f1 in ascending order.
2. Sort f1 f2 f3
It will sort the contents of file f1, f2 and f3 at same time.
Options:
Reverse order
With numerical data such as frequencies or money, you often want the output to show the largest values first. The –r (reverse) option tells sort to reverse the order of its output.
E.g., sort –r f1
Before After
Amit Vijaya
Vijaya Shilpa
Shilpa Amit
Numerical Sorting
The main use of this –n option is used to sort the numeric data which can’t be done by simple sort command.
E.g., sort –n f1
Before After
22 3
4 4
54 22
3 54
66 66
Suppressing Repeated Lines
Sorting often reveals that a file contains multiple copies of the same line. The next section describes the uniq command, which is designed to remove repeated lines from input files. Because this is such a common sorting task, sort also provides an option, -u (unique), that removes repeated lines from its output. Repeated lines are likely to occur when you combine and sort data from several different files into a single file.
E.g. sort –u file1
Before After
Amit Amit
Amit Vijaya
Vijaya Shilpa
Vijaya
Shilpa
Sorting by Field or Column
By default, sort compares and sorts entire lines, beginning with the first character. Often, though, you want to sort on the basis of a particular field or column. The sort command provides a way for you to specify the part of each line to use for its comparisons. You do this by telling sort to skip one or more fields or columns.
E.g. sort –k 2 f1
Before After
Amit 004 Amit 004
Vijaya 084 Shilpa 067
Shilpa 067 Vijaya 084
Replacing the Original File
Very often when you sort a file, you want to replace the original with the sorted version as shown in the following example:
$ sort telnos > telnos.sort
$ mv telnos.sort telnos
The first sorts telnos and puts the sorted data into the new file telnos.sort. Then telnos.sort is renamed telnos.
Replacing a file with a sorted data version is such a common task that UNIX provides an option, -o (output) that you can use to tell sort to replace the input file with the sorted version. The following command sorts telnos and replaces its contents with the sorted output:
$ sort –o telnos telnos
uniq
The uniq commands filters or removes repeated lines from files. It is usually with files that have first been sorted by sort. In its simplest form it has the same effect as the –u option to sort, and it is an alternative to using (or remembering) that option. Uniq also provides several useful options, including one to count the number of times each line is repeated and others to display either repeated lines or unique lines.
Syntax: uniq <filename>
E.g. uniq f1
Before After
45 45
45 32
32 64
64
64
Counting Repetitions
One of the most valuable uses of uniq is in counting the number of occurrences of each line. This is a very convenient way to collect frequency data.
E.g. uniq –c f1
Before After
45 2 45
45 1 32
32 2 64
64
64
The limitation of this command is that if the numbers are not contiguous than it will count every line as unique line. For e.g.,
Before After
45 1 45
32 1 32
45 1 45
64 1 64
Though there is a duplicate data in a file (means 45) than also it is showing “1” instead of “2”.
COMPARING FILES
Often you need to see whether two files have different contents and to list the differences if there are any. For example, you may have several versions of a letter or several drafts of a memo. It is easy to lose track of whether or how versions differ. It is also sometimes useful to be able to tell whether files having the same name in two different directories are simply different copies of the same file, or whether the files themselves are different.
Consider there are two files:
F1 F2
Good Bad
Hello Hi
Yellow Yellow
$cmp
Use:
The cmp command is the simplest of the file comparison tools. It tells you whether two files differ, and if they do, it reports the position in the file where the first difference occurs. It compares two files byte wise.
Syntax:
cmp <filename1> <filename2>
Example: cmp f1 f2
Output: f1 f2 differ: char 1, line 1
$comm
Use:
The comm (common) command is designed to compare two sorted files and show the lines that are the same or different. You can display lines that are found only in the first file, lines that are found only in the second file, and lines found in both, respectively. By default, comm prints its output in three columns: lines unique to the first file, those unique to the second file, and lines found in the both, respectively.
Syntax: comm <filename1> <filename2>
Example:
comm f1 f2
Output:
Good Bad Yellow
Hello Hi
$diff
Use:
The diff command compares two files, line by line, and prints out differences. In addition, for each block of text that differs between the two files, diff tells you how the text from the first file would have to be changed to match the text from the second.
Syntax: diff f1 f2
Output:
1,2c1,2
< good
< hello
……………..
> bad
> hi
Lines containing text that is found only in the first file begin with “<” and lines containing text found only in the second file begin with >. Dashed lines separate parts of the diff output that refer to different sections of the file. Here, in example, it compares both files f1 and f2. It also tells that 1,2 lines from first file and 1,2 lines from second files are different.
$patch
Use:
If you have the description of differences between an original file and another produced by the diff command, you can use patch to update the original to include the differences. The updated (patched) version replaces the original, and the original file is saved with the same name and an extension of “.orig”. patch tries to do an intelligent job of applying the diff description. It will skip over irrelevant (from the viewpoint of differences and updates) material at the beginning and end of the file.
Syntax: diff <file1> <file2> newfile
Patch <file1> newfile
Example:
diff f1 f2 > new
patch f1 new
Output:
Bad
Hi
Yellow
WORKING WITH COLUMNS AND FIELDS
Many files contain information that is organized in terms of position within a line. These include tables, which organize information in columns, and files consisting of lines or records that are made up of fields. The UNIX system includes a number of tools designed specifically to work with files organized in columns or fields. You can use the commands described in this section to extract and modify or rearrange information field-structured or column-structured files.
Consider an employee file:
Empno Empname Salary Age
1 Amit 20000 21
2 Shilpa 21500 20
3 Vijaya 20000 20
$cut
Often you are interested in only some of the fields or columns contained in a table or file. For example, you may want to get only a name from a personnel file that contains name, employee number, address, telephone number, and so forth. cut allows you to extract from such files only the fields or columns you want.
When you use cut, you have to tell it how to identify fields (by character position or by the use of field separator characters) and which fields to select. You must specify either the –c or the –f option and the field or fields to select.
Use:
It allows you to select particular column or fields from the files.
Syntax:
cut <option> <filename>
Two options are using with cut command:
-c for cutting column
-f for cutting fields
Example:
1. cut –f2 emp
It will cut second field from file emp that is empname.
2. cut –f2,4 emp
It will cut second and fourth field from file emp that is empname and age.
3. cut –c2-4 emp
It will cut second to fourth characters from file emp.
$colrm
The colrm command is a specialized command that you can use to remove one or more columns from a file or set of files. Although you can use the cut command to do this, as previously described, colrm is a simple alternative when that is exactly what you need to do. Column here refers to the character position in a line. You specify the first and last columns to remove.
Syntax: colrm <character> <filename>
Example:
colrm 5 8 file1
Output:
Before After
Hello India Helldia
$paste
The paste command joins files together line by line. You can use it to create new tables by joining together fields or columns from two or more files.
Syntax: paste <file1> <file2>
Example:
paste f1 f2
Suppose f1 has:
Hello
See you soon
and f2 has:
Amit
Abhay
Output:
Hello Amit
See you soon Abhay
$join
The join command creates a new file by joining together two existing files on the basis of a key field that contains entries common to both of them. It is similar to paste, but join matches lines according to the key field, rather than simply joining them together. The key field appears only once in the output.
Syntax: join <file1> <file2>
Example:
join f1 f2
Suppose file f1 has:
No Name
1 Amit
2 Shilpa
3 Vijaya
and file f2 has:
No Dept
1 Sales
2 Production
3 Accounting
4 Purchase
Output:
No Name Dept
1 Amit Sales
2 Shilpa Production
3 Vijaya Accounting
2. join -1 2 -2 3 f1 f2
In this command, key field is second field from f1 and third field from f2.
$tac
The tac command is a backward version of cat. It concatenates one or more files (or standard input) and prints them out by line by line to standard output, in reverse line order; the last line is printed first, the first line, last. Compared with sed or tr, tac carries to an extreme the idea of a tool that is designed to do just one thing. Although it is not something you will often use, and although you can do the same thing with a small awk program, when you need it, it’s nice to have it.
Syntax: tac <file1>
Example: tac f1
It will display the contents of file f1 in reverse order.
$sed
The sed command is a powerful tool for filtering text files. It can perform all most all of the editing functions of the interactive editors ed and vi. You can use it to make global changes involving individual characters, words, or patterns, and to make context-specific changes including deleting or adding text. It differs from ed, vi and similar editors in that it is designed to read its commands in a batch mode – from the command line or from a script rather than interactively from a user at a keyboard. The command name sed stands for stream editor, which means that it is designed to modify or edit input presented to it as a stream of characters.
The sed command is often used in shell scripts, or as part of a pipeline to filter the output of another command. sed is particularly useful when you need to modify a very large file. Interactive editors such as ed and vi have an upper limit on the size of the file they will accept. Since it is basically processes one line at a time, sed can be used with files of any size.
The sed command is an example of UNIX text processing tools that continue to be useful even in an era of word processors and interactive, window-oriented applications. The ability to create a simple script that can carry out a complex editing operation on any input is an example of the continuing, unique value of the UNIX System tools.
Line addressing:
Syntax: sed <option> <filename>
Examples:
1. sed ‘3q’ f1
It will display first 3 lines from file f1.
2. sed –n ‘1,4p’ f1
It will display lines from 1 to 4 from file f1.
3. sed –n ‘$p’ f1
It will display last line from file f1.
4. sed -n ‘5,$!p’ f1
It will not display lines from 5th to last line.
5. sed –ne ‘1,3p’ –e ‘5,8’ f1
It will display lines from 1 to 3 and 5 to 8.
Context Addressing:
Examples:
1. sed –n ‘/Amit/p’ f1
It will display all the lines that contain “Amit” from f1.
2. Sed –n ‘/[Aa]mit/p’ f1
It will display all the lines that contains either Amit or amit word.
Writing Selected lines to a files
sed –n ‘/Abhay/w’ f1
Inserting or Deleting Lines
Examples:
1. sed ‘2i/Hello world/’ f1
It will insert Hello world in the file f1.
2. sed ‘/sales/d’ f1
It will delete lines that containing “sales” word, from file f1.
$tr
The best way to explain what tr does is with a simple example. Suppose you have a file that use a colon (:) to separate fields, and you need to change every colon to another character, say a tab. You might need to do this to use the file with a program that expects a tab as a field separator. Or you might feel that colon-seperated fields are difficult to read, and prefer to use tabs to make the file easier to view.
Use: The tr command is used to replace set of characters with another set.
Example:
1. tr : ‘\t’ < f1
Before After
Amit:Shilpa:Vijaya Amit Shilpa Vijaya
It will replace all (:) with tab space.
2. tr ‘[a-z]’ ‘[A-Z]’ < f1
Before After
Amit AMIT
Shilpa SHILPA
Vijaya VIJAYA
3. tr ‘[0-9]’ ‘[#]’ < f1
Before After
1 #
2 #
3 #
4 #
Note that the tab character in the arguments is enclosed in single quotation marks to prevent the shell from treating it as white space. Also note that the examples uses the input redirection symbol (<) to specify the input file. (tr is one of the few common UNIX system tools that does not allow you to specify a filename as an argument. tr only reads standard input, so you have to use input redirection or a pipe to give it its input).
The tr command can translate any number of characters. In general, you give tr two lists of characters, an input list (characters in the input to be translated), and an output list (characters to which they are translated in the output). tr then replaces every instance of a character in the input set with the corresponding character in the output set. That is, it translates the first character in the input list to the first character in the output list, the second to the second, and so on.
Command | Function | Meaning |
A | Append | Append text after address |
I | Insert | Insert text before address |
C | Change | Replace lines with text |
D | Delete | Delete specified lines |
S | Substitute | Replace pattern |
P | Print | Print specified lines |
Q | Quit | Terminate at line 20 |
R | Read | Read in file before line |
W | Write | Write specified lines to file |
Comments
Post a Comment