Slide 1 : Advance UNIX Training
Slide 2 : UNIX shells and variables Understanding UNIX shells Unix shells come with variables that are used by the shell or related commands. In addition to variables that you create, the shell itself requires or takes advantage of variables that can be set up for it. Displaying the default variables The C shell also takes a similar approach, but uses more files. It runs /etc/csh.cshrc, then /etc/csh.login, then an entire raft of files in your home directory, such as ~/.cshrc, ~/.history, ~/.login, and, finally, ~/.cshdirs. Regardless of the approach, the result is an environment in which the user will run, including environment variables. You can see your environment variables by using printenv or env. The following is a short example of the output.
Slide 3 : Creating user variables For example, PS1, listed above as an environment variable, is the prompt displayed on the screen when the shell is waiting for a new command. Another shell variable, PS2, contains the prompt to be used when a command is begun but not completed before Enter is pressed. $ echo $PS2$ cat /etc/passwd|grep mjb$ echo $PS2
$ PS2="more please> "$ cat /etc/passwd|grep mjb# echo $PS2$ printenvUSERNAME=HISTSIZE=1000HOSTNAME=my.system.comLOGNAME=mjb
MAIL=/var/spool/mail/mjbTERM=xtermPATH=/usr/bin:/bin:/usr/local/bin:/usr/bin/X11:/home/mjb/binHOME=/home/mjbPS1=[\u@\h \W]\$
Slide 4 : The shell sets up some default shell variables; PS2 is one of them. Other useful shell variables that are set or used in the Korn shell are: * _ (underscore) -- When an external command is executed by the shell, this is set in the environment of the new process to the path of the executed command. In interactive use, this parameter is also set in the parent shell to the last word of the previous command. * COLUMNS -- The number of columns on the terminal or window. * ENV -- If this parameter is found to be set after any profile files are executed, the expanded value is used as a shell startup file. It typically contains function and alias definitions. * ERRNO -- Integer value of the shell's errno variable -- this indicates the reason the last system call failed. * HISTFILE -- The name of the file used to store history. When assigned, history is loaded from the specified file. Multiple invocations of a shell running on the same machine will share history if their HISTFILE parameters all point to the same file. If HISTFILE isn't set, the default history file is $HOME/.sh_history. * HISTSIZE -- The number of commands normally stored in the history file. Default value is 128.
Slide 5 : * LINENO -- The line number of the function or shell script that is being executed. This variable is useful for debugging shell scripts. Just add an echo $LINENO at various points and you should be able to determine your location within a script. * LINES -- Set to the number of lines on the terminal or window. * PPID -- The process ID of the shell's parent. A read-only variable. * PATH -- A colon-separated list of directories that are searched when seeking commands. * PS1 -- The primary prompt for interactive shells. * PS2 -- Secondary prompt string; default value is >. Used when more input is needed to complete a command. * PWD -- The current working directory. This may be unset or null if shell does not know where it is. * RANDOM -- A simple random number generator. Every time RANDOM is referenced, it is assigned the next number in a random number series. The point in the series can be set by assigning a number to RANDOM.
Slide 6 : * SECONDS -- The number of seconds since the shell started or, if the parameter has been assigned an integer value, the number of seconds since the assignment plus the value that was assigned. * TMOUT -- If set to a positive integer in an interactive shell, it specifies the maximum number of seconds the shell will wait for input after printing the primary prompt (PS1). If this time is exceeded, the shell exits. * TMPDIR -- Where the directory shell temporary files are created. If this parameter is not set, or does not contain the absolute path of a directory, temporary files are created in /tmp.
Slide 7 : Chapter 2 - The UNIX Shell user environment 1. Viewing and modifying your .login file 2. Viewing and modifying your .cshrc file 3. Creating and modifying your .xeric file 4. Creating and modifying your .logout file 5. Examining the order of execution of the environmental filesChapter 3 - Aliases in the Shell 1. Introducing the alias concept 2. Creating custom aliases at the prompt 3. Making your aliases permanent Chapter 4 - Advanced file management 1. Using the find command to locate files 2. Using advanced grep to search file contents 3. Introducing the egrep commandIn place of 3 and 4 unit :Text Processing Tools
Slide 8 : Upon completion of this unit, you should be able to: * Use tools for extracting, analyzing and manipulating text dataTools for Extracting Text * File Contents: less and cat * File Excerpts: head and tail * Extract by Column: cut * Extract by Keyword: grep Viewing File Contents : less and cat * cat: dump one or more files to STDOUT
o Multiple files are concatenated together
*less: view file or STDIN one page at a time
o Useful commands while viewing: + /text searches for text + n/N jumps to the next/previous match + v opens the file in a text editor o less is the pager used by man
Slide 9 : Viewing File Excerpts : head and tail * head: Display the first 10 lines of a file o Use -n to change number of lines displayed * tail: Display the last 10 lines of a file o Use -n to change number of lines displayed o Use -f to "follow" subsequent additions to the file + Very useful for monitoring log files! Extracting Text by Keyword : grep * Prints lines of files or STDIN where a pattern is matched $ grep 'john' /etc/passwd $ date --help | grep year * Use -i to search case-insensitively * Use -n to print line numbers of matches * Use -v to print lines not containing pattern * Use -AX to include the X lines after each match * Use -BX to include the X lines before each match
Slide 10 : Extracting Text by Column : cut * Display specific columns of file or STDIN data $ cut -d: -f1 /etc/passwd $ grep root /etc/passwd | cut -d: -f7 * Use -d to specify the column delimiter (default is TAB) * Use -f to specify the column to print * Use -c to cut by characters $ cut -c2-5 /usr/share/dict/wordsTools for Analyzing Text * Text Stats: wc * Sorting Text: sort * Comparing Files: diff and patch * Spell Check: aspell
Slide 11 : Gathering Text Statistics : wc (word count) * Counts words, lines, bytes and characters * Can act upon a file or STDIN * Use -l for only line count * Use -w for only word count * Use -c for only byte count * Use -m for character count (not displayed) Sorting Text : sort * Sorts text to STDOUT - original file unchanged $ sort [options] file(s) * Common options o -r performs a reverse (descending) sort o -n performs a numeric sort o -f ignores (folds) case of characters in strings o -u (unique) removes duplicate lines in output o -t c uses c as a field separator o -k X sorts by c-delimited field X + Can be used multiple times
Slide 12 : Eliminating Duplicate Lines : sort and uniq * sort -u: removes duplicate lines from input * uniq: removes duplicate adjacent lines from input o Use -c to count number of occurrences o Use with sort for best effect: $ sort userlist.txt | uniq -cComparing Files : diff *Compares two files for differences $ diff foo.conf-broken foo.conf-works 5c5 < use_widgets = no --- > use_widgets = yes o Denotes a difference (change) on line 5 * Use gvimdiff for graphical diff
o Provided by vim-X11 package
Slide 13 : Duplicating File Changes : patch * diff output stored in a file is called a "patchfile" o Use -u for "unified" diff, best in patchfiles * patch duplicates changes in other files (use with care!) o Use -b to automatically back up changed files $ diff -u foo.conf-broken foo.conf-works > foo.patch $ patch -b foo.conf-broken foo.patchSpell Checking with aspell * Interactively spell-check files: $ aspell check letter.txt * Non-interactively list mis-spelled words in STDIN $ aspell list < letter.txt $ aspell list < letter.txt | wc -l Tools for Manipulating Text : tr and sed
* Alter (translate) Characters: tr
Slide 14 : o Converts characters in one set to corresponding characters in another set o Only reads data from STDIN $ tr 'a-z' 'A-Z' < lowercase.txt * Alter Strings: sed o stream editor o Performs search/replace operations on a stream of text o Normally does not alter source file o Use -i.bak to back-up and alter source filesed : Examples * Quote search and replace instructions! * sed addresses o sed 's/dog/cat/g' pets o sed '1,50s/dog/cat/g' pets o sed '/digby/,/duncan/s/dog/cat/g' pets * Multiple sed instructions o sed -e 's/dog/cat/' -e 's/hi/lo/' pets o sed -f myedits pets
Slide 15 : Special Characters for Complex Searches : Regular Expressions * ^ represents beginning of line * $ represents end of line * Character classes as in bash: o [abc], [^abc] o [[:upper:]], [^[:upper:]] * Used by: o grep, sed, less, others Finding and Processing Files
Slide 16 : Upon completion of this unit, you should be able to: * Use locate * Use findlocate * Queries a pre-built database of paths to files on the system o Database must be updated by administrator o Full path is searched, not just filename * May only search directories where the user has read and execute permissionlocate Examples * locate foo
o Search for files with "foo" in the name or path * locate -r '\.foo$‘ o Regex search for files ending in ".foo" * Useful options o -i performs a case-insensitive search o -n X lists only the first X matches
Slide 17 : find * find [directory...] [criteria...] * Searches directory trees in real-time o Slower but more accurate than locate o CWD is used if no starting directory given o All files are matched if no criteria given * Can execute commands on found files * May only search directories where the user has read and execute permissionfind and Permissions * Can match ownership by name or id o find / -user joe -o -uid 500
* Can match octal or symbolic permissions o find -perm 755 matches if mode is exactly 755 o find -perm +222 matches if anyone can write o find -perm -222 matches if everyone can write o find -perm -002 matches if other can write
Slide 18 : * Many find criteria take numeric values * find -size 1024k o Files with a size of exactly 1 megabyte *find -size +1024k o Files with a size over 1 megabyte *find -size -1024k o Files with a size less than 1 megabytefind and Access Times * find can match by inode timestamps o -atime when file was last read o -mtime when file data last changed o -ctime when file data or metadata last changed * Value given is in days o find -ctime -10 + Files modified less than 10 days ago
Slide 19 : Executing Commands with find * Commands can be executed on found files o Command must be preceded with -exec or -ok + -ok prompts before acting on each file o Command must end with Space\; o Can use {} as a filename placeholder o find -size +102400k -ok gzip {} \; find Execution Examples * find -name "*.conf" -exec cp {} {}.orig \; o Back up configuration files, adding a .orig extension
* find /tmp -ctime +3 -user joe -ok rm {} \; o Prompt to remove Joe's tmp files that are over 3 days old
* find ~ -perm +o+w -exec chmod o-w {} \; o Fix other-writable files in your home directory
Slide 20 : Chapter 5 - Command line database processing 1. Using awk to display file contents 2. Formatting files with the tr command 3. Using sed to edit file contents 4. Scripting your database reporting