Shell expansion is nothing new for us, we have been using it all along, we just did not know it by it’s proper name.
In Lab No.4 we worked with globbing characters, mostly to find files using very basic patterns.
When we used patterns that included the *, ? and [] characters, the shell applied one type of expansion called Pathname expansion.
When we used the curly braces {} to create patterns, the shell applied a different type of expansion that created lists (appropriately known as Brace Expansion). We have also seen how the Tilde Expansion (~) works when we needed to resolve a user’s home directory.
When the shell process commands, it applies a process of tokenization and expansion to the arguments that are passed. In this lab we are going to apply the most commonly used types of expansions.
For this lab you are going to need some files that are available in an archive located in ~jmora/lab07/lab07files.tar.gz. Extract those files to a folder called lab07 under your home directory:
[you@blue ~]$ mkdir lab07 [you@blue ~]$ cd lab07 [you@blue lab07]$ tar -xzvf ~jmora/lab07/lab07files.tar.gz
What we covered in Lab 04 was mostly pathname expansion, so we are not going to cover that again on this lab.
We have been using tilde expansion on a daily basis on previous labs. However, it is appropriate to introduce formally in this lab the basic rules that govern tilde expansion:
/), it expands into the pathname of the home directory of the user that matches the word.HOME environment variable (or the current user’s home directory as defined in the /etc/passwd file.). We will explain what environment variables are in just a few sections later in this lab.~+ expands to the current working directory~- expands to the previous working directorySince we already covered this on previous labs, there are no exercises about Tilde expansion on this lab.
Brace expansion is a mechanism to generate arbitrary strings. Patterns to be brace expanded take the form PREAMBLE*{expression}*POSTSCRIPT. The preamble is prefixed to each string generated by the expression within the braces, and the postcript is appended to each resulting string. Brace expansion can also be nested.
For an expression to be consider a brace expansion, it needs to be a list of string tokens.
You can use a comma to define an arbitrary list of string elements.
You can also define sequences by using an expression of the form {x..y} where x and y are either single characters or integers.
You can skip elements from the sequence by using the form {x..y..z} where z is the number of elements to skip.
Brace expansion is performed before any other expansions, which means that we can include other expressions that can be expanded as well.
Brace Expansion
Complete the exercises on Moodle.
This type of expansion allows the output of a command to replace the command itself. Command substitution has two forms:
$(command)`command`This form is very useful and common in scripts to assign values to variables.
Command Substitution
Complete the exercises on Moodle.
It turns out that you can use the shell to perform arithmetic operations for you. Arithmetic expansion uses the form $((expression)). Unfortunately, arithmetic expansion only supports integers.
Arithmetic Expansion
Complete the exercises on Moodle.
Any script that goes beyond the most basic shell scripts use this type of expansion.
It is a mechanism to asign a variable name to a value that we need to make a reference later.
The basic form of parameter expansion is the form ${PARAMETER}. Braces can be ommitted and use the form $PARAMETER except when it needs to be concatenated with other strings and if PARAMETER refers to a positional parameter in a script that requires more than one digit.
Remember from the previous lab how we made references to arguments using the $ in front of a number that represented the position of the argument in the provided command.
When the number of positional parameters exceeds 9, then instead of using $10 to refer to tenth argument, you need to use the form S{10}.
We are at a point were we can introduce formally the concept of environment variables. When you start a session, the shell creates a set of variables that describe the session, in what is called the environment.
The data stored in the environment is used by many programs to determine how they need to function. The environment provides a very consistent and centralized way to access some of the most elemental configuration parameters.
Most programming languages provide an API that allows you to get access to the variables in the environment. Examples of the most common environment variables are the username, the home directory, the language, the current working directory, and the path where executables can be found.
Many programs require you to define environment variables so they can work. If you have done any python programming you will probably had to set the PYTHONPATH variable, if you are a java programmer you must have seen the JAVA_HOME and the CLASSPATH environment variables, if you a C++ programmer, you must have seen the CPATH environment variables.
In Linux the env command prints a list of the environment variables. You can get the value of any specific variable using variable expansion, for example instead of using the command cd ~ to change to your home directoty, you could also the (longer but equivalent) command cd $HOME.
To set an environment variable so it can be accessible to other commands (the correct term is other child processes, but we have not talked about processes yet) during your active session you use the export command.
Arithmetic Expansion
Complete the exercises on Moodle.
There are cases when we need to control how the input of the commands is evaluated by the shell. The most common example of how input is treated by the shell occurs when we need to pass a sentence that contains spaces. Another example of this behavior occurs then we want to prevent the shell from expanding an expression that has a dollar sign in it (e.g. a currency value). To deal with these situations, the shell allows us to enclose input using quotes.
When enclosing characters in double quotes, the shell preserves their literal value, with the exception of $, ` and \.
This means that quotes won’t allow Tilde, Pathname and Brace Expansion within.
Variable and Arithmetic expansions are still allowed within double quotes.
When enclosing characters in single quotes, the shell preserves the literal value of each charactre within the quotes, which means that no expansions are executed at all. One caveat with single enclosing characters is that a single quote may not occur between single quoates, even when preceded by a backslash.
Quoting
Complete the exercises on Moodle.
In this lab we are going to learn some very powerful features of the bash shell. In order to put them in practice we are going to learn also a new utility that let us do some very useful tasks.
Awk is a utility and a programming language at the same time.
Its most common usage is parsing of files based on delimiters.
We are going to cover awk with greater detail in a future lab.
Proabably the most basic example of awk in action is the following:
[you@blue ~]$ echo "This is some data" | awk '{print $3}'
some
You can see that awk takes the string This is some data and tokenizes it based on the spaces, and then it uses the '{print $3}' bit to figure out that you wanted to print the third field.
You can override the field delimiter by using the -F option. For example, the /etc/passwd file is delimited by colons (:). You can use awk to get a list of user ids by running this command:
[jmora@blue lab07]$ cat /etc/passwd | awk -F: '{print $1}'
Note that in this context the expression $1 is not subject to shell expansion because it is enclosed in single quotes.