Skip main navigation

Linux: How to Track Script Progress and Redirect Outputs

In this article you will learn about some ways to track the progress of your script and to redirect script output to files
© Wellcome Genome Campus Advanced Courses and Scientific Conferences
In this article you will learn about some ways to track the progress of your script and to redirect script output to files

Tracking the progress of your script

Now, let us imagine you have a long and complex Bash script. You execute your script, it’s started running and you’ve gone off to make a cup of tea. Ten minutes later, you come back to check on its progress but, how do you know what’s going on and where you’ve gotten up to in your script?
There are many different ways in which we can track the progress of our scripts. The simplest is to break your script down into sections and output a progress statement when you start and/or finish each section.
For example, let’s set our name as a variable and count the number of characters it contains.
#!/usr/bin/env bash
 
# Set your name as a variable
name="Victoria"
 
echo "Counting number of characters in name"
printf -- "${name}" | wc -m
 
As expected, we have our progress statement and the number of characters in our name:
 
Counting number of characters in name
 8
 
Now, while this may seem excessive given the simple example, it’s clear that once we start to build up our scripts, adding progress statements will be invaluable. Particularly when discussing loops, where it’s possible for your scripts to get stuck in an infinite loop, failing to exit. In those situations, progress statements are absolutely essential for debugging!
 
Can you see a situation where you would need to track the progress of your script?
 

Redirecting script outputs and errors

 
Despite your hardest efforts, sometimes your Bash scripts will do unexpected things. This is when we need to debug. If you have a long Bash script, it can be tricky to work out where things went wrong.
 
To help with debugging, we can output progress statements at key points in our code e.g. “Reading in file: x”. However, these can easily fill up your terminal and become difficult to follow. A simple solution is to write these progress statements to one or more log files.
 

Redirecting the output of scripts and commands to files

 
Simply put, redirection is the mechanism by which we can send the output of a command or script to another place. When we want to capture the output from a command or script, we usually choose to redirect those outputs into a file.
 
To redirect the outputs of a script, we can use the > symbol:
 
script.sh > output.txt
 
Redirection using the > symbol works not only for scripts, but any Bash command:
 
echo "hello world" > hello.txt
cat hello.txt
hello world
 

Linux streams and file descriptors

 
Before we take an in depth look at how we redirect our outputs and errors to log files, we first need a crash course in Linux streams and file descriptors. These streams are handled like files – i.e. you can read from them and you can write to them.
 
There are three streams you should be aware of:
 
 
    • stdin (standard input)
 
    • stdout (standard output)
 
    • stderr (standard error)
 
 
This sounds much more complicated than it really is. In a nutshell, stdout refers to the output from a command and stderr refers to the errors a command generates. The final stream, stdin refers to command line inputs.
 
Next, we need to understand file descriptors. A file descriptor is just a (positive) integer that represents an open file. Each of our Linux streams (i.e. stdin, stdout and stderr) has been allocated a unique number in order to identify them.
 
All you need to remember is which of the ids below corresponds to each of the streams:
 
0 => stdin
 
1 => stdout
 
2 => stderr
 

I/O redirection

 
To start understanding how these streams work, let’s look at redirecting the output from a script into a single file.
 
Example script:
 
#!/usr/bin/env bash
 
# A script that tries to change directory
 
echo "Changing to a directory that doesn't exist"
cd foo
 
As you can see, our script returns the printed progress statement and an error that tells us that the directory we’re trying to migrate to doesn’t exist on our filesystem.
 
./script.sh
Changing to a directory that doesn't exist
script.sh: line 6: cd: foo: No such file or directory
 
These two messages are being delivered to the terminal by two different Linux streams. The first message, our progress statement, is delivered via stdout. Meanwhile, the error message is delivered via stderr.
 
Now, let’s see what happens when we try to redirect the outputs from that script into a file called output.txt:
 
./script.sh > output.txt
./script.sh: line 6: cd: foo: No such file or directory
 
OK, so, we can see that the stdout has been redirected to our output file but, the error is still being displayed.
 
cat output.txt
Changing to a directory that doesn't exist
 
Why is this? Well, when we use > to redirect to a file, by default, the system will only redirect the stdout.
 
But, what about our errors being delivered via stderr, how can we capture those?
 
To simplify things, let’s first look at how to redirect stdout and stderr to two different files. We’ll use the > symbol with our file descriptors (1 for stdout and 2 for stderr) to redirect our outputs to output.txt and our errors to error.txt respectively.
 
./script.sh 1>output.txt 2>error.txt
 
This command returns nothing back to our terminal. Using the cat command, we can see that, as expected, our outputs and errors have been written to output.txt and error.txt respectively.
 
Our stdout (progress statement returned using echo):
cat output.txt
Changing to a directory that doesn’t exist
 
And our stderr (errors):
 
cat error.txt
./script.sh: line 6: cd: foo: No such file or directory
 
In order to redirect the stdout and the stderr to the same place, we need to use a new term: 2>&1. When we use this, we redirect using the same syntax as before, but add 2>&1 to the end of our command.
 
This is how it works in practice:
 
./script.sh > combined_output.txt 2>&1
 
Now, if we look at our combined output file, we can see that we’ve captured both the stdout and the stderr.
 
cat combined_output.txt
Changing to a directory that doesn't exist
./script.sh: line 6: cd: foo: No such file or directory
© Wellcome Genome Campus Advanced Courses and Scientific Conferences
This article is from the free online

Bioinformatics for Biologists: An Introduction to Linux, Bash Scripting, and R

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education

close