Skip main navigation

New lower prices! Get up to 50% off 1000s of courses. 

Explore courses

Linux: How to Track Script Progress and Redirect Outputs

In this article you will learn about some ways to track the progress of your script and to redirect script output to files

In this article you will learn about some ways to track the progress of your script and to redirect script output to files

Tracking the progress of your script

Now, let us imagine you have a long and complex Bash script. You execute your script, it’s started running and you’ve gone off to make a cup of tea. Ten minutes later, you come back to check on its progress but, how do you know what’s going on and where you’ve gotten up to in your script?

There are many different ways in which we can track the progress of our scripts. The simplest is to break your script down into sections and output a progress statement when you start and/or finish each section.

For example, let’s set our name as a variable and count the number of characters it contains.

#!/usr/bin/env bash
 
# Set your name as a variable
name="Victoria"
 
echo "Counting number of characters in name"
printf -- "${name}" | wc -m

 

As expected, we have our progress statement and the number of characters in our name:

 

Counting number of characters in name
 8

 

Now, while this may seem excessive given the simple example, it’s clear that once we start to build up our scripts, adding progress statements will be invaluable. Particularly when discussing loops, where it’s possible for your scripts to get stuck in an infinite loop, failing to exit. In those situations, progress statements are absolutely essential for debugging!

 

Can you see a situation where you would need to track the progress of your script?

 

Redirecting script outputs and errors

 

Despite your hardest efforts, sometimes your Bash scripts will do unexpected things. This is when we need to debug. If you have a long Bash script, it can be tricky to work out where things went wrong.

 

To help with debugging, we can output progress statements at key points in our code e.g. “Reading in file: x”. However, these can easily fill up your terminal and become difficult to follow. A simple solution is to write these progress statements to one or more log files.

 

Redirecting the output of scripts and commands to files

 

Simply put, redirection is the mechanism by which we can send the output of a command or script to another place. When we want to capture the output from a command or script, we usually choose to redirect those outputs into a file.

 

To redirect the outputs of a script, we can use the > symbol:

 

script.sh > output.txt

 

Redirection using the > symbol works not only for scripts, but any Bash command:

 

echo "hello world" > hello.txt
cat hello.txt
hello world

 

Linux streams and file descriptors

 

Before we take an in depth look at how we redirect our outputs and errors to log files, we first need a crash course in Linux streams and file descriptors. These streams are handled like files – i.e. you can read from them and you can write to them.

 

There are three streams you should be aware of:

 

 

    • stdin (standard input)

 

    • stdout (standard output)

 

    • stderr (standard error)

 

 

This sounds much more complicated than it really is. In a nutshell, stdout refers to the output from a command and stderr refers to the errors a command generates. The final stream, stdin refers to command line inputs.

 

Next, we need to understand file descriptors. A file descriptor is just a (positive) integer that represents an open file. Each of our Linux streams (i.e. stdin, stdout and stderr) has been allocated a unique number in order to identify them.

 

All you need to remember is which of the ids below corresponds to each of the streams:

 

0 => stdin

 

1 => stdout

 

2 => stderr

 

I/O redirection

 

To start understanding how these streams work, let’s look at redirecting the output from a script into a single file.

 

Example script:

 

#!/usr/bin/env bash
 
# A script that tries to change directory
 
echo "Changing to a directory that doesn't exist"
cd foo

 

As you can see, our script returns the printed progress statement and an error that tells us that the directory we’re trying to migrate to doesn’t exist on our filesystem.

 

./script.sh
Changing to a directory that doesn't exist
script.sh: line 6: cd: foo: No such file or directory

 

These two messages are being delivered to the terminal by two different Linux streams. The first message, our progress statement, is delivered via stdout. Meanwhile, the error message is delivered via stderr.

 

Now, let’s see what happens when we try to redirect the outputs from that script into a file called output.txt:

 

./script.sh > output.txt
./script.sh: line 6: cd: foo: No such file or directory

 

OK, so, we can see that the stdout has been redirected to our output file but, the error is still being displayed.

 

cat output.txt
Changing to a directory that doesn't exist

 

Why is this? Well, when we use > to redirect to a file, by default, the system will only redirect the stdout.

 

But, what about our errors being delivered via stderr, how can we capture those?

 

To simplify things, let’s first look at how to redirect stdout and stderr to two different files. We’ll use the > symbol with our file descriptors (1 for stdout and 2 for stderr) to redirect our outputs to output.txt and our errors to error.txt respectively.

 

./script.sh 1>output.txt 2>error.txt

 

This command returns nothing back to our terminal. Using the cat command, we can see that, as expected, our outputs and errors have been written to output.txt and error.txt respectively.

 

Our stdout (progress statement returned using echo):
cat output.txt
Changing to a directory that doesn’t exist

 

And our stderr (errors):

 

cat error.txt
./script.sh: line 6: cd: foo: No such file or directory

 

In order to redirect the stdout and the stderr to the same place, we need to use a new term: 2>&1. When we use this, we redirect using the same syntax as before, but add 2>&1 to the end of our command.

 

This is how it works in practice:

 

./script.sh > combined_output.txt 2>&1

 

Now, if we look at our combined output file, we can see that we’ve captured both the stdout and the stderr.

 

cat combined_output.txt
Changing to a directory that doesn't exist
./script.sh: line 6: cd: foo: No such file or directory
© Wellcome Genome Campus Advanced Courses and Scientific Conferences
This article is from the free online

Bioinformatics for Biologists: An Introduction to Linux, Bash Scripting, and R

Created by
FutureLearn - Learning For Life

Reach your personal and professional goals

Unlock access to hundreds of expert online courses and degrees from top universities and educators to gain accredited qualifications and professional CV-building certificates.

Join over 18 million learners to launch, switch or build upon your career, all at your own pace, across a wide range of topic areas.

Start Learning now