Debugging code can be time consuming and tricky at times. Programmers have traditionally taken advantage of the techniques like syntax highlighting, code completion, intellisense, etc while writing and debugging code. Also no amount of such features can reduce logic based errors, where the script executes but not in a way we expected. This is more problematic with complex bash scripts as bash itself has no support for advanced features like these and most of the sysadmins spend their time with the editors like vi/nano/atom. Sometimes, even simple mistakes like missing a space, can make your scripts fail to run. In this blog post, we’ll discuss some of the common ways to debug bash shell scripts.

Using Bash Debugging Options

The debugging options available in Bash can be turned off/on in multiple ways. They can be supplied at command line while running the scripts, they can be embedded within the scripts and they can be set/unset in existing sessions.

Syntax Check using noexec mode

There can be situations where we want to check if the syntax of the script is valid before executing it. In fact, it is always a best practice to do the same. For this, we can use the noexec mode in the bash by using bash -n switch.

Consider, below code where we removed the then statement:

#!/bin/bash
read -p "Enter a value: " val

if [ ! -z $val ]; # missing then statement
  echo "The value of \$val is $val"
fi

We can identify the errors by calling bash -n followed by name of the script:

Using Verbose Mode with -v

While some of us would like the echo based debugging, it gets hard at times. One of the useful options is enabling verbose mode with -v option. This allows to print each statement, before executing it. For example, consider below code to output value of a variable:

#!/bin/bash
read -p "Enter a value: " val

# this works - 
if [ ! -z $val ];then
  echo "The value of \$val is $val"
fi

Consider below output with and without -v option:

Alternatively, we can also enable it within the script, at the shebang line:

#!/bin/bash -v

Another way, is to use the set -v option:

#!/bin/bash
set -v

Using Tracing with xtrace mode

To identify logical errors, we need to analyze the values stored inside our variables, and know the commands that are being executed. Consider below faulty code for outputting the value of variable, if not null:

#!/bin/bash
read -p "Enter a value: " val

# echo value if not null
if [ -z $val ];then # missing negate ! operator
  echo "The value of \$val is $val"
else
  echo "Got a null !"
fi

However, we made a logical mistake by not using the ! operator inside if condition. So it will execute like below:

We can identify our mistake by enabling trace with -x option and then running the script. We can see that, inside if condition, we are comparing null with 10 and therefore it does not pass. So echo statement to print the value of the variable is skipped:

However, as you can see, we are not sure what statement are being executed. Note that the line preceded by + sign are generated by xtrace mode. Fortunately, we can combine -x and -v options together and run our script, which will make it more useful:

Again, we can also enable it within the script, at the shebang line:

#!/bin/bash -x

Another way, is to use the set -x option:

#!/bin/bash
set -x

Identify Unset Variables

Unset variables refers to the variables, where we think that we have defined the variable in the code, but unfortunately, we did not. We have all been to situations, where we have mistyped variable names in our code. Consider below faulty code, where we mistyped the name of the variable two_val:

#!/bin/bash
read -p "Enter first value: " one_val
read -p "Enter second value: " two_val

# echo the sum of values
sum=$(($one_val + $too_val)) # mistyped the name of second variable
echo $sum

Notice how it runs:

The error message is not very useful, since we have not made syntax error as indicated by it. To detect such situations, we can use -u option. The -u option treats unset variables and parameters as an error when performing parameter expansion. Consequently, we get an error notification that a variable is not bound to value, while executing the script with -u option:

Again, we can combine this option with xtrace and verbose options:

Debugging Parts of the Script

Enabling xtrace and verbose is useful. however at times, scripts can be complex and long and you just want to focus on specific parts of the script. We can achieve the same by enabling/disabling these options, inside script, by using set command before problematic code block.

Consider the previous code to print the input variable, let’s just focus to enable trace and verbose only for line containing if condition:

#!/bin/bash
read -p "Enter a value: " val

# echo value if not null
set -x # enables xtrace
if [ ! -z $val ];then  
  echo "The value of \$val is val"  
else
  echo "Got a null !"
fi
set +x # disables xtrace

echo "Script ends here !!!"

Consider, the output below, which certainly looks less cluttered:

Enable/Disable file globbing

One of the other useful debugging option is ability to disable / enable globbing for filenames. It is set with -f. Setting this option will turn off globbing (expansion of wildcards to generate file names) while it is enabled. This -f option can be a switch used at the command line with bash, after the shebang in a file or, as in this example to surround a block of code.

Consider below code to search for keyword sum in all files in current directory:

#!/bin/bash

echo "By default, file globbing option is turned off"
grep -i "sum" *

echo "Disable file globbing option"
set -f # disable file globbing option 
grep -i "sum" *
echo "Enable file globbing option"
set +f # enables file globbing option

Notice the difference between the output of grep command when globbing is disabled versus when it is enabled:

Debugging Scripts using Trap

Debug Trap feature of bash allows us to execute a command repetitively. The command specified in the arguments of trap command is executed before each subsequent statement in the script. Consider below code to find the sum of two variables:

#!/bin/bash
trap 'echo "Line#${LINENO}: one_val=${one_val}, two_val=${two_val}, sum=${sum}"' debug # enables debug trap
one_val=10
two_val=5

# echo the sum of values
sum=$(($one_val + $two_val)) 
echo $sum
trap - debug # disables debug trap

Here, below command is executed repetitively:

echo "Line#${LINENO}: one_val=${one_val}, two_val=${two_val}, sum=${sum}"

Notice how its executed:

Using Breakpoints in the Script

Breakpoints is the ability to halt the program execution at specific lines of code and then resume execution from there. Although this is not directly supported in Bash, a somewhat workaround is available. For this, we can use the debug trap feature combined with the read statement, which will halt at current line and wait for user to press Enter.

We can further combine it with -x or -v option to make it more useful. Consider below code:

#!/bin/bash
trap read debug # enables debug trap
set -x # enables trace mode
one_val=10
two_val=5

# echo the sum of values
sum=$(($one_val + $two_val))
echo $sum
set +x # disables trace mode
trap - debug # disables debug trap

Notice how its executed:

Debug using special variable PS4

The PS4 variable defines the prompt that gets displayed when we execute a shell script in xtrace mode. The default value of PS4 is  +. We can also change this to reflect a certain modified trace output. For few, this might be preferable rather than using debug trap option. For example, consider below code:

#!/bin/bash
export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

one_val=10
two_val=5

# echo the sum of values
sum=$(($one_val + $two_val)) 
echo $sum

Note that we will enable the trace at run time using bash -x option rather than specifying it within the script. This will allow us to get debug output only when we want. Notice how its executed normally versus with the xtrace option:

Note that the first character of $PS4 i.e. + is replicated as many times as needed to denote levels of indirection in the currently executing command:

Debugging using Process Tree

Sometimes, it is helpful to know, what processes are being initiated by the script and what those processes are doing. With the help of pstree, we can see the processes generated by the script. We can also use -p to display the process id for the process as well. For example, consider below code:

echo $( echo $( for i in *; do echo $i; sleep 5; done ) )

Take a look at the process tree output:

/mohit/src/bash$ pstree -p
 init(1)─┬─init(7)───bash(8)─┬─bash(94)───bash(95)───bash(96)───sleep(98)
         │                   └─pstree(99)
         └─{init}(6)

We can also use strace command to check more the system calls generated by a process:

mohit/src/bash$ sudo strace -c -fp 108
 [sudo] password for mgoyal:
 strace: Process 108 attached
 break-point.sh debug-trap.sh file-globbing.sh ps4-variable.sh pstree.sh script-parts.sh set-verbose.sh set-xtrace.sh syntax-check.sh unset-var.sh
 % time     seconds  usecs/call     calls    errors syscall
 
 100.00    0.000317         159         2         1 wait4
   0.00    0.000000           0         3           read
   0.00    0.000000           0         1           write
   0.00    0.000000           0         1           close
   0.00    0.000000           0         2           rt_sigaction
   0.00    0.000000           0         4           rt_sigprocmask
   0.00    0.000000           0         1           rt_sigreturn
 
 100.00    0.000317                    14         1 total
 [2]+  Done                    ./pstree.sh