Computing Resources

1.1.1. Shell script maintainability and reliability

1. Introduction
   

   The Shell Command Language is very liberal: altering source code in small ways will likely not cause an interpreter to produce different results, in most circumstances, but if things outside the programs (such as file contents or execution environment) vary, some programs will produce undesirable results while other program's behavior will continue to produce results you like. For example, these commands write different numbers of lines:

   statement="hello world" && printf '%s\n'  ${statement} 
   statement="hello world" && printf '%s\n' "${statement}"
   

   If your program expects both of those commands to write only one line, you will be surprised by the results, despite the only difference being some <space> ( ) characters are changed into quote (") characters!

   Because small changes in programs and inputs can produce severely negative results, great care should be taken to account for various types of reasonably likely situations

2. Common inconsistencies
   
   1. Parameter expansion without braces
      

      The $ character can introduce parameter expansion.

      Sometimes you want text to immediately follow the substituted value. For example, you may want to append _2 to the name of a file to make a backup of it, but doing this without braces produces results you wouldn't like:

      file=test.txt && touch -- "$file" && cp -- "$file" "$file_2"
      file=test.txt && touch -- "${file}" && cp -- "${file}" "${file}_2"
      

      The first command will likely fail while the second command succeeds, as variables that are unset often expand to zero characters, and "If the parameter is not enclosed in braces, and is a name, the expansion shall use the longest valid name […], whether or not the variable represented by that name exists."

      shellcheck documents that it can automatically fix this problem by running this command with a value like src/my_file.sh assigned to file:

      { shellcheck -x -o all -i SC2250 -f diff -- "./${file}" > shellcheck_diff || [ "${?}" = 1 ]; } && patch -p1 <shellcheck_diff
      

   2. Allowing Field Splitting
      

      Generally, text that is not <space> or <tab> is passed to a command as an argument, and the number of arguments is determined by how many sequences of characters that are not <space> or <tab> appear. For example, this command has three arguments:

      ls first second third
      

      However, a command can have additional arguments due to Field Splitting (substantially similar processes are "word splitting" and "word expansion"). This often happens when an expression or substitution caused new white space characters to be on the command line, and it results in the expansion being interpreted as a list of "fields" that undergo further expansion.

      For example, the ls command on each of these lines will have six arguments:

      mkdir -- test && cd -- test && touch -- a b c && unset -v -- IFS && three_fields="a b c" && ls ${three_fields} ${three_fields}
      mkdir -- test && cd -- test && touch -- a b c && unset -v -- IFS && star="*" && ls ${star} ${star}
      mkdir -- test && cd -- test && touch -- a b c && IFS="," && a_comma_b_comma_c="a,b,c" && ls ${a_comma_b_comma_c} ${a_comma_b_comma_c}
      mkdir -- test && cd -- test && touch -- a b c && IFS="," && star_comma_star="*,*" && ls ${star_comma_star}
      mkdir -- test && cd -- test && touch -- a b c && unset -v -- IFS && star_comma_star="*,*" && ls ${star_comma_star} ${star_comma_star} ${star_comma_star} ${star_comma_star} ${star_comma_star} ${star_comma_star}
      

      Unfortunately, field splitting often produces unexpected results (as you might have guessed from the examples), and it changes its behavior in reaction to the environment in cases that aren't commonly checked for.

      Because field splitting often leads to unexpected behavior, it should often be avoided by surrounding expansions and substitutions with double quote (") characters, as "the shell shall scan the results of expansions and substitutions that did not occur in double-quotes for field splitting".

      These commands will write different things, as the first does not handle the value of troublesome_text having spaces so writes many lines, whereas the second avoids field splitting so only prints one line that doesn't contain "bad":

      unset -v -- IFS && troublesome_text="bad -E" && printf '%s\n' "${troublesome_text}" "good text" > text.txt && opts="-v -e ${troublesome_text}" && grep -x ${opts} -- text.txt
      unset -v -- IFS && troublesome_text="bad -E" && printf '%s\n' "${troublesome_text}" "good text" > text.txt && should_invert_match="y" && grep -x ${should_invert_match:+"-v"} -e "${troublesome_text}" -- text.txt
      

      Even if field splitting isn't depended upon, it is still good to avoid by adding double-quotes: even if the program won't change its behavior in most cases due to lack of quoting, leaving a variable expansion unquoted makes it unclear whether you actually intended for field splitting to happen occasionally, and it may happen when you don't intend it to. For example, these commands write different things because the first allows field splitting of one expansion but the second doesn't use field splitting at all:

      unset -v -- IFS && troublesome_text="bad -E" && printf '%s\n' "${troublesome_text}" "good text" > text.txt && should_invert_match="y" && grep -x ${should_invert_match:+"-v"} -e ${troublesome_text} -- text.txt
      unset -v -- IFS && troublesome_text="bad -E" && printf '%s\n' "${troublesome_text}" "good text" > text.txt && should_invert_match="y" && grep -x ${should_invert_match:+"-v"} -e "${troublesome_text}" -- text.txt
      

   3. Not providing context for unusual paths
      

      Sometimes, people will make poor decisions about what to start their directory entries with. If they use a dash (-) as the first character of a name, that will often cause problems, as it could be interpreted as specifying one or more options instead of a file. For example, a file named -r would often be confusing to use rm with.

      To enable more types of commands to be run, many commands support the "end of options delimiter" (--) (described in the Utility Syntax Guidelines as linked to from various utility pages, to be recognized by default, named in a description of eval). For example, the first of these commands will not write anything as -x (the value of my_file) will be interpreted as specifying that grep should match only input lines where all characters in the line match, and the second command will match twice (one line in the file named -x and another in other_file.txt):

      my_file="-x" && printf '%s\n' "foo" "bar" > "${my_file}" && printf '%s\n' "baz" "qux" > other_file.txt && grep -e 'a' "${my_file}" other_file.txt
      my_file="-x" && printf '%s\n' "foo" "bar" > "${my_file}" && printf '%s\n' "baz" "qux" > other_file.txt && grep -e 'a' -- "${my_file}" other_file.txt
      

      Using the end of options delimiter makes it clear to everyone what you intend to be an option (with or without an option-argument) (like -e pattern) and what you intend to be an operand (like other_file.txt).

      Additionally, you should make sure every path string starts with a character other than - by checking whether they start with / and prepending ./ if not, to make sure the path is either explicitly absolute or clearly relative. This may sometimes be the only method available to get the behavior you want, especially to if a version of find interprets a file name as part of the "expression" (as both are operands and operands starting with "-" or that are "!" or "(" are treated differently from others). For example, the first of these commands has written a different number of lines when compared to the second:

      my_file="-print" && mkdir -p -- "${my_file}" && touch -- "${my_file}"/a "${my_file}"/b "${my_file}"/c && find -- "${my_file}" -type f
      my_file="-print" && { [ -z "${my_file##/*}" ] || my_file="./${my_file}"; } && mkdir -p -- "${my_file}" && touch -- "${my_file}"/a "${my_file}"/b "${my_file}"/c && find -- "${my_file}" -type f
      

      Making sure paths start with a safe character and using the end of options delimiter can be done at the same time, and doing both of these things is probably preferable to only doing one in case a mistake is made in implementing one of the practices.

   4. Not constructing commands such that they fail safely
      

      Commands can fail for unpredictable reasons, like being killed by a signal or having too many variables passed between environments. For example, one of these comands has failed in a similar context to a time when the other has succeeded:

      ( seq_numbers="$(seq 100000)" && for i in ${seq_numbers}; do eval 'export var_'"${i}"'=0'; done; ls; )
      ( seq_numbers="$(seq 10000)" && for i in ${seq_numbers}; do eval 'export var_'"${i}"'=0'; done; ls; )
      

      Because failure is possible and reasonably likely, commands should be constructed with the assumption that any command can fail unexpectedly, such that "the system's design prevents or mitigates unsafe consequences of the system's failure."

      For example, one of these commands fails to detect problems with a path due to it not being a valid directory entry but the other handles the fact the file is nonexistent gracefully by reacting the same way to gzip or awk exiting with a nonzero status (by changing != to == in the awk program to make it fail in an undesirable situation and swapping the "then compound-list" and "else compound-list"):

      set -o pipefail && file="nonexistent" && [ ! -e "${file}" ] && check_gzip_file() { if gzip -l -- "${1}" | awk 'NR==2 {exit($2!=0)}'; then printf '%s\n' the file may be empty and unsafe to use && false; else printf '%s\n' "the file may be populated"; fi; } && if check_gzip_file "${file}"; then printf '%s\n' "the file passed the check"; else printf '%s\n' "the file failed the check"; fi
      set -o pipefail && file="nonexistent" && [ ! -e "${file}" ] && check_gzip_file() { if gzip -l -- "${1}" | awk 'NR==2 {exit($2==0)}'; then printf '%s\n' "the file may be populated"; else printf '%s\n' the file may be empty and unsafe to use && false; fi; } && if check_gzip_file "${file}"; then printf '%s\n' "the file passed the check"; else printf '%s\n' "the file failed the check"; fi
      

      Essentially, when the exit status of a command is being checked, you should only assume the command succeeded if its exit status is exactly 0, and ensure that it is not likely to be 0 if something unexpected happened. Also, every if statement should have an associated else word since if it doesn't it can be replaced with an AND-OR list to better express what your expectations for the command are (though using if with an appropriate else list is often preferred over using an AND-OR list, and storing exit status in variables over checking them directly or indirectly is likely preferred), and "In some cases, status values are listed more loosely, such as >0. A strictly conforming application shall not rely on any specific value in the range shown and shall be prepared to receive any value in the range."

      You can construct a list to react to a specific exit value if that is necessary. For example, grep exits with status 1 if no lines were selected but a different nonzero status if an error occured, so checking for an exit status of 1 differentiates a non-match from both a match and a problem that prevented a full search. This command succeeds because the exit status of grep is mapped to the typical "0 is good, anything else is bad" space by [:

      printf '%s\n' "hello" >test.txt && ( set +e && { grep -e 'goodbye' -- test.txt; [ "${?}" = 1 ]; }; )
      

   5. Not using variable names that include lowercase letters
      

      It is documented that "The name space of environment variable names containing lowercase letters is reserved for applications. Applications can define any environment variables with names from this name space without modifying the behavior of the standard utilities."

      This implies that any variable name that does not contain lowercase leters can modify the behavior of the standard utilities.

      Note that a system may provide non-standard extensions, so variable names that are not mentioned in existing documentation could be made to affect standard utilities in the future. Consider this example, where sh could be a different shell: unset -v -- SHLVL && SHLVL=400 && export -- SHLVL && sh -c -- 'printf "%s\n" "${SHLVL}"'

   6. Not making a non-zero exit status of a compound-list available when it is used with a compound command
      

      It can be difficult to tell if a compound-list used with a compound command has a non-zero exit status, since the exit status of the compound command can often be 0 (if no compound-list was executed) or the status of the last compound-list that was executed. This means that if no compound-list was executed, it's not necessarily clear whether no compound-list was executed or if the final compound-list to be executed had an exit status of 0, and if more than one compound-list was executed it's not necessarily clear what the exit status of any other compound-list was (or whether another one was executed at all).

      This is not true of grouping commands since the exit status of a grouping command shall be the exit status of its only compound-list and it is always executed.

      In many cases, making exit be executed as the final command of the compound-list if the most recently executed command had a non-zero exit status and making printf '%s\n' "${?}" be executed just before that will at least record the exit status or exit the program (and return the exit status of the printf command if it had a non-zero exit status), so more details will be made available for debugging purposes. This might require changing a single AND-OR list into a sequence of lists, and doing that while retaining all the useful properties of the original AND-OR list might require a lot of thought.

      1. Examples
         

         for path in / /dev/null ~; do [ -d "${path}" ]; done
         for path in / /dev/null ~; do { { [ -d "${path}" ]; }; printf '%s\n' "${?}"; } || exit; done
         

         sudo -- systemctl restart monero && sudo -- systemctl stop monero && sudo -- nixos-rebuild switch --upgrade && reboot
         if sudo -- systemctl restart monero; then if sudo -- systemctl stop monero; then if sudo -- nixos-rebuild switch --upgrade; then reboot; else printf '%s\n' "${?} nixos-rebuild" || exit; fi; else printf '%s\n' "${?} systemctl stop" || exit; fi; else printf '%s\n' "${?} systemctl restart" || exit; fi
         

   7. Not using until instead of while
      

      The exit status of a while loop only depends on one compound-list used with it, so it isn't very clear how the exit value of the other can be recorded. On the contrary, if the exit status of an until loop is 0 it is clear that each compound-list used with it succeeded if it was executed at all, and by using the ? parameter with the first command of the second compound-list enclosed in the loop it can be determined what the exit status of the other compound-list was.

      1. Examples
         

         touch -- test.txt && until [ ! -e test.txt ]; do printf '%s\n' "test result: ${?}" || exit; rm -- test.txt; printf '%s\n' "action result: ${?}" || exit; done
         

      2. See also
         

         https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04_09

         https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_09_04_11

         https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_05_02

   8. Using the <<- redirection operator instead of the << redirection operator
      

      A here-document is often useful for embedding lines you want to use with the input of a command into a script file. However, it is unclear if using <<- should result in leading <space> characters being stripped, as that behavior is not well documented with POSIX or bash, but that often happens anyway. Fixer programs often change any leading <tab> characters into leading sequences of <space> characters, but sometimes don't undo that when converting the file to use <tab> characters for indentation. This leads to the fixing process producing more changes than is necessary, so the <<- redirection operator should be replaced with the << redirection operator where possible.

      For example, make source code more like the first of these snippets than the second:

      cat <<"END"
      hello
      END
      

      cat <<-"END"
              hello
      END
      

3. Examples of non-obvious assumptions in documentation
   

   Nobody should be able to create or rename entries in a directory your program knows about: https://www.gnu.org/software/findutils/manual/html_mono/find.html#A-more-secure-version-of-_002dexec

   File names and paths should be short: file names must be as short as 14 characters and paths must be as short as 255 characters: "If any pathname component is longer than {NAME_MAX}, the implementation shall consider this an error", "In the cases where prefixing occurs, if the combined length exceeds {PATH_MAX}, and the implementation considers this to be an error, pathname resolution shall fail" ({NAME_MAX} and {PATH_MAX} defined by https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/limits.h.html and given more context by https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pathchk.html)

   File names should not contain a <space> ( ) and should be restricted to a small set of characters: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_08 (which justifies the limited functionality of find and xargs compared to the GNU versions).

4. See also
   

   A condensed explanation of how software like Unix is designed and how it affects the process of making a user program (its wikipedia page)

   Analyses of how software development works in the Unix ecosystem (especially The Art of Unix Programming and The Cathedral and the Bazaar (the main text))

   shellcheck git repository (also links to the project's website which has a wiki)

   shfmt git repository

   some pages which may be of interest for people doing Unix shell scripting

   Commentary on how to handle input with links to other resources

   Google Shell Style Guide

   POSIX.1-2017 defines a standard operating system interface and environment

   Linux Foundation Referenced Specifications (a more reliable link to The Single UNIX® Specification, Version 2)

   IETF standards

   Bash Reference Manual (and other manuals by project)

   ALE for automated linting with the vim text editor that works well with the shell command language and other languages

1.1.2. See also

1.2.1. See also

1.3.1. See also

1.5.1. See also

1.11.1. See also

1.14.1. Avoid inline comments

# Writes text
echo hello
# Writes more text
for item in "${@}"; do
  echo "a parameter is ${item}"
done

echo hello             # Writes text
for item in "${@}"; do # Writes more text
  echo "a parameter is ${item}"
done

1.14.2. Function comments

1.18.1. See also