Automating things using the Linux command line

Hello World

• echo is the the same as "print" or "write" in other languages. It takes arguments and shows them - without or with very little actual processing being done.
  • It's handy to test expressions or to show the content of variables (See section Variables).
  • echo is not a Bash command per se but an application being run by Bash. The application's path is /bin/echo. There are thousands of those commands available even on a basic Linux system.
• The command is given two strings: "Hello" and "World". It's not a single one "Hello World", as one could expect.
  • Bash splits up command arguments on white spaces.
  • This means: To send a white space to a command read more about that in the Quoting section.
  • Other effects: You can use an arbitrary number of white spaces between "Hello" and "World" - the output will be the same.
• There are only a few pure Bash commands but every program installed as a software packages becomes available as a command here. In a shell, most commands are just regular software binaries. For example the firefox command will start a web browser.
  • Examples:
    • Yes, echo is part of a software package. However, it is guaranteed to exist on every UNIX system - including your Internet router at home and your cell phone.
    • Matlab (Command: matlab )
    • FSL (Command example: fsl_reg )

• A Google search for "list files linux" will tell us that ls is the command of choice here.
• Most Linux commands do have a short help which is displayed by typing [command] --help (more likely) or [command] -help .
• Most Linux commands go even further and provide a separate help page via man [command]
  • The man command opens an interactive text reader
  • Use cursor keys and pgup/pgdown to scroll
  • Use q to quit
  • Use /[text] (slash followed by some text) to search for text
• You'll quickly find the necessary options: -1 , -t and depending on if you want ascending or descending order: -r
• Solution: ls -1 -t -r

• Options given to commands usually look like --someoption (long option) or -o (short option) and are location-independent ( -a -o equals -o -a ). However, this is just a convention most programmers of Linux software abide by. Some commands may behave differently.
• USE GOOGLE . It's amazing how much specific knowledge is accessible there.
  • Try to be as general as possible in your search query. This the way to get knowledge that doesn't help you instantly but will get you an idea how to approach the problem.
  • Example: You want to reverse 10000 MPEG1 video files. Do this via clicking in Final Cut Pro would be time consuming. A command line solution is better.
    • Bad google query: "How to reverse mpeg1 video files"
    • Good google query: "Linux command line reverse video"
    • You'll get some command lines as one of the first results explaining how to use ffmpeg and sox to do that.
    • Side note: ffmpeg (alias avconv) is an extremely powerful video processing software. sox is an extremely powerful audio processing tool. Both are installed on every workstation in the institute.
    • In this case: Make sure to understand, how video processing works. If in doubt ask your department's technician or an IT staff member.
    • In general: Make sure, you understand the solution. If in doubt ask your department's technician or an IT staff member.

• Putting them on separate lines (Command 1, Enter, Command 2, Enter)
• Separating them by ;

Hello

World

Hello
World

• Both examples are semantically equivalent. However, the prompt display is different. In a shell script, there would be no prompt and the output would look exactly the same.

• There are no commands called preprocess/process/postprocess by default available in the institute. You'd have to write those commands yourself. That's actually easier than it sounds.
• After the variable was assigned the Value GT4K, using the Syntax $subject later will make the shell replace this variable reference by the variable's content before the actual command is running.
• The variable is only valid in one shell. No collisions will occur if you use the same variable name in different shells, i.e. in different Terminal Windows.
• There's a catch: The subject ID can not contain white spaces or other characters other than alphanumeric ones in this example.

• Example one uses double-quotes. You'll need those most of the time. Double quotes prevent the Bash from interpreting a given string into multiple arguments. Without double quotes, process_subject would get 3 useless arguments instead of a single correct one.
• Example two uses single quotes which cover most of the special characters, e.g. $ signs.
• Example three shows a problem with variables, you might run into. Since a variable may contain special characters, you have to be careful when using it. This example shows the safe way - using double quotes. This will work even with special characters in the variable which a double quote could not handle directly.

• In this example, Bash tries to use a variable called "$myvariable_sometext" and not "$myvariable" plus some literal text "_sometext". This is because "_" can be used as part of a variable name. The command will just print an empty line.
• user@host > echo ${myvariable}_sometext will handle this correctly.

• The process_subject command will see a single argument: "GT4K Session 2" including the white spaces. It can in turn successfully open the file with this name.

• abc* is a called a globber expression. Expressions like these are replaced by filenames matching it.
• abc* means "All files and directories in the current directory whose names begin with abc followed by an arbitrary number of characters".
• The ls command will show all files and directories within the current directory starting with abc as separate arguments.
  • The option -l formats the output as a list.
  • The ls command will never see the globber expression. It does not know that...
    • ... there is such a thing as a globber expression
    • ... how to interpret them
    • ... that the filenames were presented using an expression and not via manual typing.
• Globber expressions can be more complex, but that is beyond the scope of this manual.
• Certain special characters like * , ? , [ and ] make a string a globber expression. If you want to use them as literal argument for commands, quoting must be used.
• These expressions are a good example of the sharing of labor in Linux. A command doesn't need to search files for itself or interpret * - this is done by the shell. There are thousands of commands who rely on filenames being given as arguments. None of those need to know about how to look for files. This saves precious development time for you too - if you decide to write scripts working with files.
  • This sharing of labor is a key difference between a DOS/Windows command line and a Linux shell.

• objects (since we're using rm without -r, we're most likely looking for files - not directories) called somefile.txt ...
• ... which are located in a directory starting with the letter a (minor or capital) ...
• ... which is in turn a subdirectory of a directory with a name starting a two digit number less or equal 52, continuing with the literal string _[2014] and optionally ending with an arbitrary string of characters ...
• ... which resides in the current directory

001
002
003

• {001..003} is a word generator. It acts as if you typed 001 002 003 in its place. OK, not much typing is saved in this example. Now imagine replacing 003 by 999.
• The word do has to be there.
• The word done marks the end of the loop. Only commands between do and done are running within the loop.
• We didn't use double-quotes around $subject. This is usually a bad idea. It works in this case since the word generator only makes numeric strings.

• The process_subject command is called once for each file and for each directory in the directory, you're currently in.
• You should try to keep you subject's base directory free of other files because they would be caught by the * expression otherwise.

• You can keep all the necessary processing commands together in one place.
• You can write comments explaining your thoughts when writing the script. Lines starting with # are comments.
• RevisionControl can be used to track changes in the command set being used for your data over time.
• A set of scripts can be kept per study/project.

#!/bin/bash
#
# This is the process_subject script
#
# It will do the following:
# ...

subject="$1"

echo "Test subject $subject is being processed"
# Some preprocessing
process_step1 "$subject"
# The most important part:
process_step2 "$subject"

• Lines starting with # are comments.
  • You should write a little comment header on top of your script to describe the purpose of it.
  • It's good practice to document your intentions when running a command in a comment line above the command. Try to describe why not what (globber expressions are an exception).
  • The first line of a script is not a comment but a hint for the operating system, which programming language the script is written in (shebang). The line must always be #!/bin/bash in case of a Bash script.
• There's a script directory in every AFS study storage block by default. Save the script in this directory to keep it close to your data. The path will the look like this:
  /afs/cbs.mpg.de/projects/afs000_afstest/scripts/process_subject .
• $1 is the first parameter given to the command. Calling process_subject GT4K would cause $1 to be GT4K which would cause the variable $subject to be GT4K which would send this ID to the process_step1 and process_step2 commands.

• See Expanding the search path
• and Using a central command location.

I: AFS study environment (Project id 'afs000')
I: Type 'envlist' to show all environment components incl. versions.
I: Type 'exit' to leave the environment

• The change only affects a single Bash instance. You can work with multiple studies and multiple sets of scripts in multiple windows.

• No preparation is necessary before working with a single study. The shortened command is available as soon as you log into your Linux session.

• This tool will call a command called process_subject once for every file/directory starting with the string abc in the current directory.
• This will even work, if there are hundreds of thousands of files starting with abc in that directory.
• 4 process_subject instances will run in fparallel. Whenever a job finishes, another argument (in this case another file starting with abc ) is taken from the job queue and being processed.
• Running more instances than your number of CPU cores in fparallel might knock out your computer.
  • Keep in mind that you might need processing power for your graphical session. Reserving one CPU core for that might be a good idea.

Example

• You've got a text file containing ZIP Codes of Germany.
  • I downloaded it from http://download.geonames.org/export/zip - it's called DE.txt
  • Every line looks like this: DE 04103 Leipzig Sachsen ... (ZIP Code, tab, city name, other stuff). There are >16000 lines
• You need every name of a city with a 4 as second digit of its ZIP Code

• We only need the fields 2 (ZIP Code) and 3 (city name) and throw away the rest: cut -d$'\t' -f2-3
• Then we apply a filter to only get ZIP Codes with 4 as second digit: grep '^.4'
  • The argument given to the grep command is a regular expression
• Since we're only interested in the city names, the ZIP Codes are thrown away: cut -d$'\t' -f2
  • Remember: The city name was field 3 in the beginning but field 1 was thrown away - it's field 2 now and will be field 1 after this step
• There are a lot of city names given multiple times now. We're going to remove the duplicates. However, the command for that requires that all duplicates are in adjacent lines - we have to sort first:
  • To sort all the city names: sort
  • To remove the duplicates: uniq

• The pipeline only connects STDOUT to STDIN. All error messages are displayed in the terminal - independently of its position in the pipeline.

Pros and Cons

• user@host > cat file | ( while read line;do echo "'${line}'";done ) > newfile
• user@host > cat file | awk "{ print \"'\" \$1 \"'\" }" > newfile
• user@host > cat file | sed "s/\(.*\)/'\1'/" > newfile