Linux Crashcourse

From 0 to using Linux in some minutes

Chances are good that you workstation is equipped with an operating system, you're unfamiliar with. It's called "Debian" and has a "Linux"-Core.

What you can do to your computer and what you can't

The institute's central IT takes care of configuring, upgrading and fixing your computer. You are not allowed and in most cases not able to do that yourself. IT installs software in a way that every user on a computer is able to use it. However, as a user it's perfectly fine to download and use scripts, Matlab toolboxes, etc. Those can be stored in an area of the filesystem accessible to you and be used there. The main idea of restricting software management permissions to IT staff is to keep users from interfering with each other's work.

Processing data

On a Linux based computer that you use privately, it's possible to rely on graphical tools exclusively for tasks like writing emails, writing text, surfing the Web,... However, in the institute you might have to write scripts on your own, run them on different computers over the network and process their results. To do that, you'll have to use a command line interface called shell. A shell is a command line interpreter which runs in a terminal window. A shell is automatically started, whenever you start a "Terminal" (just select the respective entry in the start menu).

It's not as fancy as graphical applications but:

• text-oriented programs--e.g. your scripts--can be developed a lot faster than graphical ones.
• Almost anything is possible regarding automation.
  • You got a script for processing one data set? Use a for ...; do ...; done statement to handle thousands of them.
  • Your computers takes an hour per data set and you need 100 processed by tomorrow? Use the institute's ComputeClusterSlurm.

How text oriented programs work

The operating system on your workstation consists mostly of small programs that

• read from an input channel (let's call it STDIN), which usually means "read text entered by the user via keyboard";
• process it in some defined way, e.g. sort the lines that were typed and
• send it to an output channel (let's call it STDOUT), which is usually the terminal window.

Hints:

• There is another output channel (let's call it STDERR) to which error text messages are printed to. This way they cannot be confused with regular output of the program.
• When working in a terminal, STDOUT and STDERR appear to be mixed together, but when automating things, it's easy to separate them.
• Programs not working like that are a rare exception which you might never see. Even graphical programs use at least STDERR to tell you about minor problems.
• Every application installed on your computer can be easily run in a shell. Just type e.g. firefox to start your browser.

Some tips for working in a shell

• To copy and paste text, just select the text to copy and click the middle mouse button (or the wheel) where you want to paste it. This very efficient and works in all Linux programs. No need for Ctrl+C/Ctrl+V
• A shell doesn't know about mice. Pasting text there is just simulating key strokes. If you need to paste at a specific place on your command line, you'll have to go there using your cursor keys first.
• To quickly select a word, double click it. To quickly select a line, tripple click it.
• Use cursor-up/cursor-down to scroll through your shell's command history.
• You might want to give mc a try. It's an extremely powerful file manager and a lot faster than any graphical one. It's particularly useful to apply scripts to a specific huge set of files. It you want to know more, just ask me--I promise to be impartial when praising the hell out of this cool app :-) .

Files and Folders

Files are stored in a single file system tree. There's a base directory called /, which contains all files available to you--even the ones in the network. macOS works the same way (although it tries to hide it from you by managing everything as "Volumes"); Windows doesn't--its file system consists of a "forest" of trees (a.k.a. Drive Letters) which might magically appear or disappear upon connecting/disconnecting mobile disk etc.

A file system looks like this: There are directories and sub directories and files in them.

Hints:

• Path components are separated by / which is like in macOS but different from Windows.
• A / on the end of a name is an indication that it's a folder.
• Whenever a path starts with / (e.g. /data/hu_someone), you have all available information to find the object (file or folder) it's pointing to, called an "absolute path".
• A path like hu_someone/Desktop is only useful, if you know where to look for it. It's called a "relative path".
• When you connect a flash drive or a DVD-ROM, it will magically appear as a sub folder of /media
• Choose names of folders and files wisely. While the operating system itself only forbids / and "ASCII 00" in path components, programs might have a hard time coping with German umlauts, white spaces, ! or 漢語. Try to restrict file name characters to [a-z] , [A-Z] , [0-9] , . and _ .
• Folders on other computers in the network are sometimes invisible (e.g. /nobackup/somecomputer1 ) but if you go to them, they will magically appear. To be more precise: If you open /nobackup in a file manager, the folder might be empty. But if you type cd /nobackup/somecomputer1 you'll see the remote computer's folder. Read more about this here .
• The filesystem looks almost the same on all workstation and servers that are accessible to you. Only paths like /ALLCAPITALLETTERS and /tmp are local which means, they won't have the same content on two different computers.
• Files starting with a dot (.) are considered hidden. This means, most programs choose not to show these files by default, although the operating system itself doesn't consider them special in any way. If you think there are files not being displayed, try to look for a "Show hidden files" option in your file manager software.

Your home directory

A home directory is just a directory tree containing your settings, some special files and your desktop, including the files you put there. Without the home directory, you'll not be able to login. In the institute, your home directory resides on a file server which means:

• A home directory is protected very well. It's located on an expensive machine in an air conditioned server room which is locked.
• IT does nightly snapshots (backups) to ensure that you'll regain access to that important file, you accidentally deleted 3 days ago.
• Every workstation of the institute can access this folder. Your computer is broken? Take the next one.

A home directory is not exclusive. You can login:

• ... multiple times
• ... on multiple computers
• ... from multiple continents via the institute's remote access service.

There's a catch:

• Your browser is exclusive. The browser's profile (history, bookmarks, cache, ...) can only be used by a single running instance. There are ways to cope with that restriction. Just ask IT if you want to know more.

Working with files in a shell

When you find yourself in a shell in newly opened terminal, you are in your home directory. Every program is assigned a current working directory. In a shell, you can change it easily.

Command	Description
cd /some/folder	You'll go to a specific directory independent of the current working directory
cd somefolder	You'll go to a folder which resides inside your current working directory
cd ~	You'll end up in your home directory. ~ is a symbol that the shell replaces with your home directory before calling cd.
cd ~/Desktop	You'll end up in your desktop which is the folder Desktop/ one level below your home directory.
cd -	You'll end up in the directory you were in before the last cd command.

It's perfectly fine to use a graphical file manager like Thunar to manage your files. However, there's a text based file manager available, you should give a try: mc (Midnight Commander).

Patterns to select files

Shells are very powerful when it comes to handle many things at once. Here's an example task:

• In a folder /NOBACKUP/subjects there are a thousand test subjects, one subfolder per test subject each.
• In some of them there is a folder stimuli.
• Some of the stimuli folders contain video files.
• From these video files, all .avi files with an a as second character of their name should be copied to an external disk.

This can be done with a single command:

user@host > cp /NOBACKUP/subjects/*/stimuli/?a*.avi /media/myusbdrive

Hints:

• The cp application is just a dumb copy program. It doesn't know about patterns
• The shell interprets the given pattern and substitute it by all the filenames matching it.
• * is short for "arbitrary number of characters except /=". And =? is short for "single character except / ". There are other special characters (metacharacters). Ask IT if you want to know more.
• If you want to try the pattern first--no problemo:
  user@host > echo /NOBACKUP/subjects/*/stimuli/?a*.avi
  Echo is a dumb program was well. It will just show all the parameters it is given, after the shell did its interpretations of metacharacters.