What every SRE should know about GNU/Linux shell related internals: file descriptors, pipes, terminals, user sessions, process groups and daemons #

Despite the era of containers, virtualization, and the rising number of UI of all kinds, SREs often spend a significant part of their time in GNU/Linux shells. It could be debugging, testing, developing, or preparing the new infrastructure. It may be the good old bash, the more recent and fancy zsh, or even fish or tcsh with their interesting and unique features.

But it is common nowadays how little people know about the internals of their shells, terminals, and relations between processes. All are taken primarily for granted without really thinking about such aspects.

Have you ever thought about how a shell pipe works, how pressing the CTRL+C combination delivers the interrupt signal to the currently running shell processes, or how vim rewrites the content of the console when you change the size of your terminal window?

I want to show you some indeed neat parts of pipes, file descriptors, shells, terminals, processes, jobs, and signals in this series of posts. We’ll touch on how all of them interact with each other to build a responsible, simple, and reliable environment. And all of this, of course, will be shown in the context of the Linux kernel, its internals, and various debugging tools and approaches.

We are going to play with file descriptors, pipes, different tools such as nohup and pv, experiment with background and foreground processes, understand how tmux gives us the ability to continue where we stopped, why and how the CTRL+C interrupts the currently running pipeline of commands and much much more. Also, we will use strace to trace syscalls, read the Linux kernel source code, and use bpftrace to get under the hood of arbitrary kernel functions.

Prepare environment #

During the series, I’ll mix python and golang for my examples. Also, we’ll need a file for our experiments. I use /var/tmp/file1.db. You can easily generate it using the following command:

$ dd if=/dev/random of=/var/tmp/file1.db count=100 bs=1M

Dive #

With all that said, let’s learn, experiment, and have fun.