What every SRE should know on Viacheslav Biriukov

Async Rust gotcha: evolving tokio::select! code has sharp edges

Mon, 01 Jan 0001 00:00:00 +0000

… or a story about Cancellation safety, FutureExt::then(), insufficient tests, and I/O actors.

Posted: Feb 2026

TL;DR #

Be careful evolving and refactoring your application with tokio::select!, because the missing or too simple tests could lead to data losses due to cancellation safety issues. Assume any branch can be dropped at any .await unless you’ve designed it to be restart/cancel-safe.
stream.next().then() is not cancellation safe if you have an await point inside the async closure – data can be lost.
Make your tests exercise Poll::Pending paths. If you test with tokio::io::duplex(), check that it goes pending, otherwise cancellation bugs might not show up.
Follow the single I/O actor model – no mutexes, no sharing I/O objects.
Use tokio::mpsc channels with allocated permits by calling reserve() to consume data from streams only when you have capacity on the next step (producer side). This propagates backpressure and preserves data.

Intro #

Rust adoption keeps accelerating, and many new teams are writing Async Rust for production systems for the first time. Developers arrive from both low-level systems backgrounds and enterprise ecosystems – but Async Rust, and Tokio especially, comes with a few crucial concepts that can make or break your ability to build high-performance, zero-cost abstraction networking code.

Trixter: A Chaos Proxy for Simulating Network Faults

Mon, 01 Jan 0001 00:00:00 +0000

Posted: Oct 2025

Github: https://github.com/brk0v/trixter

Contents

Break your network before production does

Async Rust with Tokio I/O Streams: Backpressure, Concurrency, and Ergonomics

Mon, 01 Jan 0001 00:00:00 +0000

Async Rust with Tokio I/O Streams: Backpressure, Concurrency, and Ergonomics #

Last updated: Oct 2025

Contents

There are many excellent, straightforward guides for getting started with Async Rust and Tokio. Most focus on the core building blocks: Tokio primitives, Rust futures, and concepts such as Pin/Unpin, and often finish with a simple TCP client-server sample. In fact, the typical tutorial example can usually be reduced to something as simple as:

GNU/Linux shell related internals

Mon, 01 Jan 0001 00:00:00 +0000

Last updated: Oct 2025

Contents

Despite the era of containers, virtualization, and the rising number of UI of all kinds, SREs often spend a significant part of their time in GNU/Linux shells. It could be debugging, testing, developing, or preparing the new infrastructure. It may be the good old bash, the more recent and fancy zsh, or even fish or tcsh with their interesting and unique features.

Linux Page Cache for SRE

Mon, 01 Jan 0001 00:00:00 +0000

SRE deep dive into Linux Page Cache #

Last updated: Oct 2025

Contents

In this series of articles, I would like to talk about Linux Page Cache. I believe that the following knowledge of the theory and tools is essential and crucial for every SRE. This understanding can help both in usual and routine everyday DevOps-like tasks and in emergency debugging and firefighting. Page Cache is often left unattended, and its better understanding leads to the following:

What every SRE should know about GNU/Linux resolvers and Dual-Stack applications

Mon, 01 Jan 0001 00:00:00 +0000

What every SRE should know about GNU/Linux resolvers and Dual-Stack applications #

Last updated: Oct 2025

Contents

In this series of posts, I’d like to make a deep dive into the GNU/Linux local facilities used to convert a domain name or hostname into IP addresses, specifically in the context of dual-stack applications. This process of resolution is one of the oldest forms of networking abstraction, designed to replace hard-to-remember network addresses with human-readable strings. Although it may seem simple at first glance, the entire process involving stub resolvers is filled with complexities and subtle nuances. One contributing factor to this complexity is the growing number of IPv6 addresses, which, although not increasing at the pace everyone might want, is gradually changing servers and clients to support dual-stack hosts. Thus a seamless transition to IPv6 become an important feature and should occur without degrading user experience or increasing response latency.

File descriptor and open file description

Mon, 01 Jan 0001 00:00:00 +0000

File descriptor and open file description #

Last updated: Oct 2025

Contents

First of all, I want to touch on the two fundamental concepts of working with files:

I/O loop

Mon, 01 Jan 0001 00:00:00 +0000

I/O loop #

Last updated: Oct 2025

Contents

Before changing the code, let’s outline our goals for I/O loops:

Preserve backpressure propagation: blocked writes should slow down the local producer over the rx channel.
Retain concurrency: other work (e.g., cancellation, timeouts) should make progress while reads/writes are pending. Reads and writes should not block each other.

Backpressure propagation #

To address the first issue we could flip the perspective: instead of eagerly pulling from the rx channel, wait until the writer proves it has capacity. Some async primitives support this. For example, tokio::sync::mpsc::Sender offers reserve(), which awaits for a slot to become available to write. When the permission is granted, we can dequeue the message from our rx channel. But one important note here is that mpsc is slot-based, and not byte-sized, which is not what we usually need when working with I/O streams.

Prepare environment for experiments

Mon, 01 Jan 0001 00:00:00 +0000

Prepare environment for experiments #

Last updated: Oct 2025

Before starting, I want to be on the same page with the reader so that any example or code snippet can be executed, compiled, and checked. Therefore we need a modern GNU/Linux installation to play with code and kernel.

If you are using Windows or Mac OS, I would suggest installing Vagrant with Virtual Box. For the GNU/Linux distributive, I’d like to use Arch Linux. Arch is a good example of an actual modern version of the GNU/Linux system (BTW, I use Arch Linux). It supports the latest kernels, systemd and cgroup v2.

What is a stub resolver?

Mon, 01 Jan 0001 00:00:00 +0000

1. What is a stub resolver? #

Last updated: Oct 2025

First of all, let’s shed some light on what a stub resolver is.

Whenever someone begins talking about hostname resolution issues or nameserver changes, the first thing most people think of is the /etc/resolv.conf configuration file. Indeed, /etc/resolv.conf is a core and fundamental part of the local resolver system, and we will discuss it in detail later in this series, including how it’s managed on modern GNU/Linux distributions with systemd. However, it’s far from being the only component involved in converting a hostname string into a list of IP addresses. Often, other lesser-known parts of the system may cause unpredictable behaviors and have their own unique features and limitations.

History: gethostbyname() and old good friends

Mon, 01 Jan 0001 00:00:00 +0000

2. History: `gethostbyname()` and old good friends #

Last updated: Oct 2025

Please do not use any of the code snippets from this chapter in your projects. They are provided solely for historical and educational purposes. Instead, you should use getaddrinfo().

The gethostbyname (man 3 gethostbyname) function first appeared in the 1980s and has been a part of the networking landscape ever since. Despite its obsoletion, some programs still use it. It was deprecated in POSIX.1-2001, over two decades ago, due to its internal design limitations and limited functionality. However, for a long time, it was the preferred and standardized helper function for resolving a domain name into a list of IP addresses.

Pipes

Mon, 01 Jan 0001 00:00:00 +0000

Pipes #

Last updated: Oct 2025

Contents

The pipe is a neat feature of the Linux kernel that allows us to build one-directional communication channels between related processes (often a parent and a child).

Tokio I/O patterns

Mon, 01 Jan 0001 00:00:00 +0000

Tokio I/O Patterns #

Last updated: Oct 2025

Contents

Now let’s look at the possible solutions and weigh their pros and cons.

TCP split stream #

The first obvious solution, suggested by the documentation, is to split the TCP stream into two parts: a reader and a writer.

Essential Linux Page Cache theory

Mon, 01 Jan 0001 00:00:00 +0000

Essential Page Cache theory #

Last updated: Oct 2025

Contents

First of all, let’s start with a bunch of reasonable questions about Page Cache:

What is the Linux Page Cache?
What problems does it solve?
Why do we call it «Page» Cache ?

In essence, the Page Cache is a part of the Virtual File System (VFS) whose primary purpose, as you can guess, is improving the IO latency of read and write operations. A write-back cache algorithm is a core building block of the Page Cache.

getaddrinfo() and POSIX spec

Mon, 01 Jan 0001 00:00:00 +0000

3. `getaddrinfo()` and POSIX spec #

Last updated: Oct 2025

Contents

Thus, instead of the deprecated gethostbyname(), getaddrinfo() should be used within libc. The getaddrinfo() function is a POSIX-standardized function and is defined in RFC 3943. It is IP version agnostic and returns data structures that can be easily reused in subsequent socket API calls (such as socket(), connect(), sendto()).

First of all, if you have a codebase that uses gethostbyname() and you are looking to migrate to the modern getaddrinfo(), I have bad news: it’s not a drop-in replacement. You need to understand the new data structures, logic and flags.

Process groups, jobs and sessions

Mon, 01 Jan 0001 00:00:00 +0000

Process groups, jobs and sessions #

Last updated: Oct 2025

Contents

A new process group is created every time we execute a command or a pipeline of commands in a shell. Inside a shell, a process group is usually called a job. In its turn, each process group belongs to a session. Linux kernel provides a two-level hierarchy for all running processes (look at figure 3 below). As such, a process group is a set of processes, and a session is a set of related process groups. Another important limitation is that a process group and its members can be members of a single session.

getaddrinfo() from glibc

Mon, 01 Jan 0001 00:00:00 +0000

4. `getaddrinfo()` from `glibc` #

Last updated: Oct 2025

Contents

The standard in POSIX describes only the behavior and interface of the getaddrinfo() function. However, the actual implementation can vary between different frameworks. In this chapter, we will examine the internals of the getaddrinfo() implementation from glibc version 2.39. In the GNU/Linux world the glibc remains the default C library for the overwhelming majority of systems.

Terminals and pseudoterminals

Mon, 01 Jan 0001 00:00:00 +0000

Terminals and pseudoterminals #

Last updated: Oct 2025

Contents

Terminals come to us from the history of UNIX systems. Basically, terminals provided an API for the console utils (physical ones!) to generalize interaction with users. It includes ways of reading input and writing to it in two modes:

the canonical mode (default) – input is buffered line by line and read into after a new line char \n occurs;
the noncanonical mode – an application can read terminal input a character at a time. For example vi, emacs and less use this mode.

Nowadays, with the widespread use of rich graphical UIs, the significance of the terminals are lesser than it was, but still, we use this protocol implicitly every time we start an ssh connection.

getaddrinfo() from musl libc

Mon, 01 Jan 0001 00:00:00 +0000

5. `getaddrinfo()` from `musl` `libc` #

Last updated: Oct 2025

musl libc is a lightweight, fast, and simple implementation of the standard C library (libc) that aims for efficiency, standards compliance, and security.

It gained popularity following its extensive use in Alpine Linux, a security-oriented, lightweight Linux distribution often used as a base image for Docker containers.

However, it is crucial for us to understand that musl libc incorporates a completely new resolver code that behaves differently in certain situations. The most significant differences include:

Page Cache and basic file operations

Mon, 01 Jan 0001 00:00:00 +0000

Page Cache and basic file operations #

Last updated: Oct 2025

Contents

Now it’s time to roll up our sleeves and get started with some practical examples. By the end of this chapter, you will know how to interact with Page Cache and which tools you can use.

Dual-Stack applications

Mon, 01 Jan 0001 00:00:00 +0000

6. Dual-Stack applications #

Last updated: Oct 2025

Contents

Let’s now focus on dual-stack programs, which support both IPv4 and IPv6. Here are some critical questions to consider:

For server code:

How can we easily listen on all IPv4 and all IPv6 addresses? Do we need separate listeners for each?
Are there any tools or helpers available to manage multiple listeners?

For client code:

Async non-blocking resolvers in C

Mon, 01 Jan 0001 00:00:00 +0000

7. Async non-blocking resolvers in `C` #

Last updated: Oct 2025

Contents

Now that we’ve covered the essential theory, let’s explore alternative stub resolver libraries and frameworks for the C language. Other languages will be discussed next, but don’t skip this chapter, as it contains foundational information that will be referenced later.

7.1 `getaddrinfo_a()` #

getaddrinfo_a (man 7 getaddrinfo_a) is an asynchronous version of getaddrinfo() but with some limitations: results can be collected by polling or notified by a signal.

Page Cache eviction and page reclaim

Mon, 01 Jan 0001 00:00:00 +0000

Page Cache eviction and page reclaim #

Last updated: Oct 2025

Contents

So far, we have talked about adding data to Page Cache by reading and writing files, checking the existence of files in the cache, and flushing the cache content manually. But the most crucial part of any cache system is its eviction policy, or regarding Linux Page Cache, it’s also the memory page reclaim policy. Like any other cache, Linux Page Cache continuously monitors the last used pages and makes decisions about which pages should be deleted and which should be kept in the cache.

Stub resolvers in languages

Mon, 01 Jan 0001 00:00:00 +0000

8. Stub resolvers in languages #

Last updated: Oct 2025

Contents

Let’s now take a look at other popular languages and understand the capabilities, features and options they provide in the context of resolvers.

Dual-stack software examples

Mon, 01 Jan 0001 00:00:00 +0000

9. Dual-stack software examples #

Last updated: Oct 2025

Contents

9.1 Nginx #

Nginx treats the hostname as a set of distinct entries rather than multiple paths to the same host. From the upstream module doc:

A domain name that resolves to several IP addresses defines multiple servers at once.

On start Nginx resolves all hostnames using its static resolver with getaddrinfo() and the AF_UNSPEC and the AI_ADDRCONFIG flags.

More about mmap() file access

Mon, 01 Jan 0001 00:00:00 +0000

More about `mmap()` file access #

Last updated: Oct 2025

Contents

Before we start the cgroup chapter, where I’m showing how to leverage memory and IO limits in order to control Page Cache eviction and improve the reliability of services, I want to delve a bit deeper into mmap() syscall. We need to understand what is happening under the hood and shed more light on the reading and writing process with mmap().

systemd-resolved

Mon, 01 Jan 0001 00:00:00 +0000

10. systemd-resolved #

Last updated: Oct 2025

Contents

10.1 Managing `/etc/resolv.conf` content #

The main issue with /etc/resolv.conf is managing it in modern distributions, which can have multiple sources of nameserver and search domain information due to multiple interfaces (both real and virtual, such as VPN tunnels) with concurrent DHCP clients.

cgroup v2 and Page Cache

Mon, 01 Jan 0001 00:00:00 +0000

cgroup v2 and Page Cache #

Last updated: Oct 2025

Contents

The cgroup subsystem is the way to distribute and limit system resources fairly. It organizes all data in a hierarchy where the leaf nodes depend on their parents and inherit their settings. In addition, the cgroup provides a lot of helpful resource counters and statistics.

Querying Nameservers on dual-stack hosts

Mon, 01 Jan 0001 00:00:00 +0000

11. Querying Nameservers on dual-stack hosts #

Last updated: Oct 2025

The already seen RFC 8305 Happy Eyeballs Version 2: Better Connectivity Using Concurrency force the same preference for IPv6 name servers as it does for establishing new connections:

If multiple DNS server addresses are configured for the current network, the client may have the option of sending its DNS queries over IPv4 or IPv6. In keeping with the Happy Eyeballs approach, queries SHOULD be sent over IPv6 first (note that this is not referring to the sending of AAAA or A queries, but rather the address of the DNS server itself and IP version used to transport DNS messages). If DNS queries sent to the IPv6 address do not receive responses, that address may be marked as penalized and queries can be sent to other DNS server addresses.

The Present and the future of resolvers and DNS related features

Mon, 01 Jan 0001 00:00:00 +0000

Last updated: Oct 2025

Contents

Let me briefly review some important features and topics related to DNS, stub resolvers, and dual-stack applications. While these are beyond the scope of this series, they are worth mentioning.

Troubleshooting tools for resolvers and DNS

Mon, 01 Jan 0001 00:00:00 +0000

Tools for troubleshooting in one place #

Last updated: Oct 2025

Let me reiterate and consolidate all the tools that can be used to troubleshoot applications and systems when the stub resolver is under suspicion.

• `getent` #

man 1 getent

When you need to query hostname via all NSS modules:

$ getent host microsoft.com
$ getent ahost microsoft.com

• `tcpdump` #

To dump in a user friendly format all requests to 53 port:

Unique set size and working set size

Mon, 01 Jan 0001 00:00:00 +0000

How much memory my program uses or the tale of working set size #

Last updated: Oct 2025

Contents

Currently, in the world of containers, auto-scaling, and on-demand clouds, it’s vital to understand the resource needs of services both in norman regular situations and under pressure near the software limits. But every time someone touches on the topic of memory usage, it becomes almost immediately unclear what and how to measure. RAM is a valuable and often expensive type of hardware. In some cases, its latency is even more important than disk latency. Therefore, the Linux kernel tries as hard as it can to optimize memory utilization, for instance by sharing the same pages among processes. In addition, the Linux Kernel has its Page Cache in order to improve storage IO speed by storing a subset of the disk data in memory. Page Cache not only, by its nature, performs implicit memory sharing, which usually confuses users, but also actively asynchronously works with the storage in the background. Thus, Page Cache brings even more complexity to the table of memory usage estimation.

Direct IO

Mon, 01 Jan 0001 00:00:00 +0000

Direct IO (DIO) (NOT READY) #

Last updated: Oct 2025

Contents

As usual, there is always an exception to any rule. And Page Cache is no different. So let’s talk about file reads and writes, which can ignore Page Cache content.

Why it’s good #

Some applications require low-level access to the storage subsystem and the linux kernel gives such a feature by providing O_DIRECT file open flag. This IO is called the Direct IO or DIO. A program, which opens a file with this flag, bypasses the kernel Page Cache completely and directly communicates with the VFS and the underlying filesystem.

Advanced Page Cache observability and troubleshooting tools

Mon, 01 Jan 0001 00:00:00 +0000

Advanced Page Cache observability and troubleshooting tools #

Last updated: Oct 2025

Contents

Let’s touch on some advanced tools we can use to perform low-level kernel tracing and debugging.

eBPF tools #

First of all, we can use eBPF tools. The [bcc]https://github.com/iovisor/bcc and bpftrace are your friends when you want to get some internal kernel information.

What every SRE should know on Viacheslav Biriukov

Async Rust gotcha: evolving tokio::select! code has sharp edges

TL;DR #

Intro #

Trixter: A Chaos Proxy for Simulating Network Faults

Async Rust with Tokio I/O Streams: Backpressure, Concurrency, and Ergonomics

Async Rust with Tokio I/O Streams: Backpressure, Concurrency, and Ergonomics #

GNU/Linux shell related internals

What every SRE should know about GNU/Linux shell related internals: file descriptors, pipes, terminals, user sessions, process groups and daemons #

Linux Page Cache for SRE

SRE deep dive into Linux Page Cache #

What every SRE should know about GNU/Linux resolvers and Dual-Stack applications

What every SRE should know about GNU/Linux resolvers and Dual-Stack applications #

File descriptor and open file description

File descriptor and open file description #

I/O loop

I/O loop #

Backpressure propagation #

Prepare environment for experiments

Prepare environment for experiments #

What is a stub resolver?

1. What is a stub resolver? #

History: gethostbyname() and old good friends

2. History: gethostbyname() and old good friends #

Pipes

Pipes #

Tokio I/O patterns

Tokio I/O Patterns #

TCP split stream #

Essential Linux Page Cache theory

Essential Page Cache theory #

getaddrinfo() and POSIX spec

3. getaddrinfo() and POSIX spec #

Process groups, jobs and sessions

Process groups, jobs and sessions #

getaddrinfo() from glibc

4. getaddrinfo() from glibc #

Terminals and pseudoterminals

Terminals and pseudoterminals #

getaddrinfo() from musl libc

5. getaddrinfo() from musl libc #

Page Cache and basic file operations

Page Cache and basic file operations #

Dual-Stack applications

6. Dual-Stack applications #

Async non-blocking resolvers in C

7. Async non-blocking resolvers in C #

7.1 getaddrinfo_a() #

Page Cache eviction and page reclaim

Page Cache eviction and page reclaim #

Stub resolvers in languages

8. Stub resolvers in languages #

Dual-stack software examples

9. Dual-stack software examples #

9.1 Nginx #

More about mmap() file access

More about mmap() file access #

systemd-resolved

10. systemd-resolved #

10.1 Managing /etc/resolv.conf content #

cgroup v2 and Page Cache

cgroup v2 and Page Cache #

Querying Nameservers on dual-stack hosts

11. Querying Nameservers on dual-stack hosts #

The Present and the future of resolvers and DNS related features

12. The Present and the future of resolvers and DNS related features #

Troubleshooting tools for resolvers and DNS

Tools for troubleshooting in one place #

• getent #

• tcpdump #

Unique set size and working set size

How much memory my program uses or the tale of working set size #

Direct IO

Direct IO (DIO) (NOT READY) #

Why it’s good #

Advanced Page Cache observability and troubleshooting tools

Advanced Page Cache observability and troubleshooting tools #

eBPF tools #

2. History: `gethostbyname()` and old good friends #

3. `getaddrinfo()` and POSIX spec #

4. `getaddrinfo()` from `glibc` #

5. `getaddrinfo()` from `musl` `libc` #

7. Async non-blocking resolvers in `C` #

7.1 `getaddrinfo_a()` #

More about `mmap()` file access #

10.1 Managing `/etc/resolv.conf` content #

• `getent` #

• `tcpdump` #