7. Async non-blocking resolvers in C #

Now that we’ve covered the essential theory, let’s explore alternative stub resolver libraries and frameworks for the C language. Other languages will be discussed next, but don’t skip this chapter, as it contains foundational information that will be referenced later.

7.1 getaddrinfo_a() #

getaddrinfo_a (man 7 getaddrinfo_a) is an asynchronous version of getaddrinfo() but with some limitations: results can be collected by polling or notified by a signal.

Internally it creates a thread for each request using a thread pool:

 /* Now try to start a thread. If we fail, no big deal,
because we know that there is at least one thread (us)
that is working on lookup operations. */
if (__pthread_create (&thid, &attr, handle_requests, NULL) == 0)
	++nthreads;

and running original getaddrinfo() inside:

	req->__return = getaddrinfo (req->ar_name, req->ar_service,
	req->ar_request, &req->ar_result);

The notification mechanism is not perfectly suitable for the present-day high performance application development. Its callbacks are implemented either via POSIX signal handler, or by creating a new waiting thread and running a caller’s callback function on it.

The only user I could find on github is pgbouncer (lightweight connection pooler for PostgreSQL):

7.2 c-ares library #

It should now be clear that neither getaddrinfo() nor getaddrinfo_a() can meet all the needs of modern software development, where asynchronous and non-blocking code is an essential building block. Such code enables great performance by wisely utilizing system resources. This is why the curl developers forked the ares library some time ago and continue to maintain an advanced stub resolver framework called c-ares.

c-ares is a modern DNS (stub) resolver library, written in C. It provides interfaces for asynchronous queries while trying to abstract the intricacies of the underlying DNS protocol. It was originally intended for applications which need to perform DNS queries without blocking, or need to perform multiple DNS queries in parallel.

c-ares likely has the most features of any alternative stub resolver library I’ve encountered. The list of supported RFCs is massive and comprehensive.

Unlike other resolvers, c-ares doesn’t use signals for notifications nor does it spawn threads to resolve domain names. Instead, it leverages non-blocking sockets and the epoll (man 7 epoll) syscall on Linux. This makes it an ideal candidate for integration into other epoll-based modern programs.

Some of the most significant users of c-ares include curl, NodeJS, and Envoy.

c-ares, like getaddrinfo(), uses getenv(), which could potentially lead to a segmentation fault in a multithreaded environment if there is a concurrent setenv() call."

7.2.1 Essentials #

The configuration step happening during the initialization process of a communications channel (ares_init_options) allows to change the basic resolver behavior. Here are some important options that could divergence from the glibc getaddrinfo():

• ARES_FLAG_NOSEARCH – don’t use the search domain from /etc/resolv.conf and always ask a nameserver as is.
• ARES_OPT_TIMEOUTMS, ARES_OPT_TRIES, ARES_OPT_DOMAINS which overwrite the settings from /etc/resolv.conf.
• ARES_OPT_RESOLVCONF, ARES_OPT_HOSTS_FILE – alternative paths for /etc/resolv.conf and /etc/hosts.
• ARES_OPT_SERVER_FAILOVER – an equivalent of rotate option from /etc/resolv.conf.

This is just a small compatibility gist with /etc/resolv.conf and libc getaddrinfo(), but the full list of options contains much more settings.

The resolve functions are:

• ares_gethostbyname() – an alternative for gethostbyname()(it’s not deprecated);
• ares_getaddrinfo() – an equivalent for getaddrinfo().

Version 1.28.0 of c-ares made a significant change in how it reads system configuration files. This is important to mention because Ubuntu 24.04 LTS includes c-ares version 1.27 in its repository. Therefore, in this chapter, we will be using version 1.27.

The default c-ares initialization process for resolving order is a bit different from the glibc getaddrinfo().

① – at first it reads /etc/nsswitch.conf to determine the order of sources. It only supports files, dns and resolve (systemd-resolved, we will touch it later in the series).

② – if /etc/host.conf exists, overwrite the order with its content. This is important to know because if your distribution of GNU/Linux still has the /etc/host.conf (man 5 host.conf) file it could cost you some time to troubleshoot this behavior.

③ – if /etc/svc.conf exists, overwrite the order with its content.

This behavior changed in version 1.28, where the old deprecated system files /etc/host.conf and /etc/svc.conf were dropped."

Let’s finally write an example with c-ares:

/* Example from https://c-ares.org/docs.html */

#include <stdio.h>
#include <string.h>
#include <ares.h>

/* Callback that is called when DNS query is finished */
static void addrinfo_cb(void *arg, int status, int timeouts,
                        struct ares_addrinfo *result)
{
  (void)arg; /* Example does not use user context */
  printf("Result: %s, timeouts: %d\n", ares_strerror(status), timeouts);

  if (result) {
    struct ares_addrinfo_node *node;
    for (node = result->nodes; node != NULL; node = node->ai_next) {
      char        addr_buf[64] = "";
      const void *ptr          = NULL;
      if (node->ai_family == AF_INET) {
        const struct sockaddr_in *in_addr =
          (const struct sockaddr_in *)((void *)node->ai_addr);
        ptr = &in_addr->sin_addr;
      } else if (node->ai_family == AF_INET6) {
        const struct sockaddr_in6 *in_addr =
          (const struct sockaddr_in6 *)((void *)node->ai_addr);
        ptr = &in_addr->sin6_addr;
      } else {
        continue;
      }
      ares_inet_ntop(node->ai_family, ptr, addr_buf, sizeof(addr_buf));
      printf("Addr: %s\n", addr_buf);
    }
  }
  ares_freeaddrinfo(result);
}

int main(int argc, char **argv)
{
  ares_channel_t            *channel = NULL;
  struct ares_options        options;
  int                        optmask = 0;
  struct ares_addrinfo_hints hints;

  if (argc != 2) {
    printf("Usage: %s domain\n", argv[0]);
    return 1;
  }

  /* Initialize library */
  ares_library_init(ARES_LIB_INIT_ALL);

  if (!ares_threadsafety()) {
    printf("c-ares not compiled with thread support\n");
    return 1;
  }

  /* Enable event thread so we don't have to monitor file descriptors */
  memset(&options, 0, sizeof(options));
  optmask      |= ARES_OPT_EVENT_THREAD;
  options.evsys = ARES_EVSYS_DEFAULT;

  /* Initialize channel to run queries, a single channel can accept unlimited
   * queries */
  if (ares_init_options(&channel, &options, optmask) != ARES_SUCCESS) {
    printf("c-ares initialization issue\n");
    return 1;
  }

  /* Perform an IPv4 and IPv6 request for the provided domain name */
  memset(&hints, 0, sizeof(hints));
  hints.ai_family = AF_UNSPEC;
  hints.ai_flags  = ARES_AI_CANONNAME;
  ares_getaddrinfo(channel, argv[1], NULL, &hints, addrinfo_cb,
                   NULL /* user context not specified */);

  /* Wait until no more requests are left to be processed */
  ares_queue_wait_empty(channel, -1);

  /* Cleanup */
  ares_destroy(channel);

  ares_library_cleanup();
  return 0;
}

Install dependencies:

$ sudo apt install libc-ares-dev
$ # I use aarch64 if you're not please change it
$ gcc -L/usr/lib/aarch64-linux-gnu/ -I/usr/lib ./resolver.c -o resolver -lcares

If we run it with strace, we can see all the above config files in the output:

$ strace -e openat ./resolver micrisoft.com

openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libcares.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/host.conf", O_RDONLY) = 3
openat(AT_FDCWD, "/etc/svc.conf", O_RDONLY) = -1 ENOENT (No such file or directory)

As you can see my cloud image of Ubuntu 24.04 has /etc/host.conf file:

$ cat /etc/host.conf

# The "order" line is only used by old versions of the C library.
order files bind
multi on

Another important observation from the strace output is there is no /etc/gai.conf. So there is no way to change the sorting rules by providing a custom policy table.

7.2.1 Dual stack application #

Due to the subtle logic of AI_ADDRCONFIG heuristics,the c-ares developers decided not to bring it.

Thus if you set AF_UNSPEC flag, two DNS requests for A and AAAA will be sended and two address families will be returned (if of course there is anything to return).

It is worth mentioning, though, that thanks to RFC 6724’s sorting algorithm—which includes Rule 2: Prefer matching scope in Section 6—the return order will prefer IPv4 over IPv6 if there are no global scope IPv6 source addresses on the device.

With IPv6 global scope address on a machine:

$ ./resolver microsoft.com

Result: Successful completion, timeouts: 0

Addr: 2603:1030:b:3::152
Addr: 2603:1030:20e:3::23c
Addr: 2603:1030:c02:8::14
Addr: 2603:1020:201:10::10f
Addr: 2603:1010:3:3::5b
Addr: 20.76.201.171
Addr: 20.231.239.246
Addr: 20.70.246.20
Addr: 20.112.250.133
Addr: 20.236.44.162

And without:

$ ./resolver microsoft.com

Result: Successful completion, timeouts: 0

Addr: 20.70.246.20
Addr: 20.236.44.162
Addr: 20.231.239.246
Addr: 20.76.201.171
Addr: 20.112.250.133
Addr: 2603:1030:c02:8::14
Addr: 2603:1010:3:3::5b
Addr: 2603:1030:b:3::152
Addr: 2603:1020:201:10::10f
Addr: 2603:1030:20e:3::23c

Sorting code:

  /* Rule 2: Prefer matching scope. */
  scope_src1 = ARES_IPV6_ADDR_SCOPE_NODELOCAL;
  if (a1->has_src_addr) {
    scope_src1 = get_scope(&a1->src_addr.sa);
  }
  scope_dst1   = get_scope(a1->ai->ai_addr);
  scope_match1 = (scope_src1 == scope_dst1);

  scope_src2 = ARES_IPV6_ADDR_SCOPE_NODELOCAL;
  if (a2->has_src_addr) {
    scope_src2 = get_scope(&a2->src_addr.sa);
  }
  scope_dst2   = get_scope(a2->ai->ai_addr);
  scope_match2 = (scope_src2 == scope_dst2);

  if (scope_match1 != scope_match2) {
    return scope_match2 - scope_match1;
  }

If we run the code under strace with network trace, we see how it performs the same socket calls as getaddrinfo() does in order to retrieve a source address for the destination:

[pid 209162] socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDP) = 7
[pid 209162] connect(7, {sa_family=AF_INET6, sin6_port=htons(0), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "2603:1030:b:3::152", &sin6_addr), sin6_scope_id=0}, 28) = 0
[pid 209162] getsockname(7, {sa_family=AF_INET6, sin6_port=htons(54532), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "fec0::5055:55ff:fe8e:3d07", &sin6_addr), sin6_scope_id=0}, [28]) = 0
[pid 209162] close(7)

But notice, if you disable sorting by setting ARES_AI_NOSORT flag in our above code by change hints.ai_flags to:

hints.ai_flags  = ARES_AI_NOSORT;

it always returns IPv4 addresses first. However, I’m not sure if this order is guaranteed or if it’s due to the order of DNS requests, with the A request being sent before the AAAA. Typically, without network reordering, the answers also arrive in this order.

$ ./resolver microsoft.com

Result: Successful completion, timeouts: 0

Addr: 20.231.239.246
Addr: 20.70.246.20
Addr: 20.76.201.171
Addr: 20.112.250.133
Addr: 20.236.44.162
Addr: 2603:1030:c02:8::14
Addr: 2603:1020:201:10::10f
Addr: 2603:1010:3:3::5b
Addr: 2603:1030:b:3::152
Addr: 2603:1030:20e:3::23c

Without a fully functional alternative to the AI_ADDRCONFIG flag, using the AF_UNSPEC family can result in unnecessary work if there is no IPv6 routing on the system, as it will still issue AAAA DNS requests. However, these calls could be amortized by the built-in c-ares’ cache. If this isn’t sufficient, you have at least two options, though they add complexity:

1. Write your own IPv4/IPv6 routable addresses/stack detection.

2. Resolve both families just in case but make per address family DNS calls asynchronously. You can add deadline logic and use the fastest answer with the Happy Eyeballs algorithm. This approach is suggested in RFC 8305 Happy Eyeballs Version 2: Better Connectivity Using Concurrency:

   Implementations SHOULD NOT wait for both families of answers to return before attempting connection establishment. If one query fails to return or takes significantly longer to return, waiting for the second address family can significantly delay the connection establishment of the first one. Therefore, the client SHOULD treat DNS resolution as asynchronous. Note that if the platform does not offer an asynchronous DNS API, this behavior can be simulated by making two separate synchronous queries on different threads, one per address family.