Reducing the io-sock startup time

Allocations for packet filtering (pf) states

If io-sock starts with the bootverbose option, you can observe that about 45% of the startup time is in the pf module. For example:

# io-sock -o bootverbose
# slog2info
Dec 31 19:06:21.214 io_sock.331784 main_buffer* 1 io-sock -o bootverbose
...
Dec 31 19:06:21.250 io_sock.331784 main_buffer 1 module_register_init: pf
Dec 31 19:06:21.285 io_sock.331784 main_buffer 1 pflog0: bpf attached
Dec 31 19:06:21.285 io_sock.331784 main_buffer 1 module_register_init: pfsync
...
Dec 31 19:06:21.293 io_sock.331784 main_buffer 1 io-sock startup complete

This additional time is the result of the default memory allocation for some large hash tables: a pf hash table size setting of 128K, and a source hash size that is one quarter the size of the pf hash table size (i.e., 32 K):

# sysctl net.pf
net.pf.rule_tag_hashsize: 128
net.pf.queue_tag_hashsize: 128
net.pf.states_hashsize: 131072
net.pf.source_nodes_hashsize: 32768
net.pf.request_maxcount: 65535

You can adjust these settings using tunables in the io-sock configuration file. Reducing their size speeds up io-sock startup time signficantly. In the following example, io-sock starts with a configuration file that sets the pf hash table size tunable and others to lower values:

$ io-sock -o config=/etc/io-sock.conf,bootverbose
$ slog2info
Dec 31 19:05:20.570 io_sock.397320 main_buffer* 1 io-sock -o config=/etc/io-sock.conf,bootverbose
...
Dec 31 19:05:20.571 io_sock.397320 main_buffer 1 net.pf.states_hashsize="8"
Dec 31 19:05:20.571 io_sock.397320 main_buffer 1 net.pf.source_nodes_hashsize="2"
...
Dec 31 19:05:20.605 io_sock.397320 main_buffer 1 module_register_init: pf
Dec 31 19:05:20.608 io_sock.397320 main_buffer 1 pflog0: bpf attached
Dec 31 19:05:20.608 io_sock.397320 main_buffer 1 module_register_init: pfsync
...
Dec 31 19:05:20.616 io_sock.397320 main_buffer 1 io-sock startup complete

Considerations

These adjustments impact the scalability and performance of pf. The drastic reduction (to 8 and 2) should only be considered when pf isn't being used. If you require both pf and a fast startup, you need to determine the appropriate values that can enable both.

For details on packet filtering, go to the Packet Filtering chapter in the High-Performance Networking Stack (io-sock) User's Guide and the pf entry in the Utilities Reference.

Allocations for driver buffers

Memory for driver descriptors and buffers needs to be both allocated by the kernel and have its physical address provided by the kernel. Because these are expensive operations, io-sock caches these allocations and in the steady state should not be requesting more from the kernel. At startup, this cache is empty, so any memory requested by the driver involves two calls to the kernel.

The calls are made in bulk with a default of 64 items at once, but you can use a tunable to adjust this value. For example:

# sysctl -d qnx.paddr_items
qnx.paddr_items: Minimum items on a paddr slab

In addition to increasing the number of items allocated at once, reducing the number of descriptors allocated by the driver can also reduce the number of calls to the kernel. Often this reduction is done using a driver-specific sysctl.

For details on sysctl, go to the sysctl entry in the Utilities Reference.

Considerations

Increasing the number of items allocated at once may increase the amount of unused memory allocated to io-sock, which increases its memory footprint.

Reducing the number of descriptors available to a driver may reduce its throughput.

Driver hardware loading time

After io-sock starts, it detaches from the controlling terminal and runs in the background as a system daemon. After it has switched to the background, its own internal startup is complete and its resource manager is active, but any network drivers may still be loading. Network interface hardware often needs a significant amount of time for resets, PLLs to settle and lock, PHYs to negotiate link capabilities, and so on. When an interface is ready, its presence is announced on the routing socket (via a RTM_IFANNOUNCE message, and RTM_IFINFO for the link).

For best performance, start io-sock without forcing it to the background and then watch for interfaces to appear either via the routing socket API or using ifwatchd. The ifwatchd utility uses the routing socket API to watch for interfaces. Forcing io-sock to the background early requires polling with waitfor to determine when its resource manager is ready and is slower overall.

Note that if you are using DHCP, then dhcpcd also uses the routing socket API and can be started directly after io-sock has completed its startup, without using ifwatchd.

For details on routing socket APIs, go to the FreeBSD route documentation (https://www.freebsd.org/cgi/man.cgi?query=route&sektion=8&manpath=FreeBSD+13.4-RELEASE+and+Ports.) For more information on the networking utilities, go to the ifwatchd and dhcpcd entries in the Utilities Reference.