FreeBSD: All mbuf clusters exhausted

LOADER TUNABLES
Some aspects of the system behavior may not be tunable at runtime because
memory allocations they perform must occur early in the boot process. To
change loader tunables, you must set their values in loader.conf(5) and
reboot the system.

kern.maxusers controls the scaling of a number of static system tables,
including defaults for the maximum number of open files, sizing of net-
work memory resources, etc. As of FreeBSD 4.5, kern.maxusers is automat-
ically sized at boot based on the amount of memory available in the sys-
tem, and may be determined at run-time by inspecting the value of the
read-only kern.maxusers sysctl. Some sites will require larger or
smaller values of kern.maxusers and may set it as a loader tunable; val-
ues of 64, 128, and 256 are not uncommon. We do not recommend going
above 256 unless you need a huge number of file descriptors; many of the
tunable values set to their defaults by kern.maxusers may be individually
overridden at boot-time or run-time as described elsewhere in this docu-
ment. Systems older than FreeBSD 4.4 must set this value via the kernel
config(8) option maxusers instead.

kern.ipc.nmbclusters may be adjusted to increase the number of network
mbufs the system is willing to allocate. Each cluster represents approx-
imately 2K of memory, so a value of 1024 represents 2M of kernel memory
reserved for network buffers. You can do a simple calculation to figure
out how many you need. If you have a web server which maxes out at 1000
simultaneous connections, and each connection eats a 16K receive and 16K
send buffer, you need approximately 32MB worth of network buffers to deal
with it. A good rule of thumb is to multiply by 2, so 32MBx2 = 64MB/2K =
32768. So for this case you would want to set kern.ipc.nmbclusters to
32768. We recommend values between 1024 and 4096 for machines with mod-
erates amount of memory, and between 4096 and 32768 for machines with
greater amounts of memory. Under no circumstances should you specify an
arbitrarily high value for this parameter, it could lead to a boot-time
crash. The -m option to netstat(1) may be used to observe network clus-
ter use. Older versions of FreeBSD do not have this tunable and require
that the kernel config(8) option NMBCLUSTERS be set instead.

More and more programs are using the sendfile(2) system call to transmit
files over the network. The kern.ipc.nsfbufs sysctl controls the number
of filesystem buffers sendfile(2) is allowed to use to perform its work.
This parameter nominally scales with kern.maxusers so you should not need
to modify this parameter except under extreme circumstances.