Xenion Pty Ltd - Australian VPS, Managed and Co-lo Hosting

Squid / Lusca tuning on FreeBSD

Last Modified: 27-Apr-2009; 16:00 WST; adrian

This document covers basic tuning of a FreeBSD server for Squid/Lusca.

Background

The default FreeBSD kernel parameters are not set correctly to run a high-throughput HTTP proxy-cache. This document will outline some information regardining kernel and Squid/Lusca tuning.

32-bit versus 64-bit

Disk system issues

Filesystem selection

As of the last update time, ZFS under FreeBSD (or Solaris) has not yet been benchmarked by Xenion for use with Squid/Lusca. It is quite possible that performance will improve over FreeBSD FFS but please keep in mind that ZFS is -very- memory intensive and thus a server may need double the memory needed compared to FFS operation.

Filesystem tuning

Remember to mount FFS partitions with "noatime"
Enable softupdates

Cache disk store selection

Squid-2 and Lusca both provide a cyclic filestore which is optimised for small objects. This filestore, known as COSS, should be used for objects smaller than ~ 256 kilobytes. This can run either to a raw disk or on top of an existing filesystem.

FreeBSD-7 supports full kernel supported threading (as does FreeBSD-6.x) and as such the "aufs" store model should be used by default. LUSCA_HEAD has some tuning to make AUFS perform better under FreeBSD than the Squid defaults.

Instead of dedicating one disk per cache_dir, consider creating multiple cache_dirs per physical disk. Use separate cache_dir's for small, medium and huge objects. This allows you to control how many of what kind of object you wish to store and avoid wasting space on small objects which you will never again use.

Kernel Network tuning

The following stuff should be put in /etc/sysctl.conf to allow FreeBSD to scale to many concurrent connections:

# Don't limit connection rates
net.inet.icmp.icmplim=0
net.inet.icmp.icmplim_output=0

# This breaks the TCP RFC but it is needed when the server is handling
# a very high request rate and you run out of ports.
net.inet.tcp.msl=3000

# Tune these to allow more ephemeral ports for outbound connections.
# (Tuning this isn't required to allow more incoming connections; the local
# port number is always one of the proxy listening ports.)
#
# The following are the defaults.
#
net.inet.ip.portrange.hilast: 65535
net.inet.ip.portrange.hifirst: 49152
#
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.first: 49152
#

# In -this- example, the send/receive socket buffer sizes are kept
# low to allow for tens of thousands of concurrent connections on
# relatively memory starved machines (< 4GB RAM) - this has a noticable
# impact on TCP throughput. Increase these if needs be (and enable
# TCP window scaling and other related options) but make sure you do the
# math to know how much RAM you need for the number of concurrent sockets
# you wish to run!
net.inet.tcp.sendspace=8192
net.inet.tcp.recvspace=8192

# Bump up the global and per-process FD limits
kern.maxfilesperproc=65536
kern.maxfiles=262144
kern.ipc.maxsockets=32768

# The default mbuf cluster size is determined by your available RAM - it
# is almost certainly not enough.
kern.ipc.nmbclusters=131072
# Increase incoming socket queue length - important for high connection
# throughput testing (say, > 5000 req/sec.) Otherwise, just set it to
# 1024.
kern.ipc.somaxconn=10240

Tuning network throughput

Increasing TCP network throughput always involves increasing the memory footprint of your application. A rough rule of thumb is to assume the total memory needed for socket buffers is (numsockets * socket buffer size * 2) (for userland and kernel buffering) even though RAM will not always be 100% allocated.

An important thing to keep in mind here is the implications of configuring Squid/Lusca with both a large TCP socket buffer size and a large number of sockets. Squid-2 and Lusca can scale to 30,000 concurrent sessions and beyond but this involves hundreds of megabytes of socket buffers and potentially another hundred or so megabytes of user process RAM to track the connection state.

This will tax the RAM available on a 32 bit UNIX installation which typically limits kernel virtual memory space to 1-2GB.

Additionally, Lusca allows the socket buffers on client connections to be changed from the system and compiled defaults. This allows the administrator to use a larger socket buffer for upstream connections but smaller socket buffers for closer client connections. (This holds true for most forward proxy deployments but not necessarily for reverse proxy deployments.) The configuration option is "client_socksize" and is available in the LUSCA_HEAD branch.

Increasing kernel limits on process sizes

The 32 bit FreeBSD install limits the process data size to 512 megabytes by default. This needs to be modified to support large Squid processes (up to 2gigabytes on 32 bit machines before paging to swapspace must occur.)

This can be achieved by modifying /boot/loader.conf to set boot-time limits on process sizes:

kern.maxdsiz="2048M"
kern.dfldsiz="768M"
kern.maxssiz="256MB"

How much RAM is required?

Memory Cache issues

The FreeBSD malloc seems to round up >4k allocations to the nearest page boundary. This means that the ~ 4100 byte allocation made for the memory cache gets rounded up to ~8k on FreeBSD 7 and above (jemalloc). Fixing this requires some further storage and disk work (or, as a cheap hack, restore the separate buffer and memory copy which used to exist in the disk swapout code path.) As such, please keep in mind that for now, memory cache usage will be quite inefficient under FreeBSD-7. Disk caching is still quite efficient (ie, using the operating system buffer/VM caching to cache disk accesses) and works very well for large file caching (up to and past 1 gbit/sec in testing.)

Xenion

Communications