Optimizing NGINX and PHP-fpm for high traffic sites

1. TCP Sockets vs UNIX domain sockets

UNIX domain sockets offer slightly better performance than TCP sockets over loopback interface (less copying of data, fewer context switches).

upstream backend {
  # UNIX domain sockets
  server unix:/var/run/fastcgi.sock;

  # TCP sockets
  # server 127.0.0.1:8080; }

If you need to support more than 1,000 connections per server, use TCP sockets - they scale much better.

2. Adjust Worker Processes

Modern hardware is multiprocessor and NGINX can leverage multiple physical or virtual processors.

In most cases your web server machine will not be configured to handle multiple workloads (like providing services as a Web Server and a Print Server at the same time) so you will want to configure NGINX to use all the available processors since NGINX worker processes are not multi-threaded.

You can determine how many processors your machine has by running:

On Linux - 

cat /proc/cpuinfo | grep processor

On FreeBSD - 

sysctl dev.cpu | grep location

Set the worker_processes in your nginx.conf file to the number of cores your machine has.

While you're at it, increase the number of worker_connections (how many connections each core should handle) and set "multi_accept" to ON, as well as "epoll" if you're on Linux:

 # We have 16 cores worker_processes 16;
# connections per worker events {
  worker_connections 4096;
  multi_accept on;
}

3. Setup upstream load balancing

In our experience, multiple upstream backends on the same machine, produce higher throughout than a single one.

For example, if you're looking to support 1,000 max children, divide that number across two backends, letting each handle 500 children:

upstream backend {
  server unix:/var/run/php5-fpm.sock1 weight=100 max_fails=5 fail_timeout=5;
  server unix:/var/run/php5-fpm.sock2 weight=100 max_fails=5 fail_timeout=5;
}

Here are the two pools from php-fpm.conf:

  <section name="pool">

      <value name="name">www1</value>
      <value name="listen_address">/var/run/php5-fpm.sock1</value>

      <value name="listen_options">
        <value name="backlog">-1</value>
        <value name="owner"></value>
        <value name="group"></value>
        <value name="mode">0666</value>
      </value>

      <value name="user">www</value>
      <value name="group">www</value>

      <value name="pm">
        <value name="style">static</value>
        <value name="max_children">500</value>
      </value>

      <value name="rlimit_files">50000</value>
      <value name="rlimit_core">0</value>
      <value name="request_slowlog_timeout">20s</value>
      <value name="slowlog">/var/log/php-slow.log</value>
      <value name="chroot"></value>
      <value name="chdir"></value>
      <value name="catch_workers_output">no</value>
      <value name="max_requests">5000</value>
      <value name="allowed_clients">127.0.0.1</value>

      <value name="environment">
        <value name="HOSTNAME">$HOSTNAME</value>
        <value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
        <value name="TMP">/usr/tmp</value>
        <value name="TMPDIR">/usr/tmp</value>
        <value name="TEMP">/usr/tmp</value>
        <value name="OSTYPE">$OSTYPE</value>
        <value name="MACHTYPE">$MACHTYPE</value>
        <value name="MALLOC_CHECK_">2</value>
      </value>

    </section>

  <section name="pool">

      <value name="name">www2</value>
      <value name="listen_address">/var/run/php5-fpm.sock2</value>
     
      <value name="listen_options">
        <value name="backlog">-1</value>
        <value name="owner"></value>
        <value name="group"></value>
        <value name="mode">0666</value>
      </value>

      <value name="user">www</value>
      <value name="group">www</value>

      <value name="pm">
        <value name="style">static</value>
        <value name="max_children">500</value>
      </value>

      <value name="rlimit_files">50000</value>
      <value name="rlimit_core">0</value>
      <value name="request_slowlog_timeout">20s</value>
      <value name="slowlog">/var/log/php-slow.log</value>
      <value name="chroot"></value>
      <value name="chdir"></value>
      <value name="catch_workers_output">no</value>
      <value name="max_requests">5000</value>
      <value name="allowed_clients">127.0.0.1</value>
     
      <value name="environment">
        <value name="HOSTNAME">$HOSTNAME</value>
        <value name="PATH">/usr/local/bin:/usr/bin:/bin</value>
        <value name="TMP">/usr/tmp</value>
        <value name="TMPDIR">/usr/tmp</value>
        <value name="TEMP">/usr/tmp</value>
        <value name="OSTYPE">$OSTYPE</value>
        <value name="MACHTYPE">$MACHTYPE</value>
        <value name="MALLOC_CHECK_">2</value>
      </value>

    </section>       

4. Disable access log files

This can make a big impact, because log files on high traffic sites involve a lot of I/O that has to be synchronized across all threads.

access_log off; log_not_found off; error_log /var/log/nginx-error.log warn;

If you can't afford to turn off access log files, at least buffer them:

access_log /var/log/nginx/access.log main buffer=16k;

5. Enable GZip

gzip on; gzip_disable "msie6"; gzip_vary on; gzip_proxied any; gzip_comp_level 6; gzip_min_length 1100; gzip_buffers 16 8k; gzip_http_version 1.1; gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

6. Cache information about frequently accessed files

open_file_cache max=2000 inactive=20s; open_file_cache_valid 60s; open_file_cache_min_uses 5; open_file_cache_errors off;

7. Adjust client timeouts

client_max_body_size 50M; client_body_buffer_size 1m; client_body_timeout 15; client_header_timeout 15; keepalive_timeout 2 2; send_timeout 15; sendfile on; tcp_nopush on; tcp_nodelay on;

8. Adjust output buffers

fastcgi_buffers 256 16k; fastcgi_buffer_size 128k; fastcgi_connect_timeout 3s; fastcgi_send_timeout 120s; fastcgi_read_timeout 120s; fastcgi_busy_buffers_size 256k; fastcgi_temp_file_write_size 256k; reset_timedout_connection on; server_names_hash_bucket_size 100;

9. /etc/sysctl.conf tuning

 # Recycle Zombie connections net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.maxtcptw=200000
# Increase number of files kern.maxfiles=65535
kern.maxfilesperproc=16384
# Increase page share factor per process vm.pmap.pv_entry_max=54272521
vm.pmap.shpgperproc=20000
# Increase number of connections vfs.vmiodirenable=1
kern.ipc.somaxconn=3240000
net.inet.tcp.rfc1323=1
net.inet.tcp.delayed_ack=0
net.inet.tcp.restrict_rst=1
kern.ipc.maxsockbuf=2097152
kern.ipc.shmmax=268435456
# Host cache net.inet.tcp.hostcache.hashsize=4096
net.inet.tcp.hostcache.cachelimit=131072
net.inet.tcp.hostcache.bucketlimit=120
# Increase number of ports net.inet.ip.portrange.first=2000
net.inet.ip.portrange.last=100000
net.inet.ip.portrange.hifirst=2000
net.inet.ip.portrange.hilast=100000
kern.ipc.semvmx=131068
# Disable Ping-flood attacks net.inet.tcp.msl=2000
net.inet.icmp.bmcastecho=1
net.inet.icmp.icmplim=1
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

10. Monitor

Continually monitor the number of open connections, free memory and number of waiting threads.

Set alerts to notify you when thresholds exceed. You can build these alerts yourself, or use something likeServerDensity.

Be sure to install the NGINX stub_status module

You'll need to recompile NGINX -

 ./configure --with-http_ssl_module --with-http_stub_status_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module
make install BATCH=yes