How the web changes with HTTP/2: Performance and Security

Insights from the cURL developer

The first talk of the day was by Daniel Stenberg. If you ever used cURL, you might recognize him name from his personal domain haxx.se. The world is changing, and a tool like cURL must evolve as well. One of those big changes is support for HTTP/2, a fairly new protocol.

Daniel started off with explaining the importance of HTTP for the internet. We run almost everything on it, and usage grew tremendously. In the last four years, the average website got heavier. For example the number of objects, from 80 growing to 100 objects (HTML, CSS, JavaScript, etc). More impressive was the size of an average web page: four years ago that was about 800 KB, now it is 2300 KB.

Problem: latency

The main issue with “current” HTTP/1.1 protocol is what Daniel calls the “round trip bonanza”. A lot of connections are initiated (usually around 40 for the average website), each requesting a single object. The requests itself need to ping-pong between receiver and sender, adding up a lot of delay, as they are queued. The slightest addition of latency makes the current HTTP protocol inefficient. For most users bandwidth is no longer the issue, but this inefficient queuing. Daniel also referred to it as “head of line blocking”, like choosing in the supermarket the quickest line. The first person in that line will have a huge impact on the rest.

HTTP/1.1 workarounds

Last years people came up with clever ideas, or hacks, to improve the efficiency. Thinks like spriting, where many objects are merged and delivered as one single object. Think of small icons, which are merged into a single PNG. Then there isconcatenation (cat *.js > newfile.js), to reduce the number of files. Another example includes embedding those images in your CSS files, with base64 encoding. This is named inlining. Last but not least, Daniel mentioned sharding. As browsers generally allow up to 6 connections per domain, people started to spread their objects on more domains. This effectively allows more connections. With many tabs open in your browser, you can imagine the impact your browser has on the TCP stack of your operating system.

HTTP History

The HTTP protocol itself has a rich history, even though it doesn’t have that many versions. It started in 1996 with HTTP/1.0 (RFC 1945), followed by HTTP/1.1 in 1997 (RFC 2068) and an update in 1999 (RFC 2616). Then it was silent for a long time, till 2007 and httpbis refreshed HTTP/1.1.

The real big change started in 2009 by the well-known search engine Google. Their protocol SPDY wanted to solve many of the inefficiencies of HTTP/1.1. It was implemented in their data centers and Chrome browser in 2011. In the next year, the work on HTTP/2 started. In 2014 we got a slight revision to HTTP/1.1 (RFC 7230), making room for finally HTTP/2 in 2015 (RFC 7540).

What is HTTP/2?

If you create a new protocol, adoption is key. This is especially true for something with the importance of HTTP/2, making a serious change to the web. So Daniel explained the protocol is actually a framing layer, and maintaining a lot. So no changes to the old protocol, nor changing too many things. For example, POST and GET requests will remain. The naming convention (HTTP:// and HTTPS://) will also be unchanged. Another thing pointed out is that HTTP/1.1 will remain for a long time.

One of the biggest lessons learned is that protocols should not have too many optional components. HTTP/2 has therefore mostly mandatory items, specifying how it should be implemented. You are compliant with the specification, or not at all. That keeps things simple. For that same reason there is no minor version. The next version will most likely be HTTP/3.

Binary versus plaintext

One of the biggest changes is that the new protocol is no longer plaintext. Telnetting to your web server no longer gives you the typical response. Instead, the protocol is binary, which makes framing much easier. It has also performance benefits, as there are less translations needed. With default support for TLS and compression, things definitely changed. On the question how to do analysis now? Popular tools like Wireshark already have support. More tools for debugging, or security testing, will have to adopt.

Headers

The new protocol has two frame types, headers and data. Regarding these headers it is interesting to know that they grew over the past years. More data seems to be included, like cookie data. With HTTP/2 these will be compressed as well, especially as there is a lot of repetition involved.

Connections

One of the biggest changes is reusing connections. So data streams are multiplexed now, enhancing data transfers and reducing overhead. Systems together decide on how many parallel streams occur. Another improvement is so-called “connection coalescing”: if the client detects multiple sites using the same source system, that connections will be “unsharded” and merged with an existing stream.

Where previously HTTP requests were done one by one, in the new protocol they are done at the same time. So no more waiting for the first request to finish, reducing the initial request. To ensure the most optimal data flow occurs, streams can be prioritized by giving them a weight.

Server push

Normally the client does the requests. It requests the first pieces of HTML, then decides what else it will request, like CSS and JavaScript files. With the new protocol that will change. The server will be allowed to make suggestions and push data. That is, if the clients wants to accept that.

Adoption rates

As of April 2016, Daniel shared that Firefox already used 23% over HTTP/2. Regarding HTTPS, that is used for 35% of the traffic. Most browsers already support the new protocol. When it comes to the top 10 million websites, 7% has adopted the new protocol. That increases to 9% for the top 1 million, and even more for the top 500 (19%). Not bad at all, considering that most traffic will go to the last group. One of the big content delivery networks, Akamai, made the switch. The Google bots and Amazon platform are expected to follow soon.

How you can adopt

Daniel explained that you can easily run HTTP/2 yourself. Make sure to use an up-to-date browser. To see if you are using the new protocol, you can install browser extensions (search for HTTP2 browser indicator).

If you are doing things on the server side, it gets a little bit more complicated. First of all you need a web server that supports it. Your version from the repositories might not be up-to-date enough. So have a look at the latest version of litespeed, nginx, or Apache. Newer web server daemons like nghttp2, H2O, Caddy, will definitely have support for it.

Second problem is that OpenSSL might be too old, therefore missing APLN support. This stands for application protocol layer negotiation, which defines what protocol to use. Only by running the very latest versions of your Linux distribution, you might be having more luck now. Last but not least, you will have to configure a certificate. Fortunately the Let’s Encryptproject makes things easier now, for both configuration and the price tag.

So what about HTTP/2 security?

As we cover mainly security on this blog, it was interesting to note that a few items around security popped up. One of those was dealing with the some recent attacks like BEAST and CRIME.

HTTPS

A big misconception is that you need HTTPS when running HTTP/2. HTTPS is not mandatory according the specification. However, browsers don’t allow it. This means effectively that you will need a SSL/TLS certificate. But if you really want, you can still run the new protocol on port 80. It might be a good thing to have HTTPS available, as it provides privacy and user protection.

Safe defaults

The new protocol does not support things like SSL renegotiation, SSL compression, or any protocol before TLS 1.2 with weak ciphers. So on that level the protocol makes the web a lot safer.

Server push security

The security of server pushing is still vague. In this area more development is needed.

The future

A few areas are currently still vague. So is the security and implementation of server pushes under development. Also client certificates are not supported yet. Daniel listed this as a possible improvement, together with more tuning of the TCP stack, minor changes to handling of cookies.

Beyond HTTP/2 it is the goal to slowly drop the legacy of the first HTTP protocol (1.0 and 1.1). HTTP/3 will happen a lot faster, not the 16 years it now took. One of the more interesting developments is QUIC, or quick UDP internet connections. It is TCP, TLS, and HTTP/2 over UDP! More more head of line blocking, the main issue with using TCP.

About Daniel Stenberg

Daniel is from Sweden and works for Mozilla. He still keeps evolving the popular cURL toolkit (curl and libcurl). I had the honor to sit across the table with him and the other speakers. Knowledgeable, friendly and a great sense of humor.

NGINX Performance Tunning

Nginx is a widely used webserver, which ranks for better choice to serve static and dynamic web content faster. In present era, computing resources are shaped up in broader room. Optimization of service holds up as a priority to use these resources optimally. Nginx optimization is the one, which this article speaks about.

Generally in linux systems, nginx config files can be located in /etc/nginx among it nginx.conf is main config file. Tuning parameters found in this file to correct values thumbs up performance. Following points projects how to do it.

I. nginx.conf

1. "worker_processes":

• It can be considered as one worker process per processor core.

2. "worker_connections":

• Commonly, 1024 concurrent connections per server is enough. If these connections are not enough can raise it, by doubling it. But remember higher value may lockdown nginx with I/O operations like slow disks.

3. "sendfile":

• By keeping "sendfile on", makes nginx to ignore content of files it is sending, thus freeing resources for the administration level (request confirm).

4. "keepalive_timeout"

• Set this values as low as (2 or 3 seconds). It makes connection killed if no requests are received during the time frame for established connection, therfore serving requests makes better.

5. "open_file_cache"

• It helps nginx to keep open file descriptors (modification time, file and directory parameters) to serve fast. "open_file_cache max=10000 inactive=30s" Sets maximum of 10000 files and valid for 30 sec, can be tuned respective to available memory.

6. "open_file_cache_min_uses"

• It sets the value for minimum no.of access till file descriptor remains in the cache.

• "open_file_cache_min_uses 2"

7. "client_max_body_size"

• It is which client maximum request body size. Tune it with proper value to avoid (413 Request entity to large error), here we keep it as 100MB. "client_max_body_size 100M"

8. "client_body_buffer_size"

• It sets buffer size for reading client request body. Tune it w.r.t client_max_body_size. Here we kept it as 128K. (client_body_buffer_size 128k)

9. "server_names_hash_bucket_size"

•   If there is a situation where nginx need to parse large number of server names or if a long text of server names are defined, then server_names_hash_bucket_size directive need to be set. Generally it has default value of 32 and 64. "server_names_hash_bucket_size 64"

10. "gzip"

Compressing amount of network data sent by nginx upon client requests, will not only reduce size of data but also transfer in minimal time.

i)gzip on 

• To enable gzip compression.

ii)gzip_disable "msie6"

• Disables gziping for headers with MSIE 6

iii)gzip_vary on;

• It enables inserting the “Vary: Accept-Encoding” response header field

iv)gzip_comp_level 6

•  It sets compression level 1 to 9, don't' keep it higher value since it is CPU bound.

v)gzip_buffers 16 8k

•   Sets the no.of buffers and size of it, for compression. Analyze memory page of platform and set it.

vi)gzip_http_version 1.1

•   Sets the minimum HTTP version of a request required for compression.

vii)gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript text/js

• These are all MIME types which are subject to compression.

11. "access_log"

• Access logs are the one which puts on overhead of disk writes for each request. If they are  not needed one can turn off. "access_log off"

II. Virtual host file:

1. Cache Control for Static Files

• Browser caching will save resources for repaired transfer of static files. 

location ~* \.(js|css|png|jpg|jpeg|gif|ico)$ {
        expires 30d;
        log_not_found off;     
        add_header Pragma public;
        add_header Cache-Control "public";
        try_files $uri =404;
        } 

i)"expires"

• Directive sets expiry headers for static files in browser cache.

ii)"log_not_found"

• Disables logging of errors about not found files.

iii)"add_header"

• Standard headers which can be assigned for expiry files. 

iv)"try_files"

• Either serve the image or go to 404 error.

2. If nginx serves dynamic content processed by FastCGI Process Manager like PHP-FPM, then  following parameters also need to be tuned.

i)"fastcgi_buffering"

• Enables buffering of responses from fastcgi server. "fastcgi_buffering on"

ii)"fastcgi_buffer_size"

• It sets buffer size for responses received from fastcgi server, platform dependent.

"fastcgi_buffer_size 128k"

iii)"fastcgi_buffers"

• It sets the number of buffers and size of it, used for reading a response from the FastCGI server, for a single connection.

 "fastcgi_buffers 4 256k"

iv)"fastcgi_busy_buffers_size"

• It sets the busy buffer size, By default, size is limited by the size of two buffers set by the fastcgi_buffer_size.

"fastcgi_busy_buffers_size 256k"

3. To enable nginx to use FastCGI Process Manager like PHP-FPM and tune timeout parameters if php-fpm takes more time to serve.

location ~ \.php$ {       
                fastcgi_index   index.php;
                fastcgi_pass unix:/var/run/php5-fpm.sock;
                include fastcgi_params;
                fastcgi_connect_timeout 360s;
                fastcgi_send_timeout 600s;
                fastcgi_read_timeout 600s;
        }

• "fastcgi_connect_timeout"

Defines a timeout for establishing a connection with a FastCGI server

• "fastcgi_send_timeout"

To avoid 502 bad gateway, set a timeout for transmitting a request to the FastCGI server.

• "fastcgi_read_timeout"

To avoid 502 bad gateway, set a timeout for reading a response from the FastCGI server.

These config makes nginx to perform better, when compared to default settings.

- See more at: http://ktree.com/blog/Nginx-Performance-Optimisation.html#sthash.BPYV7zQ9.dpuf

Linux: Build NTP server in Centos 7

Centos7 SELinux closed, first confirm the firewall

[root@applog cat /etc/redhat-release ~]#
CentOS Linux release 7.0.1406 (Core)

The first step to install NTP ntpdate

[root@applog Yum install NTP ntpdate -y ~]#

The second step search time synchronization server

Http://www.pool.ntp.org/zone/asia

The third step to edit /etc/ntp.conf

Server time.windows.com

Server s2m.time.edu.cn

Server 0.asia.pool.ntp.org

Server 1.asia.pool.ntp.org

Server 2.asia.pool.ntp.org

Server 3.asia.pool.ntp.org

Server 127.127.1.0 iburst local clock when the external time is not available, use the local time.

IP restrict 172.100.1.0 mask 255.255.0.0 nomodify addresses allow update

Fourth steps to start the NTP service

Systemctl start ntpd

Systemctl enable ntpd.service set the boot service

The fifth step authentication service

[root@applog ntpq -p ~]#
Remote refid st T when poll reach delay offset jitter
==============================================================================
+23.102.23.44 188.114.116.1 3 u 33643295.525 -9.062 7.868
Ns.pku.edu.cn.STEP. 16 u 6400 - 0 0
-nipper.paina.jp 103.1.106.69 2 u 36643189.334 48.582 0.068
-203.158.247.150 202.28.214.2 2 u 38643258.045 -83.101 0.123
+168.63.242.24 137.189.4.10 2 u 4564366.155 -6.473 0.343
*linode.dev.fawo 218.186.3.36 2 u 4464371.535 10.114 0.157
LOCAL (0).LOCL. 5 L 1126420 0 0

[root@applog date -R ~]#
Mon, 12 Oct 2015 10:56:06 +0800

The sixth step client remote time synchronization test

[root@nginx01 date ~]#
On Monday October 12, 2015 10:30:01 CST

[root@nginx01 ntpdate 172.100.1.119 ~]#
12 Oct 11:12:11 ntpdate[19962] step time server 172.100.1.119: offset 1667.466210 sec

The seventh step client setting plan task, every night at 1 p.m. time synchronization

Crontab -e

1 * * * root /usr/sbin/ntpdate 172.100.1.119; /sbin/hwclock -w

To solve the error

[root@nginx01 ntpdate 172.100.1.119 ~]#
12 Oct 10:38:26 ntpdate[19199] server suitable for synchronization found: no

This is the wrong time server, no server time synchronization layer.

RHEL 7 Changes in Security

FIREWALLD

After Ipfwadm (2.0.X kernels), Ipchains (2.2.X kernels) and Iptables (2.4.X/2.6.X kernels), there is now Firewalld which stands for “Dynamic Firewall”.
This new firewall evolution brings several advantages:

• no connection cut during firewall reconfiguration,
• use of temporary configuration disappearing after reload/restart,
• services with their characteritics (ports, protocoles, modules) already recorded,
• predefined zones with various levels of trust,
• D-BUS applications support,
• use of “rich rules”.

The configuration consists in putting the server network interfaces into zones with the firewall-cmd command or the firewall-config graphical application.
The built-in configuration is located under the /usr/lib/firewalld directory. The configuration that you can customize is under the /etc/firewalld directory.
It is not possible to use Firewalld and Iptables at the same time. But it is still possible to disableFirewalld and use Iptables as before.
Further information can be found on the Fedora Project website, explanations about “rich rules” included.
A Firewalld beginner’s guide is available. Also, you can look at Thomas Woerner’s video (45min) about Firewalld.
In addition, the Red Hat annual Summit (2014) provided a presentation about the Next Generation Firewall.

FSS

FSS stands for Forward Secure Sealing. It’s a new mechanism invented by Lennart Poettering’s brother (Beltram Poettering) to secure systemd journal.
As FSS is disable by default, everything starts after running the following command:

# journalctl --setup-keys

This commands generates a key pair of “sealing key” and “verification key”. The verification key is only generated once, is not locally stored and must be recorded by you straight away. There will be no way to recreate it (a QR code is displayed to make the recording easier). Then, the sealing key will be used to sign all the messages written into the journal until a predefined delay is reached (15min by default). At this time, a new sealing key will be generated based on the previous one with no history kept.
An attacker will not be able to sign old messages, the messages showing when he broke into the system included, and will need to remove all of them. The removal of journal messages should make the discovery of any hack easier.
This mechanism doesn’t replace a centralized syslog server but offers minimal security when no such a server is available.
You can also check Lennart Poettering’s presentation on Google+.

Identity Management

There is now a better integration with Active Directory through cross‑realm Kerberos trust. This domain federation on the Kerberos level allows RHEL servers to accept the users coming fromActive Directory domains without loosing their native features in terms of POSIX attributes andSELinux capabilities.
Sources: Red Hat’s blog and Gordon Haff’s blog.
Additional information is available on the Red Hat Enterprise Linux Blog.

SELinux

Instead of putting all the system into SELinux permissive mode in order to debug a process, it is now possible to only put this process into SELinux permissive mode. SELinux instructions are available.
In addition, you can look at Dan Walsh’s presentation.

Also, the HTTPD SELinux policy (Apache and Nginx follow exactly the same SELinux policy) gets a slightly different behavior by default: the httpd_unified boolean that was previouslyenabled in RHEL 6 is now disabled by default in RHEL 7. A dedicated article about this HTTPD SELinux change is available.