What is the difference between a restart and a graceful restart of a web server?

Q: – What is the difference between a restart and a graceful restart of a web server?

During a normal restart, the server is stopped and then started, causing some requests to be lost. A graceful restart allows Apache children to continue to serve their current requests until they can be replaced with children running the new configuration

How the web changes with HTTP/2: Performance and Security

Insights from the cURL developer

The first talk of the day was by Daniel Stenberg. If you ever used cURL, you might recognize him name from his personal domain haxx.se. The world is changing, and a tool like cURL must evolve as well. One of those big changes is support for HTTP/2, a fairly new protocol.

Daniel started off with explaining the importance of HTTP for the internet. We run almost everything on it, and usage grew tremendously. In the last four years, the average website got heavier. For example the number of objects, from 80 growing to 100 objects (HTML, CSS, JavaScript, etc). More impressive was the size of an average web page: four years ago that was about 800 KB, now it is 2300 KB.

Problem: latency

The main issue with “current” HTTP/1.1 protocol is what Daniel calls the “round trip bonanza”. A lot of connections are initiated (usually around 40 for the average website), each requesting a single object. The requests itself need to ping-pong between receiver and sender, adding up a lot of delay, as they are queued. The slightest addition of latency makes the current HTTP protocol inefficient. For most users bandwidth is no longer the issue, but this inefficient queuing. Daniel also referred to it as “head of line blocking”, like choosing in the supermarket the quickest line. The first person in that line will have a huge impact on the rest.

HTTP/1.1 workarounds

Last years people came up with clever ideas, or hacks, to improve the efficiency. Thinks like spriting, where many objects are merged and delivered as one single object. Think of small icons, which are merged into a single PNG. Then there isconcatenation (cat *.js > newfile.js), to reduce the number of files. Another example includes embedding those images in your CSS files, with base64 encoding. This is named inlining. Last but not least, Daniel mentioned sharding. As browsers generally allow up to 6 connections per domain, people started to spread their objects on more domains. This effectively allows more connections. With many tabs open in your browser, you can imagine the impact your browser has on the TCP stack of your operating system.

HTTP History

The HTTP protocol itself has a rich history, even though it doesn’t have that many versions. It started in 1996 with HTTP/1.0 (RFC 1945), followed by HTTP/1.1 in 1997 (RFC 2068) and an update in 1999 (RFC 2616). Then it was silent for a long time, till 2007 and httpbis refreshed HTTP/1.1.

The real big change started in 2009 by the well-known search engine Google. Their protocol SPDY wanted to solve many of the inefficiencies of HTTP/1.1. It was implemented in their data centers and Chrome browser in 2011. In the next year, the work on HTTP/2 started. In 2014 we got a slight revision to HTTP/1.1 (RFC 7230), making room for finally HTTP/2 in 2015 (RFC 7540).

What is HTTP/2?

If you create a new protocol, adoption is key. This is especially true for something with the importance of HTTP/2, making a serious change to the web. So Daniel explained the protocol is actually a framing layer, and maintaining a lot. So no changes to the old protocol, nor changing too many things. For example, POST and GET requests will remain. The naming convention (HTTP:// and HTTPS://) will also be unchanged. Another thing pointed out is that HTTP/1.1 will remain for a long time.

One of the biggest lessons learned is that protocols should not have too many optional components. HTTP/2 has therefore mostly mandatory items, specifying how it should be implemented. You are compliant with the specification, or not at all. That keeps things simple. For that same reason there is no minor version. The next version will most likely be HTTP/3.

Binary versus plaintext

One of the biggest changes is that the new protocol is no longer plaintext. Telnetting to your web server no longer gives you the typical response. Instead, the protocol is binary, which makes framing much easier. It has also performance benefits, as there are less translations needed. With default support for TLS and compression, things definitely changed. On the question how to do analysis now? Popular tools like Wireshark already have support. More tools for debugging, or security testing, will have to adopt.

Headers

The new protocol has two frame types, headers and data. Regarding these headers it is interesting to know that they grew over the past years. More data seems to be included, like cookie data. With HTTP/2 these will be compressed as well, especially as there is a lot of repetition involved.

Connections

One of the biggest changes is reusing connections. So data streams are multiplexed now, enhancing data transfers and reducing overhead. Systems together decide on how many parallel streams occur. Another improvement is so-called “connection coalescing”: if the client detects multiple sites using the same source system, that connections will be “unsharded” and merged with an existing stream.

Where previously HTTP requests were done one by one, in the new protocol they are done at the same time. So no more waiting for the first request to finish, reducing the initial request. To ensure the most optimal data flow occurs, streams can be prioritized by giving them a weight.

Server push

Normally the client does the requests. It requests the first pieces of HTML, then decides what else it will request, like CSS and JavaScript files. With the new protocol that will change. The server will be allowed to make suggestions and push data. That is, if the clients wants to accept that.

Adoption rates

As of April 2016, Daniel shared that Firefox already used 23% over HTTP/2. Regarding HTTPS, that is used for 35% of the traffic. Most browsers already support the new protocol. When it comes to the top 10 million websites, 7% has adopted the new protocol. That increases to 9% for the top 1 million, and even more for the top 500 (19%). Not bad at all, considering that most traffic will go to the last group. One of the big content delivery networks, Akamai, made the switch. The Google bots and Amazon platform are expected to follow soon.

How you can adopt

Daniel explained that you can easily run HTTP/2 yourself. Make sure to use an up-to-date browser. To see if you are using the new protocol, you can install browser extensions (search for HTTP2 browser indicator).

If you are doing things on the server side, it gets a little bit more complicated. First of all you need a web server that supports it. Your version from the repositories might not be up-to-date enough. So have a look at the latest version of litespeed, nginx, or Apache. Newer web server daemons like nghttp2, H2O, Caddy, will definitely have support for it.

Second problem is that OpenSSL might be too old, therefore missing APLN support. This stands for application protocol layer negotiation, which defines what protocol to use. Only by running the very latest versions of your Linux distribution, you might be having more luck now. Last but not least, you will have to configure a certificate. Fortunately the Let’s Encryptproject makes things easier now, for both configuration and the price tag.

So what about HTTP/2 security?

As we cover mainly security on this blog, it was interesting to note that a few items around security popped up. One of those was dealing with the some recent attacks like BEAST and CRIME.

HTTPS

A big misconception is that you need HTTPS when running HTTP/2. HTTPS is not mandatory according the specification. However, browsers don’t allow it. This means effectively that you will need a SSL/TLS certificate. But if you really want, you can still run the new protocol on port 80. It might be a good thing to have HTTPS available, as it provides privacy and user protection.

Safe defaults

The new protocol does not support things like SSL renegotiation, SSL compression, or any protocol before TLS 1.2 with weak ciphers. So on that level the protocol makes the web a lot safer.

Server push security

The security of server pushing is still vague. In this area more development is needed.

The future

A few areas are currently still vague. So is the security and implementation of server pushes under development. Also client certificates are not supported yet. Daniel listed this as a possible improvement, together with more tuning of the TCP stack, minor changes to handling of cookies.

Beyond HTTP/2 it is the goal to slowly drop the legacy of the first HTTP protocol (1.0 and 1.1). HTTP/3 will happen a lot faster, not the 16 years it now took. One of the more interesting developments is QUIC, or quick UDP internet connections. It is TCP, TLS, and HTTP/2 over UDP! More more head of line blocking, the main issue with using TCP.

About Daniel Stenberg

Daniel is from Sweden and works for Mozilla. He still keeps evolving the popular cURL toolkit (curl and libcurl). I had the honor to sit across the table with him and the other speakers. Knowledgeable, friendly and a great sense of humor.

Date-Tiered Compaction in Apache Cassandra

What is a compaction strategy?

The data files that Cassandra nodes store on disk are called sorted string tables (SSTables). They are essentially plain sorted arrays of data. Cassandra’s superior performance lies in its log-structured storage; it recognizes just how much more expensive random seeks are compared to sequential operations on modern hardware. Granted, this difference is larger on hard disk drives than on solid-state drives. Still, keeping data from fragmenting holds fundamental importance to Cassandra’s overall performance.

A sorted array may seem like a primitive data structure, but it allows single-seek range queries. Since Cassandra is not interested in doing small writes to disk, the terrible cost of insertions into sorted arrays is no loss; Cassandra wouldn’t do those anyway. Apart from the commit log which optimizes against disk seeks in ways that lie outside the scope of this post, the only kind of write to disk that Cassandra does is a big sequential write of data: the dumping of an in-memory Memtable in the form of a new SSTable. As most writes only touch memory, updating the Memtable (or, as of 2.0, a skip list) makes the amortized cost of a write request very low. Only when a Memtable is large enough, is it dumped to disk.

For a read request on a Cassandra node, the story of performance is not so simple. Performance is strongly correlated to the number of disk seeks needed to find the right value. The primary key that values are stored with consists of:

• The partition key, which determines which node data is stored on. Partition keys are hashed.
• One or more clustering keys that determine clustering. The SSTables are sorted based on the clustering keys.

Cassandra stores per-SSTable indexes in memory, so there’s no need for actual on-disk binary searching to find the right spot in an SSTable. Instead, the problem is finding the right SSTable. Remember that new SSTables are periodically dumped to disk; eventually thousands of SSTables will have been written.

How does Cassandra avoid performing thousands of disk seeks on every read request? Cassandra has ways of filtering some away. These are per-SSTable, and include:

• Bloom filters, which can tell when a partition key is not in an SSTable.
• Minimum & maximum clustering keys, which can help rule out a number of SSTables.
• Minimum & maximum timestamps, which lets Cassandra reason about whether updates or deletes of values could have come after a particular value was written.
• (Hashed) partition key ranges, which in case Leveled Compaction Strategy is used, significantly reduces the number of potential SSTables to look in.

But these optimizations don’t help too much if these SSTables are just left untouched. The optimization that does do something about that is compaction. The SSTables are immutable, but they can be compacted, i.e. merged into larger SSTables which when finished lets the original SSTables be garbage collected. This, of course, is yet another sequential disk operation. In short, compaction serves these purposes:

• Ensures that Cassandra has to look in as few SSTables as possible on a read request, after applying the list of optimizations mentioned above.
• Puts ranges of contiguous clustering keys clustered on disk, so that a given range of clustering keys within a single partition lies in as few files as possible.
• Evicts tombstones and TTL expired data. A tombstone is as “logical delete”; a write with a special “delete this” value.

Which SSTables to choose and when to compact

Until version 2.0.10 and 2.1.0, Cassandra shipped with two compaction strategies: Size-Tiered Compaction Strategy (STCS) and Leveled Compaction Strategy (LCS). These strategies for choosing what to compact and when and how are as different in approach as one can imagine. Before talking about them, one thing worth noting is that a compaction operation is not at all free. It takes time and space proportional to the combined size of the participating SSTables. And while multiple compactions can run in parallel, incoming requests become slower the more compactions that are running.

STCS has one simple goal: make it so there are as few SSTables on disk as possible at any given time. It does so in an asymptotically optimal way that only compacts SSTables that are close enough in size. On average and in the worst case, the number of SSTables on disk is logarithmic to the total data size on disk.

LCS takes a radically different approach. It not only merges SSTables, but it also splits them to make only a certain range of partition keys live in one SSTable. It then keeps these ranges apart in a smart way that means a partition key can only exist in a small number of files, logarithmic to the total data size on disk. When compared to STCS, it tends to result in fewer seeks per read request, but performs a higher number of compactions with a higher number of SSTables involved in each operation.

Time Series

One of Cassandra’s most common use-cases is for time series. A time series in Cassandra uses the time of a data point as the clustering key. This kind of use case has a number of useful properties that neither of the two aforementioned compaction strategies take advantage of:

• Clustering key and timestamp (Cassandra write time) are directly correlated.
• Data tends to be sent to Cassandra in time order. Out-of-order writes only happen on a small scale (in a matter of seconds, typically).
• The only kinds of deletes used are by TTL or entire partitions at once.
• The rate at which data is written is nearly constant.

A query on a time series is usually a slice query (range query) on a given partition. Perhaps, the most common query is something like “values from the last hour/day/week”.

Since Cassandra holds in-memory metadata about maximum and minimum clustering keys per SSTable, range queries can immediately rule out SSTables that have a range that lies outside the range of the query. But STCS and LCS pay no attention to this. The result after running Size-Tiered Compaction Strategy on a time series is that nearly all SSTables have some old data and some new data in them. Even if just one value is much older or newer than the rest, that changes the minimum/maximum timestamp, and as a result Cassandra sees that the SSTable covers a much larger range of clustering keys. Below is a diagram showing an example run of continuously writing to a new time series with STCS:

The horizontal axis in this diagram represents timestamps from the start of the example run on the left to the end on the right. Compactions were let finish before generating this image. Each rectangle is an SSTable (a total of 11 in this case). The left edge of each rectangle marks its minimum timestamp, the right edge marks the maximum timestamp. While the vertical axis bears no significance, the area of each rectangle represents the size of the data file. The vertical bar represents a slice query. This query asks for all data points, in a given partition, within a certain time span.

In this example, parts of all 11 SSTables need to be read, which results in 11 disk seeks. What we see here is variously sized SSTables covering more or less the full range of clustering keys. Size and age follow no pattern.

Date-Tiered Compaction

Before getting into details of the new Date-Tiered Compaction Strategy (DTCS), with the previous image fresh in mind, let’s see what DTCS produces in the same scenario:

In this image, you can see 20 SSTables which are spread out much more nicely over the timeline. The number of SSTables is higher, but that is mostly because of a low base_time_seconds setting. That setting determines how old an SSTable has to be to not immediately enter compaction. The slice query now hits only 3 SSTables, which is great, considering the size of the slice in this example covers 20% of the time line. Also, note that all rectangles have nearly the same height. This demonstrates the correlation between size and age of DTCS-compacted SSTables in a dataset where writes are done at a constant pace. The SSTables are not only date-tiered, they are size-tiered too!

You can immediately see how most queries would require Cassandra to look in every SSTable in the STCS case. In the above STCS example, the only somewhat efficient range to query is right at the start of the time span, typically the least important section. In the DTCS case, the SSTables lie in sequence, and only some of the SSTables have to be touched in slice queries. In a single key query or narrow slice query, most of the time only one SSTable has to be touched. A “last hour” query is still less efficient than a query for an earlier hour, but DTCS does the best it can, given the constant influx of SSTables. In practice, the newest SSTables are likely to be disk cached.

When to use DTCS (and when to avoid it)

It’s obvious that DTCS is great at handling time series. That’s what it was designed for. As has been shown here, DTCS also challenges STCS on any dataset that gets writes at a steady rate. This accounts for a high percentage of Cassandra use cases. DTCS is not allergic to more sporadic write patterns either. But something that works against the efforts of the strategy is writes with highly out-of-order timestamps. If an SSTable contains timestamps that don’t match the time when it was actually written to disk, it violates the size-to-age correspondence that DTCS tries to maintain. Such SSTables do appear organically in Cassandra as a result of repairs. The problem is that small SSTables end up in windows dedicated to large SSTables, and merging large files with small files is inefficient. Consider turning offread repairs. Anti-entropy repairs and hinted handoff don’t incur as much additional work for DTCS and may be used like usual.

Also, make sure that all writes use the same timestamp format. Whether it’s microseconds or milliseconds, a mix will behave as extremely out-of-order. As always, make sure that client clocks are synced. It’s the clients, not the servers that set the timestamps. Even DTCS’s idea of “current time” is governed by client timestamps.

Great at removing data

Cassandra’s log-structured storage has a hard time dealing with deletes. Tombstones are used to represent a delete or expired TTL. Cassandra manages to delete associated values during compactions at a good rate. But existing compaction strategies have a hard time deleting the tombstones themselves, and that can actually become a problem. DTCS helps a lot in purging tombstones, because when they reach the oldest SSTable, Cassandra simply removes any tombstone that is older than GC grace period. An SSTable that is older than all the others have no reason to hold tombstones beyond the grace period.

This is particularly useful for all-TTL datasets, where Cassandra is even able to delete entire SSTablesonce they have fully expired. DTCS helps keep new and old data separate, so that optimization can kick in considerably more often. This means that an all-TTL dataset with DTCS can be trusted to stay constant in size in the long run, and in an efficient manner.

Sub-properties

To used DTCS, the user specifies the compaction strategy as DateTieredCompactionStrategy, which is accompanied by a number of sub-properties. The options with the biggest effect on how DTCS behaves are (these properties and default values are subject to change):

• timestamp_resolution (default = “MICROSECONDS”): make sure that this is set to the same format that your clients use when writing timestamps (CQL uses microseconds by default).
• base_time_seconds (default = 3600 (1 hour)): this sets the initial window size which dictates how much of the most newly written data should be eagerly compacted together. All data older than base_time_seconds will subsequently be grouped together with other data about the same age, at which point further compactions will start happening increasingly (exponentially) less frequently. This option is similar to the min_sstable_size subproperty in SizeTieredCompactionStrategy.
• min_threshold (default = 4): for data not in the current window of newest data, min_threshold dictates how many of these previously compacted windows of the same (time span) size should be created before merging them into one time window min_threshold as large. Windows of the same size are of course adjacent to each other, and there are never more than min_threshold windows of one size (unless min_threshold = 1, of course).
• max_sstable_age_days (default = 365): stop compacting SSTables only having data older than than this number of days. Setting this low, reduces the total number of times the same value is rewritten to disk. Since it prevents compaction of the largest SSTables, it can also be used to limit the amount of free disk space needed for compaction.

How DTCS works

The idea is to act similarly to STCS, but instead of compacting based on SSTable size, compact based on SSTable age. The measurement of age used is the minimum timestamp of the SSTable, subtracted from the maximum timestamp globally. Assuming that writes come at a somewhat steady rate, and that timestamps roughly reflect the time of arrival to Cassandra (two properties that are held for much more than just time series), the size of an SSTable is upper bounded by its age. Furthermore, if SSTables are always merged with similarly aged SSTables, the old ones will be large and the new ones will be small. In other words, while DTCS compacts based on age, this correlation between age and size means that it gets the same asymptotic performance characteristics as STCS, while keeping data grouped by timestamps, as seen in the diagram above.

One way to implement this would be to fully mimic the behavior of STCS and compact SSTables with a relative age difference less than a constant factor. But there is one important property that differs between the size and the age of an SSTable: the size is constant, but the age rises over time. So SSTables that didn’t have ages within that constant factor a moment ago might suddenly have. This is a complication that we want to avoid. It also becomes difficult to reason about the efficiency of such an approach.

The approach that was chosen, instead uses the current time to compute time windows between Unix epoch and now. These windows don’t slide with the passage of time. Instead, as time passes, new time windows appear and old ones get merged into larger windows. The time that needs to pass before creating a new time window is specified by the base_time_seconds subproperty. The number of windows of one size that need to be accumulated before merging them into a bigger one is specified by the min_threshold subproperty. For example, with base_time_seconds=3600 (an hour) and min_threshold=4, the placement of the last few time windows at 7 consecutive hours may look like this:

The condition for compaction is simple: if multiple SSTables have an age that falls into the same window, DTCS will nominate all of those for compaction. The current point in time (computed as the maximum timestamp globally across SSTables) is always located in the latest time window.

The precise definition for when time windows get merged is: the moment min_threshold windows of one size get accompanied by yet another same-sized window, the aforementioned group of windows merge into one. This can have a domino effect as seen in the bottom lane of the image, where a fifth 1-hour window triggered the creation of a fifth 4-hour window, triggering the creation of a 16-hour window.

Summary

To keep data clustered based on write time, Date-Tiered Compaction Strategy uses information that other strategies disregard. It is very cheap to keep that structure as long as data is seldom written very out-of-order. This separation of old and new data is excellent for time series. It also holds an advantage against the other strategies at purging deleted data quickly and predictably.

CentOS 7 Server Hardening Guide

This HowTo walks you through the steps required to security harden CentOS 7, it’s based on the OpenSCAP benchmark, unfortunately the current version of OpenSCAP that ships with CentOS does not offically support CentOS CPE’s. But there is a “workaround” that will allow OpenSCAP + OpenSCAP workbench to run on CentOS, I’ll document this in a separate post.

Based on a Minimal Install

To follow this guide you will need a minimal CentOS 7 install, ideally using the Kickstart file below or copying it’s partition layout. Installing CentOS 7 using a minimal installation reduces the attack surface and ensures youonly install software that you require.

This guide only covers the base system + SSH hardening, I will document specific service hardening separately such as HTTPD, SFTP, LDAP, BIND etc…

In the section related to removing unrequired services, if you installed a minimal centos 7 install, you’ll likely have nothing to remove or disable - I’ve included this section for completeness.

Issues with Security Hardening

After hardening a system you may run into issues, hardening a system will make it more restrictive, especially SELinux or filesystem related permission hardening. When hardening a system for a specific task I recommend creating a duplicate virtual machine you can use for troubleshooting should you run into a issue that you think is related to security hardening, you’ll be able to confirm by running it on the Vanilla system.

Obviously don’t expose the Vanilla (un-hardened) system to the network!

Why use OpenSCAP ?

After a lot of research I decided to use OpenSCAP over other security hardening benchmarks / guides, here is my reasoning for doing so:

• It’s open, free and actively worked on
• It has an audit tool, essential to verify each system
• OpenSCAP has a GUI called, workbench
• OpenSCAP Workbench supports remote audits via SSH
• OpenSCAP Workbench allows you to customize your scan, should you not agree with all hardening checks

If you don’t get on with workbench or auditing from the command line, Nessus has functionality for authenticated SCAP scans.

Kickstart

I’ve provided the following RHEL kickstart file below, it’s a minimal install with a heavy partition scheme, allowing for stricter mount options.

#version=RHEL7

install
# System authorization information
auth --enableshadow --passalgo=sha512

# Use CDROM installation media
cdrom
# Accept EULA
eula --agreed

services --enabled=NetworkManager,sshd
reboot

# Run the Setup Agent on first boot
#firstboot --enable
ignoredisk --only-use=sda
# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'
# System language
lang en_US.UTF-8
# SELinux
selinux --enforcing
# Network information
network  --bootproto=dhcp --device=eno16777736 --onboot=on --ipv6=off
network  --hostname=default-vm
# Root password
rootpw --iscrypted HASHGOESHERE
# System timezone
timezone Europe/London --isUtc --ntpservers=prime.transformers
# System bootloader configuration
bootloader --location=mbr --boot-drive=sda
# Partition clearing information
clearpart --all --drives=sda
ignoredisk --only-use=sda
# LVM

# Disk partitioning information
part pv.18 --fstype="lvmpv" --ondisk=sda --size=8004
part pv.11 --fstype="lvmpv" --ondisk=sda --size=8004
part /boot --fstype="ext4" --ondisk=sda --size=1000
volgroup lg_data --pesize=4096 pv.18
volgroup lg_os --pesize=4096 pv.11
logvol /  --fstype="xfs" --size=4000 --name=lv_root --vgname=lg_os
logvol /home  --fstype="xfs" --size=2000 --name=lv_home --vgname=lg_data
logvol /tmp  --fstype="xfs" --size=1000 --name=lv_tmp --vgname=lg_os
logvol /var  --fstype="xfs" --size=2000 --name=lv_var --vgname=lg_os
logvol /var/tmp  --fstype="xfs" --size=1000 --name=lv_var_tmp --vgname=lg_os
logvol /var/www  --fstype="xfs" --size=5000 --name=lv_var_www --vgname=lg_data
logvol /var  --fstype="xfs" --size=1000 --name=lv_var --vgname=lg_os
logvol /var/log  --fstype="xfs" --size=1500 --name=lv_var_log --vgname=lg_os
logvol /var/log/audit  --fstype="xfs" --size=500 --name=lv_var_log_audit --vgname=lg_os
logvol /var/tmp  --fstype="ext4" --size=500 --name=lv_var_tmp --vgname=lg_os
logvol swap  --fstype="swap" --size=1000 --name=lv_swap --vgname=lg_data

%packages
@core
 %end

%post
%end

Secure Partition Mount Options

Your millage will vary here, for example if you have a website that uses cgi-bin executables you won’t be able to use the noexec mount options, but you can and should use it on /tmp and /var/tmp as this is typically the first place an attacker will attempt to write and execute from when performing privilege escalation.

Your /etc/fstab file should look something like:

#
# /etc/fstab
# Created by anaconda on Sat Oct 11 14:28:47 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/lg_os-lv_root /                       xfs     defaults        1 1
UUID=d73c5d22-75ed-416e-aad2-8c1bb1dfc713 /boot                   ext4    defaults,nosuid,noexec,nodev        1 2
/dev/mapper/lg_data-lv_home /home                   xfs     defaults        1 2
/dev/mapper/lg_os-lv_tmp /tmp                    xfs     defaults,nosuid,noexec,nodev        1 2
/dev/mapper/lg_os-lv_var /var                    xfs     defaults,nosuid        1 2
/dev/mapper/lg_os-lv_var_tmp /var/tmp                xfs     defaults,nosuid,noexec,nodev        1 2
/dev/mapper/lg_os-lv_var_tmp /var/log                xfs     defaults,nosuid,noexec,nodev        1 2
/dev/mapper/lg_os-lv_var_tmp /var/log/audit                xfs     defaults,nosuid,noexec,nodev        1 2
/dev/mapper/lg_data-lv_var_www /var/www                xfs     defaults,nosuid,noexec,nodev        1 2
/dev/mapper/lg_data-lv_swap swap                    swap    defaults        0 0

Install NTP

NTP is required for a number of compliance audits and is general good practice.

yum install ntp ntpdate
chkconfig ntpd on
ntpdate pool.ntp.org
/etc/init.d/ntpd start

Configure System for AIDE

Pre-linking binaries (arguably) improved execution time, however this cause issues with AIDE, so it must be disabled.

Open /etc/sysconfig/prelink and make sure the line Set PRELINKING=no is present, if you’re writing a script:

# Disable prelinking altogether
#
if grep -q ^PRELINKING /etc/sysconfig/prelink
then
  sed -i 's/PRELINKING.*/PRELINKING=no/g' /etc/sysconfig/prelink
else
  echo -e "\n# Set PRELINKING=no per security requirements" >> /etc/sysconfig/prelink
  echo "PRELINKING=no" >> /etc/sysconfig/prelink
fi

Disable previous prelink changes to binaries:

Disable previous prelink changes to binaries

root:~# /usr/sbin/prelink -ua

Install AIDE

Install AIDE - Advanced Intrusion Detection Environment

yum install aide -y && /usr/sbin/aide --init && cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz && /usr/sbin/aide --check

Configure periodic execution of AIDE, runs every morning at 04:30

echo "05 4 * * * root /usr/sbin/aide --check" >> /etc/crontab

Prevent Users Mounting USB Storage

echo "install usb-storage /bin/false" > /etc/modprobe.d/usb-storage.conf

Enable Secure (high quality) Password Policy

The following command will Enable SHA512 instead of using MD5:

authconfig --passalgo=sha512 —update

vi /etc/security/psquality.conf

# Configuration for systemwide password quality limits
# Defaults:
#
# Number of characters in the new password that must not be present in the
# old password.
difok = 5
#
# Minimum acceptable size for the new password (plus one if
# credits are not disabled which is the default). (See pam_cracklib manual.)
# Cannot be set to lower value than 6.
minlen = 14
#
# The maximum credit for having digits in the new password. If less than 0
# it is the minimum number of digits in the new password.
dcredit = 1
#
# The maximum credit for having uppercase characters in the new password.
# If less than 0 it is the minimum number of uppercase characters in the new
# password.
ucredit = 1
#
# The maximum credit for having lowercase characters in the new password.
# If less than 0 it is the minimum number of lowercase characters in the new
# password.
lcredit = 1
#
# The maximum credit for having other characters in the new password.
# If less than 0 it is the minimum number of other characters in the new
# password.
ocredit = 1
#
# The minimum number of required classes of characters for the new
# password (digits, uppercase, lowercase, others).
minclass = 4
#
# The maximum number of allowed consecutive same characters in the new password.
# The check is disabled if the value is 0.
maxrepeat = 3
#
# The maximum number of allowed consecutive characters of the same class in the
# new password.
# The check is disabled if the value is 0.
maxclassrepeat = 3
#
# Whether to check for the words from the passwd entry GECOS string of the user.
# The check is enabled if the value is not 0.
gecoscheck = 1
#
# Path to the cracklib dictionaries. Default is to use the cracklib default.
# dictpath =

Secure /etc/login.defs Pasword Policy

Add the following to /etc/login.defs

PASS_MIN_LEN 14
PASS_MIN_DAYS 1
PASS_MAX_DAYS 60

Set Last Logon/Access Notification

Open /etc/pam.d/system-auth and add the following line immediately after session required pam_limits.so:

session required pam_lastlog.so showfailed

Max Password Login Attempts per Session

Set the amount of password reprompts per session, by editing the pam_pwquality.so statement in /etc/pam.d/system-auth to retry=3 or lower.

Set Deny For Failed Password Attempts

Blocks logins for failed authentication on accounts.

Add the following lines immediately below the pam_unix.so statement in AUTH section of both /etc/pam.d/system-auth and /etc/pam.d/password-auth:

auth [default=die] pam_faillock.so authfail deny=3 unlock_time=604800 fail_interval=900

auth required pam_faillock.so authsucc deny=3 unlock_time=604800 fail_interval=900

Limit Password Reuse

Open /etc/pam.d/system-auth, append remember=24 to the pam_unix.so line - preventing users from reusing passwords, remembering 24 times is the DoD standard.

The line should look like:

password sufficient pam_unix.so existing_options remember=24

Verify /boot/grub2/grub.cfg Permissions

Set grub.conf to chmod 600:

sudo chmod 600/boot/grub2/grub.cfg

Set Boot Loader Password

The grub2 boot loader should have a superuser account and password protection enabled to protect boot-time settings.

To do so, select a superuser account and password and add them into the appropriate grub2 configuration file(s) under /etc/grub.d. Since plaintext passwords are a security risk, generate a hash for the pasword by running the following command:

grub2-mkpasswd-pbkdf2

When prompted, enter the password that was selected and insert the returned password hash into the appropriate grub2 configuration file(s) under /etc/grub.d immediately after the superuser account. (Use the output from grub2-mkpasswd-pbkdf2 as the value of password-hash):

password_pbkdf2 superusers-accountpassword-hash

Don't use common admin account names for the grub2 superuser

Avoid using common admin account names like, root, admin or administrator for the grub2 superuser account. To meet FISMA Moderate, the bootloader superuser account passwordmust differ from the root credentials.

grub2-mkconfig -o /boot/grub2/grub.cfg

Don't manually add the superuser account to grub.cfg

Do NOT manually add the superuser account and password to the grub.cfg file as the grub2-mkconfig command overwrites this file.

Require Authentication for Single User Mode

Require root password when entering single user mode, open /etc/sysconfig/init and add the line:

SINGLE=/sbin/sulogin

Disable Ctrl-Alt-Del Reboot Activation

Prevernt ALT+CTRL+DEL from rebooting.

Open /etc/init/control-alt-delete.conf and modify the existing line:

exec /sbin/shutdown -r now "Control-Alt-Delete pressed"

To:

exec /usr/bin/logger -p security.info "Control-Alt-Delete pressed"

Enable Console Screen Locking

Install the screen Package to allow console screen locking.

sudo yum install screen

Users can now run screen and lock the console with ctrl+a x.

Disable Zeroconf Networking

Zeroconf network typically occours when you fail to get an address via DHCP, the interface will be assigned a 169.254.0.0 address.

To prevernt this:

echo "NOZEROCONF=yes" >> /etc/sysconfig/network

Disable IPv6 Support Automatically Loading

Open /etc/modprobe.d/disabled.conf and add the line:

options ipv6 disable=1

Disable Interface Usage of IPv6

Add the following to /etc/sysconfig/network

NETWORKING_IPV6=no
IPV6INIT=no

Disable Support for RPC IPv6

RPC services like NFSv4 attempt to start using IPv6 even if it’s disabled in /etc/modprobe.d. To prevent this behaviour open /etc/netconfig and comment the following lines:

udp6       tpi_clts      v     inet6    udp     -       -
tcp6       tpi_cots_ord  v     inet6    tcp     -       -

Securing root Logins

Only allow root logins via local terminal:

echo "tty1" > /etc/securetty
chmod 700 /root

Enable UMASK 077

Can causes issues on systems where users share files:

perl -npe 's/umask\s+0\d2/umask 077/g' -i /etc/bashrc
perl -npe 's/umask\s+0\d2/umask 077/g' -i /etc/csh.cshrc

Prune Idle Users

echo "Idle users will be removed after 15 minutes"
echo "readonly TMOUT=900" >> /etc/profile.d/os-security.sh
echo "readonly HISTFILE" >> /etc/profile.d/os-security.sh
chmod +x /etc/profile.d/os-security.sh

Securing Cron

echo "Locking down Cron"
touch /etc/cron.allow
chmod 600 /etc/cron.allow
awk -F: '{print $1}' /etc/passwd | grep -v root > /etc/cron.deny
echo "Locking down AT"
touch /etc/at.allow
chmod 600 /etc/at.allow
awk -F: '{print $1}' /etc/passwd | grep -v root > /etc/at.deny

Sysctl Security

/etc/sysctl.conf

net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.tcp_max_syn_backlog = 1280
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_timestamps = 0

Deny All TCP Wrappers

TCP wrappers can provide a quick and easy method for controlling access to applications linked to them. Examples of TCP Wrapper aware applications are sshd, and portmap.

Below commands block all but SSH:

echo "ALL:ALL" >> /etc/hosts.deny
echo "sshd:ALL" >> /etc/hosts.allow

Basic iptables Firewall Rules

Basic iptables Firewall rules, set to denyall as the default.

#Drop anything we aren't explicitly allowing. All outbound traffic is okay
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type echo-reply -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type destination-unreachable -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type time-exceeded -j ACCEPT
# Accept Pings
-A RH-Firewall-1-INPUT -p icmp --icmp-type echo-request -j ACCEPT
# Log anything on eth0 claiming it's from a local or non-routable network
# If you're using one of these local networks, remove it from the list below
-A INPUT -i eth0 -s 10.0.0.0/8 -j LOG --log-prefix "IP DROP SPOOF A: "
-A INPUT -i eth0 -s 172.16.0.0/12 -j LOG --log-prefix "IP DROP SPOOF B: "
-A INPUT -i eth0 -s 192.168.0.0/16 -j LOG --log-prefix "IP DROP SPOOF C: "
-A INPUT -i eth0 -s 224.0.0.0/4 -j LOG --log-prefix "IP DROP MULTICAST D: "
-A INPUT -i eth0 -s 240.0.0.0/5 -j LOG --log-prefix "IP DROP SPOOF E: "
-A INPUT -i eth0 -d 127.0.0.0/8 -j LOG --log-prefix "IP DROP LOOPBACK: "
# Accept any established connections
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Accept ssh traffic. Restrict this to known ips if possible.
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
#Log and drop everything else
-A RH-Firewall-1-INPUT -j LOG
-A RH-Firewall-1-INPUT -j DROP
COMMIT

Verify iptables Enabled

sudo systemctl enable iptables
systemctl start iptables.service

Disable Uncommon Protocols

The following Protocols will be disabled:

• Datagram Congestion Control Protocol (DCCP)
• Stream Control Transmission Protocol (SCTP)
• Reliable Datagram Sockets (RDS)
• Transparent Inter-Process Communication (TIPC)

echo "install dccp /bin/false" > /etc/modprobe.d/dccp.conf
echo "install sctp /bin/false" > /etc/modprobe.d/sctp.conf
echo "install rds /bin/false" > /etc/modprobe.d/rds.conf
echo "install tipc /bin/false" > /etc/modprobe.d/tipc.conf

Ensure Rsyslog is installed

yum -y install rsyslog

Enable Rsyslog

systemctl enable rsyslog.service
systemctl start rsyslog.service

Auditd - Audit Daemon

Enable auditd Service

systemctl enable auditd.service
systemctl start auditd.service

Audit Processes Which Start Prior to auditd

Audit process which start before the Audit Daemon.

Add the following line to /etc/grub.conf:

kernel /vmlinuz-version ro vga=ext root=/dev/VolGroup00/LogVol00 rhgb quiet audit=1

Auditd Number of Logs Retained

Open /etc/audit/auditd.conf and add or modify:

num_logs = 5

Auditd Max Log File Size

max_log_file = 30MB

Auditd max_log_file_action

Open /etc/audit/auditd.conf and set this to rotate.

max_log_file_action = rotate

Auditd space_left

Configure auditd to email you when space gets low, open /etc/audit/auditd.conf and modify the following:

space_left_action = email

Auditd admin_space_left

Configure auditd to halt when auditd log space is used up, forcing the system admin to rectify the space issue.

On some systems where monitoring is less important another action could be leveraged.

admin_space_left_action = halt

Auditd mail_acct

When space gets low auditd can send a email notification via email, to configure this and the following line to /etc/audit/auditd.conf:

action_mail_acct = root

Configure auditd to use audispd plugin

Auditd does not have the functionality to send logs directly to an external log server, however the audispd plugin pass audit records to the local syslog server, to enable this open /etc/audisp/plugins.d/syslog.confand set the active line to yes, then restart audispd daemon:

sudo service auditd restart

Auditd Rules: /etc/audit/audit.rules

Open /etc/audit/audit.rules and add the following lines to monitor various system files and activities:

# audit_time_rules - Record attempts to alter time through adjtime
-a always,exit -F arch=b64 -S adjtimex -k audit_time_rules

# audit_time_rules - Record attempts to alter time through settimeofday
-a always,exit -F arch=b64 -S settimeofday -k audit_time_rules

# audit_time_rules - Record Attempts to Alter Time Through stime
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -S clock_settime
-k audit_time_rules

# audit_time_rules - Record Attempts to Alter Time Through clock_settime
-a always,exit -F arch=b64 -S clock_settime -k audit_time_rules

# Record Attempts to Alter the localtime File
-w /etc/localtime -p wa -k audit_time_rules

# Record Events that Modify User/Group Information
# audit_account_changes
-w /etc/group -p wa -k audit_account_changes
-w /etc/passwd -p wa -k audit_account_changes
-w /etc/gshadow -p wa -k audit_account_changes
-w /etc/shadow -p wa -k audit_account_changes
-w /etc/security/opasswd -p wa -k audit_account_changes

# Record Events that Modify the System's Network Environment
# audit_network_modifications
-a always,exit -F arch=ARCH -S sethostname -S setdomainname -k audit_network_modifications
-w /etc/issue -p wa -k audit_network_modifications
-w /etc/issue.net -p wa -k audit_network_modifications
-w /etc/hosts -p wa -k audit_network_modifications
-w /etc/sysconfig/network -p wa -k audit_network_modifications

#Record Events that Modify the System's Mandatory Access Controls
-w /etc/selinux/ -p wa -k MAC-policy

#Record Events that Modify the System's Discretionary Access Controls - chmod
-a always,exit -F arch=b32 -S chmod -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S chmod  -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - chown
-a always,exit -F arch=b32 -S chown -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S chown -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fchmod
-a always,exit -F arch=b32 -S fchmod -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fchmod -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fchmodat
-a always,exit -F arch=b32 -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fchown
-a always,exit -F arch=b32 -S fchown -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fchown -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fchownat
-a always,exit -F arch=b32 -S fchownat -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fchownat -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fremovexattr
-a always,exit -F arch=b32 -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fsetxattr
-a always,exit -F arch=b32 -S fsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - lchown
-a always,exit -F arch=b32 -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S lchown -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - lremovexattr
-a always,exit -F arch=b32 -S lremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S lremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - lsetxattr
-a always,exit -F arch=b32 -S lsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S lsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - removexattr
-a always,exit -F arch=b32 -S removexattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S removexattr -F auid>=500 -F auid!=4294967295 -k perm_mod-a always,exit -F arch=b32 -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fchmodat -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fchown
-a always,exit -F arch=b32 -S fchown -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fchown -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fchownat
-a always,exit -F arch=b32 -S fchownat -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fchownat -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fremovexattr
-a always,exit -F arch=b32 -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S fremovexattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - fsetxattr
-a always,exit -F arch=b32 -S lsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S lsetxattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - removexattr
-a always,exit -F arch=b32 -S removexattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S removexattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Events that Modify the System's Discretionary Access Controls - setxattr
-a always,exit -F arch=b32 -S setxattr -F auid>=500 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S setxattr -F auid>=500 -F auid!=4294967295 -k perm_mod

#Record Attempts to Alter Logon and Logout Events
-w /var/log/faillog -p wa -k logins
-w /var/log/lastlog -p wa -k logins

#Record Attempts to Alter Process and Session Initiation Information
-w /var/run/utmp -p wa -k session
-w /var/log/btmp -p wa -k session
-w /var/log/wtmp -p wa -k session

#Ensure auditd Collects Unauthorized Access Attempts to Files (unsuccessful)
-a always,exit -F arch=b32 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access
-a always,exit -F arch=b32 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access
-a always,exit -F arch=b64 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EACCES -F auid>=500 -F auid!=4294967295 -k access
-a always,exit -F arch=b64 -S creat -S open -S openat -S open_by_handle_at -S truncate -S ftruncate -F exit=-EPERM -F auid>=500 -F auid!=4294967295 -k access

#Ensure auditd Collects Information on the Use of Privileged Commands
#
#  Find setuid / setgid programs then modify and uncomment the line below.
#
##  sudo find / -xdev -type f -perm -4000 -o -perm -2000 2>/dev/null
#
# -a always,exit -F path=SETUID_PROG_PATH -F perm=x -F auid>=500 -F auid!=4294967295 -k privileged

#Ensure auditd Collects Information on Exporting to Media (successful)
-a always,exit -F arch=ARCH -S mount -F auid>=500 -F auid!=4294967295 -k export

#Ensure auditd Collects File Deletion Events by User
-a always,exit -F arch=ARCH -S rmdir -S unlink -S unlinkat -S rename -S renameat -F auid>=500 -F auid!=4294967295 -k delete

#Ensure auditd Collects System Administrator Actions
-w /etc/sudoers -p wa -k actions

#Ensure auditd Collects Information on Kernel Module Loading and Unloading
-w /sbin/insmod -p x -k modules
-w /sbin/rmmod -p x -k modules
-w /sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k modules

#Make the auditd Configuration Immutable
-e 2

Removal of Unrequired Services

The section outlines software that should be removed, instruction for disabling the service is also documented.

Bulk Remove of Services

# Remove
yum remove xinetd
yum remove telnet-server
yum remove rsh-server
yum remove telnet
yum remove rsh-server
yum remove rsh
yum remove ypbind
yum remove ypserv
yum remove tftp-server
yum remove cronie-anacron
yum remove bind
yum remove vsftpd
yum remove httpd
yum remove dovecot
yum remove squid
yum remove net-snmpd

Bulk Enable / Disable Services

#Disable / Enable
systemctl disable xinetd
systemctl disable rexec
systemctl disable rsh
systemctl disable rlogin
systemctl disable ypbind
systemctl disable tftp
systemctl disable certmonger
systemctl disable cgconfig
systemctl disable cgred
systemctl disable cpuspeed
systemctl enable irqbalance
systemctl disable kdump
systemctl disable mdmonitor
systemctl disable messagebus
systemctl disable netconsole
systemctl disable ntpdate
systemctl disable oddjobd
systemctl disable portreserve
systemctl enable psacct
systemctl disable qpidd
systemctl disable quota_nld
systemctl disable rdisc
systemctl disable rhnsd
systemctl disable rhsmcertd
systemctl disable saslauthd
systemctl disable smartd
systemctl disable sysstat
systemctl enable crond
systemctl disable atd
systemctl disable nfslock
systemctl disable named
systemctl disable httpd
systemctl disable dovecot
systemctl disable squid
systemctl disable snmpd

Disable Secure RPC Client Service

Disable rpcgssd:

The rpcgssd service manages RPCSEC GSS contexts required to secure protocols that use RPC (most often Kerberos and NFS). The rpcgssd service is the client-side of RPCSEC GSS. If the system does not require secure RPC then this service should be disabled. The rpcgssd service can be disabled with the following command:

systemctl disable rpcgssd

Disable Secure RPC Server Service

Disable rpcsvcgssd:

systemctl disable rpcsvcgssd

Disable RPC ID Mapping Service

Disable rpcidmapd.

The rpcidmapd service is used to map user names and groups to UID and GID numbers on NFSv4 mounts. If NFS is not in use on the local system then this service should be disabled. The rpcidmapd service can be disabled with the following command:

systemctl disable rpcidmapd

Disable Network File Systems (netfs)

The netfs script manages the boot-time mounting of several types of networked filesystems, of which NFS and Samba are the most common. If these filesystem types are not in use, the script can be disabled, protecting the system somewhat against accidental or malicious changes to /etc/fstab and against flaws in the netfs script itself. The netfs service can be disabled with the following command:

sudo systemctl disable netfs

Disable Network File System (nfs)

systemctl disable nfs

If you don’t need SSH disable it

systemctl disable sshd

Disable SSH iptables Firewall rule

Only do this if you don’t need SSH.

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

Tips™ - You probable need to leave SSH alone

Unless you know you don't need SSH, leave SSH and it's iptables rule enabled.

Remove Rsh Trust Files

rm /etc/hosts.equiv
rm ~/.rhosts

Disable Avahi Server Software

The avahi-daemon service can be disabled with the following command:

systemctl disable avahi-daemon

Disable the CUPS Service

If you don’t need CUPS, disable it to further reduce your attack surface:

systemctl disable cups

Disable DHCP Service

The dhcpd service should be disabled on any system that does not need to act as a DHCP server.

systemctl disable dhcpd

Uninstall DHCP Server Package

If you don’t need a DHCP client, remove it:

yum erase dhcp

Disable DHCP Client

Open /etc/sysconfig/network-scripts/ifcfg-eth0 (if you have more interfaces, do this for each one) and make sure the address is statically assigned with the BOOTPROTO=none

Example:

BOOTPROTO=none

NETMASK=255.255.255.0
IPADDR=192.168.1.2
GATEWAY=192.168.1.1

Specify Additional Remote NTP Servers

Open /etc/ntp.conf and add the following line:

server ntpserver

Use an internal NTP server if possible.

Enable Postfix

systemctl enable postfix

Remove Sendmail

yum remove sendmail

Postfix Disable Network Listening

Open, /etc/postfix/main.cf and ensure the following inet_interfaces line appears:

inet_interfaces = localhost

Configure SMTP Greeting Banner

Change the greeting banner, the default banner discloses the SMTP server is Postfix.

Disable xinetd Service

sudo systemctl disable xinetd

System Audit Logs Permissions

System audit logs must have 0640 or less permissions set.

sudo chmod 0640 audit_file

System Audit Logs Must Be Owned By Root

sudo chown root/var/log

Disable autofs

chkconfig --level 0123456 autofs off
service autofs stop

Disable uncommon filesystems

echo "install cramfs /bin/false" > /etc/modprobe.d/cramfs.conf
echo "install freevxfs /bin/false" > /etc/modprobe.d/freevxfs.conf
echo "install jffs2 /bin/false" > /etc/modprobe.d/jffs2.conf
echo "install hfs /bin/false" > /etc/modprobe.d/hfs.conf
echo "install hfsplus /bin/false" > /etc/modprobe.d/hfsplus.conf
echo "install squashfs /bin/false" > /etc/modprobe.d/squashfs.conf
echo "install udf /bin/false" > /etc/modprobe.d/udf.conf

Disable core dumps for all users

vi /etc/security/limits.conf

* hard core 0

Disable core dumps for SUID programs

Run sysctl -w fs.suid_dumpable=0 and fs.suid_dumpable = 0.

# Set runtime for fs.suid_dumpable
#
sysctl -q -n -w fs.suid_dumpable=0

#
# If fs.suid_dumpable present in /etc/sysctl.conf, change value to "0"
#     else, add "fs.suid_dumpable = 0" to /etc/sysctl.conf
#
if grep --silent ^fs.suid_dumpable /etc/sysctl.conf ; then
     sed -i 's/^fs.suid_dumpable.*/fs.suid_dumpable = 0/g' /etc/sysctl.conf
else
     echo "" >> /etc/sysctl.conf
     echo "# Set fs.suid_dumpable to 0 per security requirements" >> /etc/sysctl.conf
     echo "fs.suid_dumpable = 0" >> /etc/sysctl.conf
fi

Buffer Overflow Protection

This section helps mitigate against Buffer Overflow attacks (BOF).

Enable ExecShield

Helps prevent stack smashing / BOF.

Enable on current kernel: sysctl -w kernel.exec-shield=1

Add to /etc/sysctl.conf:

kernel.exec-shield = 1

Check / Enable ASLR

Set runtime for kernel.randomize_va_space sysctl -q -n -w kernel.randomize_va_space=2

Add kernel.randomize_va_space = 2 to /etc/sysctl.conf if it does not already exist.

Enable XD or NX Support on x86 Systems

Recent processors in the x86 family support the ability to prevent code execution on a per memory page basis. Generically and on AMD processors, this ability is called No Execute (NX), while on Intel processors it is called Execute Disable (XD). This ability can help prevent exploitation of buffer overflow vulnerabilities and should be activated whenever possible. Extra steps must be taken to ensure that this protection is enabled, particularly on 32-bit x86 systems. Other processors, such as Itanium and POWER, have included such support since inception and the standard kernel for those platforms supports the feature.

Check bios and ensure XD/NX is enabled, not relevant for VM’s.

SELinux

Confirm SELinux is not disabled

sed -i "s/selinux=0//gI" /etc/grub.conf
sed -i "s/enforcing=0//gI" /etc/grub.conf

SELinux Targeted / Enforcing

Open /etc/selinux/config and check for SELINUXTYPE=targeted or SELINUXTYPE=enforcing, depending on your requirements.

Enable the SELinux restorecond Service

The restorecond service utilizes inotify to look for the creation of new files listed in the /etc/selinux/restorecond.conf configuration file. When a file is created, restorecond ensures the file receives the proper SELinux security context. The restorecond service can be enabled with the following command:

Enable restorecond for all run levels:

chkconfig --level 0123456 restorecond on

Start restorecond if not currently running:

service restorecond start

Check no daemons are unconfined by SELinux

Run:

sudo ps -eZ | egrep "initrc" | egrep -vw "tr|ps|egrep|bash|awk" | tr ':' ' ' | awk '{ print $NF }’

This should return no output.

Prevent Log In to Accounts With Empty Password

sed -i 's/\<nullok\>//g' /etc/pam.d/system-auth

Secure SSH

Allow Only SSH Protocol 2

Open /etc/ssh/sshd_config and ensure the following line exists:

Protocol 2

Limit Users’ SSH Access

Open /etc/ssh/sshd_config and add:

DenyUsers USER1 USER2

Set SSH Idle Timeout Interval

To set an idle timeout interval, edit the following line in /etc/ssh/sshd_config as follows:

ClientAliveInterval interval

Set SSH Client Alive Count

To ensure the SSH idle timeout occurs precisely when the ClientAliveCountMax is set, edit /etc/ssh/sshd_config as follows:

ClientAliveCountMax 0

Disable SSH Support for .rhosts Files

SSH can emulate the behavior of the obsolete rsh command in allowing users to enable insecure access to their accounts via .rhosts files.

To ensure this behavior is disabled, add or correct the following line in /etc/ssh/sshd_config:

IgnoreRhosts yes

Disable Host-Based Authentication

SSH’s cryptographic host-based authentication is more secure than .rhosts authentication. However, it is not recommended that hosts unilaterally trust one another, even within an organization.

To disable host-based authentication, add or correct the following line in /etc/ssh/sshd_config:

HostbasedAuthentication no

Disable SSH Root Login

Disable root logins via SSH, open /etc/ssh/sshd_config and ensure the following line exists:

PermitRootLogin no

Disable SSH Access via Empty Passwords

Open /etc/ssh/sshd_config:

PermitEmptyPasswords no

Enable SSH Warning Banner

Enable a warning banner (Renforce policy awareness).

Banner /etc/issue

Do Not Allow SSH Environment Options

To ensure users are not able to present environment options to the SSH daemon, add or correct the following line in /etc/ssh/sshd_config:

PermitUserEnvironment no

Use Only Approved Ciphers

Limit the ciphers to those algorithms which are FIPS-approved. Counter (CTR) mode is also preferred over cipher-block chaining (CBC) mode. The following line in /etc/ssh/sshd_config demonstrates use of FIPS-approved ciphers:

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc

Secure X Windows

Disable X Windows Startup By Setting Runlevel

Disable X windows system, further reducing your attack surface.

Add id:3:initdefault: to /etc/inittab.

Remove the X Windows Package Group

yum groupremove "X Window System"

Prompt OS update installation

A process for prompt installation of OS updates must exist

yum -y install yum-cron
chkconfig yum-cron on

Make sure yum-cron is set to “check only”, I don’t recommend installing updates automatically.