Sunday, November 9, 2014

CentOS 6 KickStart Server


This blog post will explain how to build a Kickstart server which is used to automatically perform untattended OS installation and configuration of both RedHat 6 and CentOS 6 machines.

Kickstart is basically a copy of the Solaris Jumpstart. If you manage IBM AIX machines, it's the equivalent of NIM. OrIgnite in the HP-UX world.



The following details are important in this blog post...
  • Kickstart server's FQDN : angel.company.com 
  • DNS CNAME kickstart.company.com points to angel.company.com
  • Central Syslog Server DNS CNAME syslog.company.com 
  • Kickstart server's IP : 192.168.1.1 
  • Kickstart client machine FQDN : oxygen.company.com 
  • Kickstart client machine's MAC address : 00:11:43:e4:4f:3d 
  • Kickstart client's IP : 192.168.1.2 
  • DNS servers are : 192.168.1.24 and 192.168.1.53
  • NTP server are ntp1.company.com at 192.168.1.123
  • NTP server ntp2.company.com at 192.168.1.124
  • Kerberos Key Distribution Center master server is king.company.com
  • Kerberos 2nd KDC is kong.company.com
  • NFS server for user's homes is nfs1.company.com
  • OpenLDAP servers are ldap1.company.com and ldap2.company.com

Kickstart Server Setup


We start by installing either a RedHat 6 or a CentOS 6 machine. I suggest using a CentOS 6 machine because your organisation won't have to pay any licenses for it. As always, I prefer to install the Minimal OS version and add packages as you go along. This creates a machine with a minimum amount of packages installed which means we have less updates to manage.

Make sure to create an /export file system with quite a few GB of free space. For example, a Kickstart server for RedHat 6 x86_64, CentOS 6 i386 and CentOS 6 x86_64 requires 14 GB of disk space. Use LVM to manage your disks as it's more flexible. 

The /export filesystem is where we will store the complete RedHat 6 and CentOS 6 images along with the required Kickstart scripts and OS template configuration files. Don't forget that, with a little luck, the Kickstart server machine we are building now should one day serve as the Kickstart server for many future versions of RedHat/CentOS (i.e. 7, 8, 9, etc) but it can also install VMware ESX 3.5 servers (should you need that).

Once the kickstart server is installed, we need to add and configure a few packages to it. Mainly, we will need theApache HTTP web server, a Trivial File Transfer Protocol server, the Dynamic Host Configuration Protocol from theInternet Software Consortium. We will also need a copy of the DVD iso images from both CentOS 6 and RedHat Enterprise Linux 6 for the i386 (32 bit) and x86_64 (64 bit). Of course, if you're lucky enough to have only 64 bit capable machines, then don't bother with the 32 bit versions of the OS.

Create a directory tree in which we will store the DVD images and the kickstart configuration files.

sudo mkdir -p /export/install/linux/{centos,redhat,kickstart,etc,root}
sudo mkdir -p /export/install/linux/centos/6/{x86_64,i386}
sudo mkdir -p /export/install/linux/redhat/6/{x86_64,i386}

Go to the CentOS and RedHat websites and download the complete DVD iso image for version 6. Again, select the i386 and/or the x86_64 versions. In this blog post, I'll only show how to install the x86_64 version of CentOS 6 because that's what I use the most. Assuming you're working on a PC-BSD or Linux desktop, drop the DVD image into your ~/Downloads directory. Start the download now because these are big files to download.

Next, connect to the KickStart server and install some required applications.

ssh angel.company.com
sudo yum -y install tftp-server xinetd httpd vim dhcp

DNS Configuration


For this whole thing to work, we need to setup a DNS CNAME that points to our machine. So connect to your BIND name server and set one up. Test to see if it works :

dig +short kickstart.company.com. cname
angel.company.com.

Good that means your DNS resolver can find the right machine.

Apache Configuration


We will use Apache as the transport for our KickStart. We thus need to configure it.

sudo vim /etc/httpd/conf/httpd.conf

Check that our configuration is ok?

sudo httpd -S

VirtualHost configuration:
wildcard NameVirtualHosts and _default_ servers:
*:80                   is a NameVirtualHost
         default server kickstart.company.com (/etc/httpd/conf/httpd.conf:231)
         port 80 namevhost kickstart.company.com (/etc/httpd/conf/httpd.conf:231)
Syntax OK

Make sure httpd starts at boot time.

sudo chkconfig httpd on

Then start it.

sudo /etc/init.d/httpd start

DHCP Configuration


Configuring a Multihomed DHCP Server and ISC DHCP Documentation & FAQ. But the ISC has changed their site and I can't find this documentation anymore :(

sudo vim /etc/dhcp/dhcpd.conf
sudo vim /etc/sysconfig/dhcpd
sudo chkconfig dhcpd on
sudo /etc/init.d/dhcpd configtest
sudo /etc/init.d/dhcpd start

WARNING : Do NOT install a DHCP server in your corporation without the proper consent of the network administration group!

TFTP Server Configuration


That's easy enough, just edit the configuration file. See in.tftpd(8) for more info.


Then make sure xinetd(8) starts at boot.

sudo chkconfig xinetd on

And start xinetd.

sudo /etc/init.d/xinetd start

PXE Boot Configuration


You can read the official PXE Boot Configuration documentation on how to set things up. Or simply follow the instructions in this blog post. Now the only big differences between the CentOS and RedHat kickstart configuration are the vmlinuz and initrd.img files. The pxelinux.0 and pxelinux.cfg/default files can be used for both as we shall see later on.

CentOS Kickstart Preparation


Once you have both DVD1 and DVD2 iso from one of the CentOS mirrors, be sure to double check their SHA1 signatures against the ones found on the mirror.

openssl dgst -sha1 ~/Downloads/CentOS-6.4-x86_64-bin-DVD*

If that's good, then send them to your KickStart server (and if not, make sure to alert the mirror's maintainers!)

scp ~/Downloads/CentOS-6.4-x86_64-bin-DVD* kickstart.company.com:~/

Create mount directories for both of them.

sudo mkdir /mnt/dvd1 /mnt/dvd2

Then mount each of them in turn to their respective directories.

sudo mount -t iso9660 -o loop,ro ~/CentOS-6.4-x86_64-bin-DVD1.iso /mnt/dvd1
sudo mount -t iso9660 -o loop,ro ~/CentOS-6.4-x86_64-bin-DVD2.iso /mnt/dvd2


Once we have access to the DVD's content, we need to populate the syslinux and tftpboot directories. Let's do syslinux first. Our goal is to get the PXE boot file called pxelinux.0.

mkdir /tmp/syslinux
cd /tmp/syslinux
sudo cp -rp /mnt/dvd1/Packages/syslinux-*.x86_64.rpm /tmp/syslinux
rpm2cpio /tmp/syslinux/syslinux-4.02-4.el6.x86_64.rpm | cpio -dimv


The rpm2cpio(8) command will generate a directory tree starting with usr in the /tmp/syslinux directory. We then place this new syslinux PXE boot file into our /tftpboot directory. Note that this file is identical for CentOS 5, CentOS 6, RedHat 5 and RedHat 6 for both the i386 and x86_64 versions. Which means that we don't have to recreate it for every single operating system version that we want to KickStart.

sudo mkdir -p /tftpboot/pxelinux
sudo cp /tmp/syslinux/usr/share/syslinux/pxelinux.0 /tftpboot/pxelinux
sudo cp /tmp/syslinux/usr/share/syslinux/menu.c32 /tftpboot/pxelinux

Once we have our PXE boot file, we then create a directory tree that will house the rest of the boot files. These files are different from one operating system to another. So make sure you update them when you setup a new OS for the KickStart.
sudo mkdir -p /tftpboot/centos/6/x86_64/
sudo cp /mnt/dvd1/images/pxeboot/{vmlinuz,initrd.img} /tftpboot/centos/6/x86_64/

The PXE boot environment is almost complete. We now need to place a copy of the OS on the KickStart server. The idea here is to dump the entire content of the DVDs on the KickStart server. With CentOS 6.4, it required 5.5 GB of disk space. So plan accordingly.

sudo mkdir -p /export/install/linux/centos/6/x86_64
cd /mnt/dvd1
sudo tar cf - . | (cd /export/install/linux/centos/6/x86_64; sudo tar xvf -)
cd /mnt/dvd2
sudo tar cf - . | (cd /export/install/linux/centos/6/x86_64; sudo tar xvf -)

Once that's done, we can umount /mnt/{dvd1,dvd2} and get rid of the DVD iso images.

cd /
sudo umount /mnt/{dvd1,dvd2}
rm ~/CentOS-6.4-x86_64-bin-DVD*.iso

We now have the PXE files and the CentOS distribution on the disks of our KickStart server. The next step is to configure the profiles that will be loaded by a machine that boots via PXE. Those profiles will direct the system during the OS installation and configuration. Our first file will be the default configuration loaded by any PXE client if it can't find a specific profile. We will see how to setup machine specific profiles later.

sudo mkdir -p /tftpboot/pxelinux/pxelinux.cfg
sudo vim /tftpboot/pxelinux/pxelinux.cfg/default

The default file lists several different possibilities. For this blog post example, we will configure a machine specific profile for a CentOS 6 x86_64 installation. Recall that our client machine that will be installed has a MAC address of 00:11:43:e4:4f:3d. You can find out your machine's MAC address by the BIOS or sometimes it's written on the case. Or you can simply try to PXE boot it and look at the TFTP server logs which should be printed in/var/log/messages.

When that machine will PXE boot, it will automatically look for a file named after it's MAC address, but with a twist. The MAC address has it's colons « : » transformed by dashes « - ». It also has an extra « 01- » preprended. To create the file, we simply transform the MAC address to the file name expected by PXE boot protocol. Like so :

echo 00:11:43:e4:4f:3d | sed -e "s/:/-/g" -e "s/^/01-/g"
01-00-11-43-e4-4f-3d

We can thus create the file 01-00-11-43-e4-4f-3d like this.

sudo vim /tftpboot/pxelinux/pxelinux.cfg/01-00-11-43-e4-4f-3d

OPTIONAL : it's easier to remember that machine oxygen.company.com was installed instead of file 01-00-11-43-e4-4f-3d, so we can create a symbolic link. That step is optional, but is sysadmin friendly :)

sudo ln -s /tftpboot/pxelinux/pxelinux.cfg/01-00-11-43-e4-4f-3d /tftpboot/pxelinux/pxelinux.cfg/oxygen.company.com

In the 01-00-11-43-e4-4f-3d file, we reference a particular kickstart file called centos6.ks. We thus need to create this file too. This file contains the hard disk partitions which have been created according to RedHat's Recommended Partitioning Scheme and the file itself has been built according to RedHat's Kickstart Options. Note that if this machine has two disks, we need to list both /dev/sda and /dev/sdb. Make sure the « .ks » file references the same number of disks of the client machine.

Another important thing to consider when writing « .ks » files is that those disks need to be called the with the same name as the OS would see. For instance, HP machines use /dev/ciss/c0d0 disks instead of /dev/sda. Some Dell onboard RAID controllers will show /dev/md127 to the OS. So the KickStart file has to use the good disk device name. Otherwise the KickStart will fail.

sudo vim /export/install/linux/kickstart/centos6.ks

Good, we're close to our first system installation. 

Client Post-Kickstart OS Configuration Setup


What comes next is simply the site specific configurations. These configurations will of course change from site to site. So :

Make sure you edit the files to suit your own corporation's needs!

Site specific configurations are executed via a shell script which is launched after the operating system has been installed. My version of this script will configure the DNS resolver, OpenLDAP clients, NFS clients,  Kerberos realm, AutoFS via LDAP and Kerberos, the nsswitch.conf file and a whole bunch of other files in /etc and /root. It also creates the root password and the panic user which I use in cases where the NFS and/or LDAP services are not working anymore. Your Mileage Will Vary!

sudo vim /export/install/linux/kickstart/post.install.sh

This script, in turn, depends on a lot of files that should reside on our KickStart server and be accessible via the httpd server we configured. Here's all of them, but I'll say it again...

...make sure you edit the files to suit your own corporation's needs!

NOTE : this is a bit tedious, so of course, like they say in perl, there's more than one way to do it. You can use a configuration management software to do this for you. Things like Puppet, Chef and SaltStack are good examples of configuration management software.

We will start by creating the directory hierarchy.

sudo mkdir -p /export/install/linux/etc/logrotate.d
sudo mkdir /export/install/linux/etc/mail
sudo mkdir /export/install/linux/etc/openldap
sudo mkdir /export/install/linux/etc/pam.d
sudo mkdir /export/install/linux/etc/selinux
sudo mkdir /export/install/linux/etc/skel
sudo mkdir /export/install/linux/etc/snmp
sudo mkdir /export/install/linux/etc/ssh
sudo mkdir /export/install/linux/etc/sysconfig

And so now we can populate our KickStart server with our configuration files. All these files are going to be pushed to our clients after the OS has been installed. Let's start by the files right in /etc.

sudo vim /export/install/linux/etc/autofs_ldap_auth.conf
sudo vim /export/install/linux/etc/banner
sudo vim /export/install/linux/etc/hosts
sudo vim /export/install/linux/etc/idmapd.conf
sudo vim /export/install/linux/etc/issue
sudo vim /export/install/linux/etc/kdump.conf
sudo vim /export/install/linux/etc/krb5.conf
sudo vim /export/install/linux/etc/nslcd.conf
sudo vim /export/install/linux/etc/nsswitch.conf
sudo vim /export/install/linux/etc/ntp.conf
sudo vim /export/install/linux/etc/pam_ldap.conf
sudo vim /export/install/linux/etc/resolv.conf
sudo vim /export/install/linux/etc/rssh.conf
sudo vim /export/install/linux/etc/rsyslog.conf
sudo vim /export/install/linux/etc/sudoers
sudo vim /export/install/linux/etc/sudo-ldap.conf
sudo vim /export/install/linux/etc/sysctl.conf

Next we make sure we handle those log files.

sudo vim /export/install/linux/etc/logrotate.d/ntpd
sudo vim /export/install/linux/etc/logrotate.d/sudo

Now we can do the ones under /etc/mail.

sudo vim /export/install/linux/etc/mail/sendmail.mc
sudo vim /export/install/linux/etc/mail/submit.mc

The OpenLDAP client configuration in /etc/openldap.

sudo vim /export/install/linux/etc/openldap/ldap.conf

We setup our Pluggable Authentication Modules (PAM).

sudo vim /export/install/linux/etc/pam.d/sshd
sudo vim /export/install/linux/etc/pam.d/system-auth-ac

Now the one under /etc/pki/tls/certs. This is actually our Certificate Authority (CA) used to enable Transport Layer Security (TLS) to our OpenLDAP servers. You could also simply copy the rootca.crt file into the/etc/pki/tls/certs directory. Either way is fine. Of course, the filename might be different for you. And it has to be the exact same path and filename in all your configuration files.

sudo vim /etc/pki/tls/certs/rootca.crt

While we're talking security, let's setup SELinux.

sudo vim /export/install/linux/etc/selinux/config

Then the ones under /etc/skel. Used when creating users.

sudo vim /etc/skel/.aliases
sudo vim /etc/skel/.bash_profile
sudo vim /etc/skel/.bashrc

The NetSNMP client configuration in /etc/snmp

sudo vim /etc/snmp/snmpd.conf

The OpenSSH deamon configuration /etc/ssh.

sudo vim /etc/ssh/sshd_config

And now the ones under /etc/sysconfig.

sudo vim /export/install/linux/etc/sysconfig/autofs
sudo vim /export/install/linux/etc/sysconfig/ntpd
We also need some files in the /root directory. Which means we need to create the directory tree first.
sudo mkdir -p /export/install/linux/root

And then we can populate it with the files.

Ok, we now have quite a lot of files ready and waiting. Our next target is to prepare a repository of custom RPM that we want to install.

sudo mkdir -p /export/install/linux/repository/centos/6/x86_64

Then drop the latest Java JRE in there. When I wrote this post, it was jre-7u45-linux-x64.rpm.

sudo mv ~/Downloads/jre-7u45-linux-x64.rpm /export/install/linux/repository/centos/6/x86_64
sudo ln -s /export/install/linux/repository/centos/6/x86_64/jre-7u45-linux-x64.rpm /export/install/linux/repository/centos/6/x86_64/jre.rpm

Notice that we create a default jre.rpm symbolic link. With this, we don't need to update our post.install.sh script after each Java update. We just need to change the symlink.

I also like to drop the rssh rpm in there. The home page does not provide them. Just do the same as for the JRE.

Client Machine BIOS Setup


Next thing to do is to go in the client machine's BIOS and make sure the boot order is set to CD/DVD, then local hard disk and then PXE. If it's PXE first, then once we KickStart the server, it will reboot and do the PXE boot again. Which is a KickStart infinite loop! The other thing we need to make sure is that PXE is enabled on the Network Interface Card (NIC) we plan to use.

Now, from the kickstart server, open a shell and hit this :

sudo tail -F /var/log/messages

Once this is done, let's boot the client machine, hit PXE boot and see what happens?

Your client machine should boot via PXE, get it's IP via DHCP then issue several TFTP requests to get it's kernel, initrd and KickStart configuration. Then it's going to pull lots of files via HTTP and then reboot.

Once it has rebooted, you should have a new server taylored for your own environment!

Next Steps


Once the client has booted, connect to it via SSH and create it's Kerberos setup. Don't worry about the NFS error. It's normal at this point because we configured the autofs daemon to fetch the NFS mount tables from OpenLDAP. But the autofs to LDAP authentication is done via Kerberos. But at this point, the client does not have his final Kerberos config.

ssh oxygen.company.com
sudo kadmin
kadmin> addprinc -randkey host/oxygen.company.com@COMPANY.COM
kadmin> addprinc -randkey autofsclient/oxygen.company.com@COMPANY.COM
kadmin> ktadd host/oxygen.company.com@COMPANY.COM
kadmin> ktadd autofsclient/oxygen.company.com@COMPANY.COM
kadmin> exit
sudo /etc/init.d/autofs stop
sudo /etc/init.d/autofs start

Now you should have a working autofs daemon. Try it by simply going into your own directory.

cd && pwd

Thant's it! :)

New Client Setup


Now that we have a working KickStart and Configuration setup, we should use it to setup new machines. To do so, the only thing you need to change is the /export/install/linux/kickstart/centos6.ks file. Make sure to edit the network part of it so that you don't configure two systems with the same hostname and IP address.

Let's hope it works for you as it does for me :)

No comments:

Post a Comment