Sunday, November 9, 2014

CentOS yum(8) Error « No module named cElementTree » Fixed

I've been having problems with yum(8) on one of the CentOS 6 x86_64 machines. After looking at many different forums and bug reports, I now found the solution.

The problem was very bad. When I would use yum(8), I would get this ugly Python error :

sudo yum -y update --exclude=yum
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: www.cubiculestudio.com
 * extras: centos.mirror.rafal.ca
 * updates: www.cubiculestudio.com
base                                                                                                                                                                          | 3.7 kB     00:00     
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 276, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 129, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 434, in doCommands
    self._getTs(needTsRemove)
  File "/usr/lib/python2.6/site-packages/yum/depsolve.py", line 99, in _getTs
    self._getTsInfo(remove_only)
  File "/usr/lib/python2.6/site-packages/yum/depsolve.py", line 110, in _getTsInfo
    pkgSack = self.pkgSack
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 883, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 668, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.6/site-packages/yum/repos.py", line 294, in populateSack
    sack.populate(repo, mdtype, callback, cacheonly)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 164, in populate
    if self._check_db_version(repo, mydbtype):
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 222, in _check_db_version
    return repo._check_db_version(mdtype)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1263, in _check_db_version
    repoXML = self.repoXML
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1462, in <lambda>
    repoXML = property(fget=lambda self: self._getRepoXML(),
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1454, in _getRepoXML
    self._loadRepoXML(text=self)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1444, in _loadRepoXML
    return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1419, in _groupLoadRepoXML
    if self._commonLoadRepoXML(text):
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1237, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1015, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 837, in _getFile
    size=size
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 408, in urlgrab
    return self._mirror_try(func, url, kw)
  File "/usr/lib/python2.6/site-packages/urlgrabber/mirror.py", line 394, in _mirror_try
    return func_ref( *(fullurl,), **kwargs )
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 985, in urlgrab
    return self._retry(opts, retryfunc, url, filename)
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 886, in _retry
    r = apply(func, (opts,) + args, {})
  File "/usr/lib/python2.6/site-packages/urlgrabber/grabber.py", line 980, in retryfunc
    apply(cb_func, (obj, )+cb_args, cb_kwargs)
  File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1501, in _checkRepoXML
    repoXML = repoMDObject.RepoMD(self.id, filepath)
  File "/usr/lib/python2.6/site-packages/yum/repoMDObject.py", line 124, in __init__
    self.parse(srcfile)
  File "/usr/lib/python2.6/site-packages/yum/repoMDObject.py", line 140, in parse
    parser = iterparse(infile)
  File "/usr/lib/python2.6/site-packages/yum/misc.py", line 1169, in cElementTree_iterparse
    _cElementTree_import()
  File "/usr/lib/python2.6/site-packages/yum/misc.py", line 1164, in _cElementTree_import
    import cElementTree
ImportError: No module named cElementTree

I searched the web and found quite a lot of other frustrated yum users. One of them suggested to test python right at the source : from python's shell.

sudo python
Python 2.6.6 (r266:84292, Jun 18 2012, 14:18:47) 
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from xml.etree import cElementTree
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/xml/etree/cElementTree.py", line 3, in <module>
    from _elementtree import *
ImportError: /usr/lib64/python2.6/lib-dynload/pyexpat.so: symbol XML_SetHashSalt, version EXPAT_2_0_1_RH not defined in file libexpat.so.1 with link time reference
>>> quit
Use quit() or Ctrl-D (i.e. EOF) to exit

So my pyexpat.so file is having problems? Ok, let's find it just for fun?

sudo find /lib /lib64 /usr /opt -type f -iname "libexpat.so*"

/lib64/libexpat.so.1.5.2
/opt/oracle/product/11.2.0/client_1/lib/libexpat.so.1
/opt/oracle/product/11.2.0/client_1/lib/libexpat.so.1.5.2

What's this Oracle file? Could it be causing me all those problems? 

In the end, the error was coming from the shared library dependencies. These are configured in the /etc/ld.so.conf file. Which is very basic

cat /etc/ld.so.conf
include ld.so.conf.d/*.conf

That says to load any configurations files ending with .conf found in the ld.so.conf.d directory. When I took a look at the content of this directory, I found this :

ls -1 /etc/ld.so.conf.d/*.conf

/etc/ld.so.conf.d/atlas-x86_64.conf
/etc/ld.so.conf.d/kernel-2.6.32-220.13.1.el6.x86_64.conf
/etc/ld.so.conf.d/kernel-2.6.32-220.17.1.el6.x86_64.conf
/etc/ld.so.conf.d/kernel-2.6.32-220.23.1.el6.x86_64.conf
/etc/ld.so.conf.d/kernel-2.6.32-220.4.1.el6.x86_64.conf
/etc/ld.so.conf.d/kernel-2.6.32-358.0.1.el6.x86_64.conf
/etc/ld.so.conf.d/mysql-x86_64.conf
/etc/ld.so.conf.d/qt-x86_64.conf
/etc/ld.so.conf.d/xulrunner-64.conf
/etc/ld.so.conf.d/oracle.conf

There's a lot of items here. Looking at this, I tried to remove all the non-CentOS configurations. Starting with the obvious one : oracle.conf.

sudo mv /etc/ld.so.conf.d/oracle.conf /tmp
sudo ldconfig

And voilà! Problem solved.

So basically, if you have this problem, double check your libraries. That's probably where the problem is.
Posted by at 1 comment:
Labels:

Secure Backup & Recovery with rsnapshot, rssh and OpenSSH

Overview

Wee all need to backup our machines. But we also need to keep the data private and the backup procedure secured. In UNIX and Linux machines, we need to run the backup operation as root in order to read everything on the machines. But allowing remote connections as the root user is not exactly a good idea. So how to we proceed? We use rsnapshot(1) and rssh(1) together with OpenSSH to secure the whole process. Here's how to do it on CentOS 6.

In case you're running a heterogeneous network, please note that I've successfully configured this process onFreeBSD, PC-BSD, RedHat, Ubuntu, AIX and Solaris servers.

In this example, our backup server is called backup.company.com and is running CentOS 6 while the clients are :
  1. The OpenLDAP server alice.company.com that we configured in several other blog posts and running CentOS 6.
  2. A workstation machine called charlie.company.com running PC-BSD 9.0 (i.e. FreeBSD 9.0 :)

Server Configuration (part 1 of 2)


Select a server which has a lot of hard disk space or that can grow it's storage without too much trouble. Don't forget that this solution is a disk-based backup solution, so we need disk space. Not an enormous quantity, thanks to rsnapshot, but enough to hold all of your current and future clients. If you can, I'd suggest that you use either a FreeBSD or a Solaris machine and configure the /backup filesystem on ZFS to benefit from it's data integrity feature and eliminate the risk of silent data corruption. There is ZFS on Linux, but I haven't tried it yet since it's still a release candidate. And BRTFS is also not production ready at the time of this writing. Since this is our production data, I don't want to use a non-production ready file system to store it.

So, once you selected a machine, install a minimal CentOS 6 on it and make sure to create a seperate file system with a mount point of /backup. 

Connect to the backup server and setup the /backup directories.

ssh backup.company.com
sudo mkdir /backup/{conf,data,key,log,run,scripts}
sudo chown -R root:root /backup

Then install some more packages.

sudo yum -y install openssh-clients rsnapshot

The rsnapshot installation will also install rsync as a dependency.

Next create a backup group and a backup user. Notice that our backup user has the UID zero (same as the root user). This is crucial for our purposes. Also keep in mind that you may select any GID and UID for the backup group and user. The important thing is to make sure that the backup GID and UID are exactly the same across your entire infrastructure.

sudo groupadd -g 911 backup
sudo useradd -c "Remote Backup User" -d /home/backup -o -u 0 -g 911 -m -s /bin/bash backup

Make sure to assign a password to our new backup user. This way it's account won't be locked.

sudo passwd backup

Switch to the backup user and create an SSH key pair. I use DSA here because all my other keys are RSA. Do not assign a passphrase to the new key.

sudo su - backup
ssh-keygen -t dsa -b 1024
exit

Move the key into the /backup/key directory.

sudo cp ~backup/.ssh/id_dsa ~backup/.ssh/id_dsa.pub /backup/key
sudo chmod 700 /backup/key

Create two wrapper scripts to help the process.


Make sure both scripts are executable and that they don't have any syntax errors in them.

sudo chmod a+x /backup/scripts/*.sh
sudo sh -n /backup/scripts/backup_runner.sh
sudo sh -n /backup/scripts/ssh_wrapper.sh

Exclude some files from the CentOS/RedHat machines.


Configure the clients. WARNING : rsnapshot is very sensitive with spaces and tabs. DO NOT USE ANY SPACES IN THE CONFIGURATION FILE! You have been warned :)


Make sure our backup log files don't consume too much disk space.


And make sure our new logrotate configuration is still valid.

sudo logrotate -d /etc/logrotate.conf

Before we can backup a machine, we must make sure the client allows the connection and has a copy of the backup user's public ssh key. Of course, replace my username by yours in the commands below...


sudo scp /backup/key/id_dsa.pub drobilla@alice.company.com:/tmp
sudo scp /backup/key/id_dsa.pub drobilla@charlie.company.com:/tmp

Client Configurations


In this section we will configure the CentOS and FreeBSD / PC-BSD clients.

CentOS 6 Client Configuration


We need to install rssh on each clients, but it's not in the default CentOS repositories. Thankfiully, rssh is available in RPMforge, one of the Additional CentOS Repositories. Simply follow the instructions to install the RPMforge repository. Once this is done, install rssh and rsync.

sudo yum -y install rssh rsync

We also need the same backup group and user on each clients. But here there is a subtle but very important difference : the backup user's shell is set to /usr/bin/rssh.

sudo groupadd -g 911 backup
sudo useradd -c "Remote Backup User" -d /home/backup -o -u 0 -g 911 -m -s /usr/bin/rssh backup

Make sure to assign a password to our new backup user. This way it's account won't be locked.

sudo passwd backup

Then place a copy of the backup user's public ssh key into the client's backup authorized_keys files.

sudo mkdir ~backup/.ssh
sudo mv /tmp/id_dsa.pub ~backup/.ssh/authorized_keys

Now edit the authorized_keys file to add the from="backup.company.com" keyword at the start of the file. This will restrict the use of this key to our backup server only. Any other machine trying to use this key will not be permitted. This is an extra layer of security to the whole setup. See this URL for more information on theauthorized_keys file syntax. Keep in mind that the file below is just an example!


Make sure it has the right permissions.

sudo chmod 600 ~backup/.ssh/authorized_keys
sudo chown root:root ~backup/.ssh/authorized_keys
sudo chmod 750 ~backup/.ssh

Before we can use rssh, we must configure it. Our goal is to use it only for the root/backup user, so let's do this now.

sudo vi /etc/rssh.conf

We must also add rssh to the available shells on the machine.

sudo vi /etc/shells

And of course, we must configure OpenSSH to allow our backup group to login, to permit the root user to login, but without password. Your sshd_config file may be different, so keep in mind that the important configuration keywords are AllowGroups and PermitRootLogin which are set like this :

sudo vi /etc/ssh/sshd_config

AllowGroups backup sysadmin
PermitRootLogin without-password

Restart the sshd(8) daemon so that it knows about the new configuration.

sudo /etc/init.d/sshd restart

And make sure it starts when the client comes up.

sudo chkconfig sshd on

The client machine is now ready. To test the backup, we must return to our backup server, try the ssh key and answer YES to the very first connection and then setup the crontab so that the backup happens every night without having to manually manage it.

PC-BSD 9.0 / FreeBSD 9.0 Client Configuration


Connect to the client machine.

ssh charlie.company.com

Update the ports tree.

sudo portsnap fetch
sudo portsnap update

Install rssh. We don't need rdist support, but you may compile it in if you want, that's not a problem.

cd /usr/ports/shells/rssh
sudo make install clean

Install rsync using all the default options.

cd /usr/ports/net/rsync
sudo make install clean

Create the backup group and backup user.

sudo pw group add -n backup -g 911
sudo pw user add -o -u 0 -n backup -d /usr/home/backup -g backup -c "Remote Backup User" -m -s /usr/local/bin/rssh -w random

We don't need to assign a password to our new backup user because we chose to use a random one when we created it (see the « -w random » option in the command above). You can forget about this password since you'll never need it.

Run a quick chmod(1) to fix an error if you forget to run it.

sudo chown root:wheel /usr/home/backup/.login_conf

Then place a copy of the backup user's public ssh key into the client's backup authorized_keys files.

sudo mkdir ~backup/.ssh
sudo mv /tmp/id_dsa.pub ~backup/.ssh/authorized_keys

Now edit the authorized_keys file to add the from="backup.company.com" keyword at the start of the file. This will restrict the use of this key to our backup server only. Any other machine trying to use this key will not be permitted. This is an extra layer of security to the whole setup. See this URL for more information on the authorized_keys file syntax. Keep in mind that the file below is just an example!


Make sure it has the right permissions.

sudo chmod 600 ~backup/.ssh/authorized_keys
sudo chown root:wheel ~backup/.ssh/authorized_keys
sudo chmod 750 ~backup/.ssh

Before we can use rssh, we must configure it. Our goal is to use it only for the root/backup user, so let's do this now.


And of course, we must configure OpenSSH to allow our backup group to login, to permit the root user to login, but without password. Your sshd_config file may be different, so keep in mind that the important configuration keywords are AllowGroups and PermitRootLogin which are set like this :
sudo vi /etc/ssh/sshd_config

AllowGroups backup sysadmin
PermitRootLogin without-password

And make sure it starts when the client comes up. Just add a single line to /etc/rc.conf.

sudo vi /etc/rc.conf
sshd_enable="YES"

Restart the sshd(8) daemon so that it knows about the new configuration.

sudo /etc/rc.d/sshd restart

The client machine is now ready. To test the backup, we must return to our backup server, try the ssh key and answer YES to the very first connection and then setup the crontab so that the backup happens every night without having to manually manage it.

Server Configuration (part 2 of 2)


Connect to the backup server.

ssh backup.company.com

Connect to the client using the backup user's key and answer YES. This is very important and you only need to do this once.

sudo ssh -i /backup/key/id_dsa backup@alice.company.com
The authenticity of host 'alice.company.com (192.168.1.20)' can't be established.
RSA key fingerprint is 60:0c:db:21:a2:c3:6b:0d:ae:03:f4:45:be:b5:e5:01.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'alice.company.com (192.168.1.20)' (DSA) to the list of known hosts.
Last login: Tue Jun 12 09:24:37 2012 from backup.company.com

This account is restricted by rssh.
Allowed commands: rsync

If you believe this is in error, please contact your system administrator.

As you can see, the rssh shell prevented us from have a shell on the client. But we still managed to connect to the client. All this in an encrypted ssh tunnel. Great!

IMPORTANT : make sure you answer YES once for all your backup clients!
Now let's try a backup now. Note that the first sudo -l is just there to prevent a password prompt which is usefull when you use an « & » at the end of the next sudo command.

sudo -l
sudo /backup/scripts/backup_runner.sh /backup/conf/alice.company.com daily &

And check the log as it happens.

sudo tail -F /backup/log/alice.company.com

Once the backup is over, confirm that you have the data on the backup server.

sudo ls -alFR /backup/data/alice.company.com/daily.0

And fiinally, configure cron(8) to manage all this automatically.


A good idea now would be to backup your backup server's data! Encrypt the data and send it offsite.

Recovery


Human Error


Recover with this setup is quite easy because you can simply navigate the backup directory and retrive any files as you normally do on a filesystem. For example, let's say you accidentally deleted the /etc/sysconfig/ifcfg-eth0 on host alice (and that you don't use RCS). Then simply connect to the backup server and send the file back to the client alice.company.com via scp(1). Again, replace my own username with yours in the scp(1) command below...

ssh backup.company.com
sudo scp /backup/data/alice.company.com/daily.0/etc/sysconfig/network-scripts/ifcfg-eth0 drobilla@alice.company.com:/tmp

Then connect to the client and move the file where it belongs.

ssh alice.company.com
sudo mv /tmp/ifcfg-eth0 /etc/sysconfig/network-scripts
sudo chown root:root /etc/sysconfig/network-scripts/ifcfg-eth0

There you go! Simple, secured and easy!

Total Client Failure


In the case where you suffer a total client failure, simply reinstall the OS, reconfigure the backup user but give him shell access (i.e. /bin/bash instead of /usr/bin/rssh) and use tar(1) to dump the content of the backups over to the newly installed client. Then reset the client's backup user's shell back to /usr/bin/rssh. For example :

ssh backup.company.com
sudo su -
cd /backup/data/alice.company.com/daily.0/etc
tar zcf - . | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /etc; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/home
tar zcf - . | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /home; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/root
tar zcf - .| ssh -i /backup/key/id_dsa backup@alice.company.com "cd /root; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/var/log
tar zcf - ./var | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /var/log; tar zxvf -"
cd /backup/data/alice.company.com/daily.0/var/lib/ldap
tar zcf - ./etc | ssh -i /backup/key/id_dsa backup@alice.company.com "cd /var/lib/ldap; tar zxvf -"

You can of course simply enable root access via SSH and do this only via the root user, but that would render the whole process a bit less secure.
Posted by at No comments:
Labels:

CentOS 6 KickStart Server


This blog post will explain how to build a Kickstart server which is used to automatically perform untattended OS installation and configuration of both RedHat 6 and CentOS 6 machines.

Kickstart is basically a copy of the Solaris Jumpstart. If you manage IBM AIX machines, it's the equivalent of NIM. OrIgnite in the HP-UX world.



The following details are important in this blog post...
  • Kickstart server's FQDN : angel.company.com 
  • DNS CNAME kickstart.company.com points to angel.company.com
  • Central Syslog Server DNS CNAME syslog.company.com 
  • Kickstart server's IP : 192.168.1.1 
  • Kickstart client machine FQDN : oxygen.company.com 
  • Kickstart client machine's MAC address : 00:11:43:e4:4f:3d 
  • Kickstart client's IP : 192.168.1.2 
  • DNS servers are : 192.168.1.24 and 192.168.1.53
  • NTP server are ntp1.company.com at 192.168.1.123
  • NTP server ntp2.company.com at 192.168.1.124
  • Kerberos Key Distribution Center master server is king.company.com
  • Kerberos 2nd KDC is kong.company.com
  • NFS server for user's homes is nfs1.company.com
  • OpenLDAP servers are ldap1.company.com and ldap2.company.com

Kickstart Server Setup


We start by installing either a RedHat 6 or a CentOS 6 machine. I suggest using a CentOS 6 machine because your organisation won't have to pay any licenses for it. As always, I prefer to install the Minimal OS version and add packages as you go along. This creates a machine with a minimum amount of packages installed which means we have less updates to manage.

Make sure to create an /export file system with quite a few GB of free space. For example, a Kickstart server for RedHat 6 x86_64, CentOS 6 i386 and CentOS 6 x86_64 requires 14 GB of disk space. Use LVM to manage your disks as it's more flexible. 

The /export filesystem is where we will store the complete RedHat 6 and CentOS 6 images along with the required Kickstart scripts and OS template configuration files. Don't forget that, with a little luck, the Kickstart server machine we are building now should one day serve as the Kickstart server for many future versions of RedHat/CentOS (i.e. 7, 8, 9, etc) but it can also install VMware ESX 3.5 servers (should you need that).

Once the kickstart server is installed, we need to add and configure a few packages to it. Mainly, we will need theApache HTTP web server, a Trivial File Transfer Protocol server, the Dynamic Host Configuration Protocol from theInternet Software Consortium. We will also need a copy of the DVD iso images from both CentOS 6 and RedHat Enterprise Linux 6 for the i386 (32 bit) and x86_64 (64 bit). Of course, if you're lucky enough to have only 64 bit capable machines, then don't bother with the 32 bit versions of the OS.

Create a directory tree in which we will store the DVD images and the kickstart configuration files.

sudo mkdir -p /export/install/linux/{centos,redhat,kickstart,etc,root}
sudo mkdir -p /export/install/linux/centos/6/{x86_64,i386}
sudo mkdir -p /export/install/linux/redhat/6/{x86_64,i386}

Go to the CentOS and RedHat websites and download the complete DVD iso image for version 6. Again, select the i386 and/or the x86_64 versions. In this blog post, I'll only show how to install the x86_64 version of CentOS 6 because that's what I use the most. Assuming you're working on a PC-BSD or Linux desktop, drop the DVD image into your ~/Downloads directory. Start the download now because these are big files to download.

Next, connect to the KickStart server and install some required applications.

ssh angel.company.com
sudo yum -y install tftp-server xinetd httpd vim dhcp

DNS Configuration


For this whole thing to work, we need to setup a DNS CNAME that points to our machine. So connect to your BIND name server and set one up. Test to see if it works :

dig +short kickstart.company.com. cname
angel.company.com.

Good that means your DNS resolver can find the right machine.

Apache Configuration


We will use Apache as the transport for our KickStart. We thus need to configure it.

sudo vim /etc/httpd/conf/httpd.conf

Check that our configuration is ok?

sudo httpd -S

VirtualHost configuration:
wildcard NameVirtualHosts and _default_ servers:
*:80                   is a NameVirtualHost
         default server kickstart.company.com (/etc/httpd/conf/httpd.conf:231)
         port 80 namevhost kickstart.company.com (/etc/httpd/conf/httpd.conf:231)
Syntax OK

Make sure httpd starts at boot time.

sudo chkconfig httpd on

Then start it.

sudo /etc/init.d/httpd start

DHCP Configuration


Configuring a Multihomed DHCP Server and ISC DHCP Documentation & FAQ. But the ISC has changed their site and I can't find this documentation anymore :(

sudo vim /etc/dhcp/dhcpd.conf
sudo vim /etc/sysconfig/dhcpd
sudo chkconfig dhcpd on
sudo /etc/init.d/dhcpd configtest
sudo /etc/init.d/dhcpd start

WARNING : Do NOT install a DHCP server in your corporation without the proper consent of the network administration group!

TFTP Server Configuration


That's easy enough, just edit the configuration file. See in.tftpd(8) for more info.


Then make sure xinetd(8) starts at boot.

sudo chkconfig xinetd on

And start xinetd.

sudo /etc/init.d/xinetd start

PXE Boot Configuration


You can read the official PXE Boot Configuration documentation on how to set things up. Or simply follow the instructions in this blog post. Now the only big differences between the CentOS and RedHat kickstart configuration are the vmlinuz and initrd.img files. The pxelinux.0 and pxelinux.cfg/default files can be used for both as we shall see later on.

CentOS Kickstart Preparation


Once you have both DVD1 and DVD2 iso from one of the CentOS mirrors, be sure to double check their SHA1 signatures against the ones found on the mirror.

openssl dgst -sha1 ~/Downloads/CentOS-6.4-x86_64-bin-DVD*

If that's good, then send them to your KickStart server (and if not, make sure to alert the mirror's maintainers!)

scp ~/Downloads/CentOS-6.4-x86_64-bin-DVD* kickstart.company.com:~/

Create mount directories for both of them.

sudo mkdir /mnt/dvd1 /mnt/dvd2

Then mount each of them in turn to their respective directories.

sudo mount -t iso9660 -o loop,ro ~/CentOS-6.4-x86_64-bin-DVD1.iso /mnt/dvd1
sudo mount -t iso9660 -o loop,ro ~/CentOS-6.4-x86_64-bin-DVD2.iso /mnt/dvd2

Once we have access to the DVD's content, we need to populate the syslinux and tftpboot directories. Let's do syslinux first. Our goal is to get the PXE boot file called pxelinux.0.

mkdir /tmp/syslinux
cd /tmp/syslinux
sudo cp -rp /mnt/dvd1/Packages/syslinux-*.x86_64.rpm /tmp/syslinux
rpm2cpio /tmp/syslinux/syslinux-4.02-4.el6.x86_64.rpm | cpio -dimv

The rpm2cpio(8) command will generate a directory tree starting with usr in the /tmp/syslinux directory. We then place this new syslinux PXE boot file into our /tftpboot directory. Note that this file is identical for CentOS 5, CentOS 6, RedHat 5 and RedHat 6 for both the i386 and x86_64 versions. Which means that we don't have to recreate it for every single operating system version that we want to KickStart.

sudo mkdir -p /tftpboot/pxelinux
sudo cp /tmp/syslinux/usr/share/syslinux/pxelinux.0 /tftpboot/pxelinux
sudo cp /tmp/syslinux/usr/share/syslinux/menu.c32 /tftpboot/pxelinux

Once we have our PXE boot file, we then create a directory tree that will house the rest of the boot files. These files are different from one operating system to another. So make sure you update them when you setup a new OS for the KickStart.
sudo mkdir -p /tftpboot/centos/6/x86_64/
sudo cp /mnt/dvd1/images/pxeboot/{vmlinuz,initrd.img} /tftpboot/centos/6/x86_64/

The PXE boot environment is almost complete. We now need to place a copy of the OS on the KickStart server. The idea here is to dump the entire content of the DVDs on the KickStart server. With CentOS 6.4, it required 5.5 GB of disk space. So plan accordingly.

sudo mkdir -p /export/install/linux/centos/6/x86_64
cd /mnt/dvd1
sudo tar cf - . | (cd /export/install/linux/centos/6/x86_64; sudo tar xvf -)
cd /mnt/dvd2
sudo tar cf - . | (cd /export/install/linux/centos/6/x86_64; sudo tar xvf -)

Once that's done, we can umount /mnt/{dvd1,dvd2} and get rid of the DVD iso images.

cd /
sudo umount /mnt/{dvd1,dvd2}
rm ~/CentOS-6.4-x86_64-bin-DVD*.iso

We now have the PXE files and the CentOS distribution on the disks of our KickStart server. The next step is to configure the profiles that will be loaded by a machine that boots via PXE. Those profiles will direct the system during the OS installation and configuration. Our first file will be the default configuration loaded by any PXE client if it can't find a specific profile. We will see how to setup machine specific profiles later.

sudo mkdir -p /tftpboot/pxelinux/pxelinux.cfg
sudo vim /tftpboot/pxelinux/pxelinux.cfg/default

The default file lists several different possibilities. For this blog post example, we will configure a machine specific profile for a CentOS 6 x86_64 installation. Recall that our client machine that will be installed has a MAC address of 00:11:43:e4:4f:3d. You can find out your machine's MAC address by the BIOS or sometimes it's written on the case. Or you can simply try to PXE boot it and look at the TFTP server logs which should be printed in/var/log/messages.

When that machine will PXE boot, it will automatically look for a file named after it's MAC address, but with a twist. The MAC address has it's colons « : » transformed by dashes « - ». It also has an extra « 01- » preprended. To create the file, we simply transform the MAC address to the file name expected by PXE boot protocol. Like so :

echo 00:11:43:e4:4f:3d | sed -e "s/:/-/g" -e "s/^/01-/g"
01-00-11-43-e4-4f-3d

We can thus create the file 01-00-11-43-e4-4f-3d like this.

sudo vim /tftpboot/pxelinux/pxelinux.cfg/01-00-11-43-e4-4f-3d

OPTIONAL : it's easier to remember that machine oxygen.company.com was installed instead of file 01-00-11-43-e4-4f-3d, so we can create a symbolic link. That step is optional, but is sysadmin friendly :)

sudo ln -s /tftpboot/pxelinux/pxelinux.cfg/01-00-11-43-e4-4f-3d /tftpboot/pxelinux/pxelinux.cfg/oxygen.company.com

In the 01-00-11-43-e4-4f-3d file, we reference a particular kickstart file called centos6.ks. We thus need to create this file too. This file contains the hard disk partitions which have been created according to RedHat's Recommended Partitioning Scheme and the file itself has been built according to RedHat's Kickstart Options. Note that if this machine has two disks, we need to list both /dev/sda and /dev/sdb. Make sure the « .ks » file references the same number of disks of the client machine.

Another important thing to consider when writing « .ks » files is that those disks need to be called the with the same name as the OS would see. For instance, HP machines use /dev/ciss/c0d0 disks instead of /dev/sda. Some Dell onboard RAID controllers will show /dev/md127 to the OS. So the KickStart file has to use the good disk device name. Otherwise the KickStart will fail.

sudo vim /export/install/linux/kickstart/centos6.ks

Good, we're close to our first system installation. 

Client Post-Kickstart OS Configuration Setup


What comes next is simply the site specific configurations. These configurations will of course change from site to site. So :

Make sure you edit the files to suit your own corporation's needs!

Site specific configurations are executed via a shell script which is launched after the operating system has been installed. My version of this script will configure the DNS resolver, OpenLDAP clients, NFS clients,  Kerberos realm, AutoFS via LDAP and Kerberos, the nsswitch.conf file and a whole bunch of other files in /etc and /root. It also creates the root password and the panic user which I use in cases where the NFS and/or LDAP services are not working anymore. Your Mileage Will Vary!

sudo vim /export/install/linux/kickstart/post.install.sh

This script, in turn, depends on a lot of files that should reside on our KickStart server and be accessible via the httpd server we configured. Here's all of them, but I'll say it again...

...make sure you edit the files to suit your own corporation's needs!

NOTE : this is a bit tedious, so of course, like they say in perl, there's more than one way to do it. You can use a configuration management software to do this for you. Things like Puppet, Chef and SaltStack are good examples of configuration management software.

We will start by creating the directory hierarchy.

sudo mkdir -p /export/install/linux/etc/logrotate.d
sudo mkdir /export/install/linux/etc/mail
sudo mkdir /export/install/linux/etc/openldap
sudo mkdir /export/install/linux/etc/pam.d
sudo mkdir /export/install/linux/etc/selinux
sudo mkdir /export/install/linux/etc/skel
sudo mkdir /export/install/linux/etc/snmp
sudo mkdir /export/install/linux/etc/ssh
sudo mkdir /export/install/linux/etc/sysconfig

And so now we can populate our KickStart server with our configuration files. All these files are going to be pushed to our clients after the OS has been installed. Let's start by the files right in /etc.

sudo vim /export/install/linux/etc/autofs_ldap_auth.conf
sudo vim /export/install/linux/etc/banner
sudo vim /export/install/linux/etc/hosts
sudo vim /export/install/linux/etc/idmapd.conf
sudo vim /export/install/linux/etc/issue
sudo vim /export/install/linux/etc/kdump.conf
sudo vim /export/install/linux/etc/krb5.conf
sudo vim /export/install/linux/etc/nslcd.conf
sudo vim /export/install/linux/etc/nsswitch.conf
sudo vim /export/install/linux/etc/ntp.conf
sudo vim /export/install/linux/etc/pam_ldap.conf
sudo vim /export/install/linux/etc/resolv.conf
sudo vim /export/install/linux/etc/rssh.conf
sudo vim /export/install/linux/etc/rsyslog.conf
sudo vim /export/install/linux/etc/sudoers
sudo vim /export/install/linux/etc/sudo-ldap.conf
sudo vim /export/install/linux/etc/sysctl.conf

Next we make sure we handle those log files.

sudo vim /export/install/linux/etc/logrotate.d/ntpd
sudo vim /export/install/linux/etc/logrotate.d/sudo

Now we can do the ones under /etc/mail.

sudo vim /export/install/linux/etc/mail/sendmail.mc
sudo vim /export/install/linux/etc/mail/submit.mc

The OpenLDAP client configuration in /etc/openldap.

sudo vim /export/install/linux/etc/openldap/ldap.conf

We setup our Pluggable Authentication Modules (PAM).

sudo vim /export/install/linux/etc/pam.d/sshd
sudo vim /export/install/linux/etc/pam.d/system-auth-ac

Now the one under /etc/pki/tls/certs. This is actually our Certificate Authority (CA) used to enable Transport Layer Security (TLS) to our OpenLDAP servers. You could also simply copy the rootca.crt file into the/etc/pki/tls/certs directory. Either way is fine. Of course, the filename might be different for you. And it has to be the exact same path and filename in all your configuration files.

sudo vim /etc/pki/tls/certs/rootca.crt

While we're talking security, let's setup SELinux.

sudo vim /export/install/linux/etc/selinux/config

Then the ones under /etc/skel. Used when creating users.

sudo vim /etc/skel/.aliases
sudo vim /etc/skel/.bash_profile
sudo vim /etc/skel/.bashrc

The NetSNMP client configuration in /etc/snmp. 

sudo vim /etc/snmp/snmpd.conf

The OpenSSH deamon configuration /etc/ssh.

sudo vim /etc/ssh/sshd_config

And now the ones under /etc/sysconfig.

sudo vim /export/install/linux/etc/sysconfig/autofs
sudo vim /export/install/linux/etc/sysconfig/ntpd
We also need some files in the /root directory. Which means we need to create the directory tree first.
sudo mkdir -p /export/install/linux/root

And then we can populate it with the files.

Ok, we now have quite a lot of files ready and waiting. Our next target is to prepare a repository of custom RPM that we want to install.

sudo mkdir -p /export/install/linux/repository/centos/6/x86_64

Then drop the latest Java JRE in there. When I wrote this post, it was jre-7u45-linux-x64.rpm.

sudo mv ~/Downloads/jre-7u45-linux-x64.rpm /export/install/linux/repository/centos/6/x86_64
sudo ln -s /export/install/linux/repository/centos/6/x86_64/jre-7u45-linux-x64.rpm /export/install/linux/repository/centos/6/x86_64/jre.rpm

Notice that we create a default jre.rpm symbolic link. With this, we don't need to update our post.install.sh script after each Java update. We just need to change the symlink.

I also like to drop the rssh rpm in there. The home page does not provide them. Just do the same as for the JRE.

Client Machine BIOS Setup


Next thing to do is to go in the client machine's BIOS and make sure the boot order is set to CD/DVD, then local hard disk and then PXE. If it's PXE first, then once we KickStart the server, it will reboot and do the PXE boot again. Which is a KickStart infinite loop! The other thing we need to make sure is that PXE is enabled on the Network Interface Card (NIC) we plan to use.

Now, from the kickstart server, open a shell and hit this :

sudo tail -F /var/log/messages

Once this is done, let's boot the client machine, hit PXE boot and see what happens?

Your client machine should boot via PXE, get it's IP via DHCP then issue several TFTP requests to get it's kernel, initrd and KickStart configuration. Then it's going to pull lots of files via HTTP and then reboot.

Once it has rebooted, you should have a new server taylored for your own environment!

Next Steps


Once the client has booted, connect to it via SSH and create it's Kerberos setup. Don't worry about the NFS error. It's normal at this point because we configured the autofs daemon to fetch the NFS mount tables from OpenLDAP. But the autofs to LDAP authentication is done via Kerberos. But at this point, the client does not have his final Kerberos config.

ssh oxygen.company.com
sudo kadmin
kadmin> addprinc -randkey host/oxygen.company.com@COMPANY.COM
kadmin> addprinc -randkey autofsclient/oxygen.company.com@COMPANY.COM
kadmin> ktadd host/oxygen.company.com@COMPANY.COM
kadmin> ktadd autofsclient/oxygen.company.com@COMPANY.COM
kadmin> exit
sudo /etc/init.d/autofs stop
sudo /etc/init.d/autofs start

Now you should have a working autofs daemon. Try it by simply going into your own directory.

cd && pwd

Thant's it! :)

New Client Setup


Now that we have a working KickStart and Configuration setup, we should use it to setup new machines. To do so, the only thing you need to change is the /export/install/linux/kickstart/centos6.ks file. Make sure to edit the network part of it so that you don't configure two systems with the same hostname and IP address.

Let's hope it works for you as it does for me :)
Posted by at No comments:
Labels:
Subscribe to: Posts (Atom)