Remote Upgrade from RedHat 7.3 to 9

Marion Bates and William Stearns

The usual procedure for upgrading a computer's operating system is, you backup your data, boot off the new OS CD, and click "install" and wait. With RedHat Linux, this tends to work remarkably well. But what if the system you want to update is a remote server, say, six hours' drive away, and your only access is via remote shell? If you have Debian Linux, you simply run "apt-get install dist-upgrade" and wait. But even on a non-Debian system, it can be done. The process is a tad more complex however. ;)

This is a rough guide -- your procedure will vary depending on which packages you have/want, and if you tend to install things from source, this procedure will be of little help to you.

NOTE: Nov 14, 2003: Someone has reported a segfault problem with RPM itself after upgrading the bash package. We are investigating, but consider yourself warned.

1. BACKUP EVERYTHING FIRST!

2. Open four terminal windows as root. DO NOT CLOSE OR LOGOUT FROM NOW ON, OR YOU WILL NOT BE ABLE TO LOG BACK IN.

3. Go into the mirrors dir:

	cd /pub/mirrors/redhat/pub/redhat/linux/9/en/os/i386/RedHat/RPMS/

4. Run this script (copy and paste). You'll need its output later:

	( for X in *.rpm ; do rpm -qp --provides $X ; rpm -qpl $X ; echo "==== Above provided by $X" ; done ) > ~/PROVIDES

5. Start with glibc. Try to install the *.i686.rpm packages whenever possible. But if they don't work, go with the i386 ones.

	rpm -Uvh glibc-2.3.2-11.9.i686.rpm glibc-common-2.3.2-11.9.i386.rpm glibc-devel-2.3.2-11.9.i386.rpm binutils-2.13.90.0.18-9.i386.rpm

If that works, move on to another package.

	rpm -Uvh bash-2.05b-20.i386.rpm

Hey! This is going to be a breeze. Let's do openssl next:

	rpm -Uvh openssl-0.9.7a-2.i686.rpm openssl-devel-0.9.7a-2.i386.rpm openssl-perl-0.9.7a-2.i386.rpm

Uh-oh...unsatisfied dependencies. This is where it gets hairy. If the format of the message is

	newer-foobar >= 3.5.2 is needed by new-barfoo

	an-old-version-of-foobar is needed by an-old-version-of-barfoo

then look for a RH 9 package called foobar-x.x.x.i386.rpm or whatever, and add it to the command line and try again. You will get this sort of message in a lot of cases where you have foo-package and foo-package-devel -- each needs the same version of the other, so you need to upgrade them simultaneously.

If the format of the message is:

	foobar is needed by barfoo

with no version info, then barfoo is probably not its own package, but rather is included in another package. This is where your PROVIDES file comes in handy. In one of your other terminals, type:

	less ~/PROVIDES

Search for the dependency it asked for by typing slash-depname-return:

	/
	depname
	(return)

when you find it, then search for the next occurrence of four dashes:

	/
	====

The package listed on that line provides that resource. Add that package to command line and try again.

Be careful about searching the PROVIDES file for names with non-alphanumeric characters -- those may get interpreted by less, and you won't find your filename. Try searching for partial name in those cases, or figure out how to quote/escape the funky characters.

Repeat those steps about four hundred times until it stops whining about dependencies. You will probably end up with a command line not unlike this, at which point you will begin to wish for a quick death. But hang in there.

If, along the way, something complains about version numbers and thinks your old version of foobar is newer than the one you KNOW to be newer, then you can add --force to the command -- but do this only if the version mistake is the SOLE REMAINING ERROR (in other words, resolve any other dependencies first):

	rpm -Uvh --force blahblah.rpm...

When it FINALLY succeeds, go on to the next package...and repeat. It won't be nearly as bad, since you've already installed so many libraries etc.

When you think you're finished, you aren't. :) Run:

	rpm -qai | less

and upgrade everything you care about that has an install date earlier than today.

Additional RPMs I ended up installing.

And if you want to stick with redcarpet:

	cd /pub/mirrors/ftp.ximian.com/redcarpet/redhat-9-i386/
	rpm -Uvh rc-1.4.1-0.ximian.6.1.i386.rpm rcd-1.4.1-0.ximian.6.1.i386.rpm 
	/etc/init.d/rcd (re)start
	rc channels
	rc sub redhat-9-i386
	rc sub redcarpet
	rc refresh
	rc update -y

Now you have to merge some key config files. RPM is smart, and doesn't overwrite your old config files with new versions; rather, if it sees an existing config, it installs a clean new config called "nameofconfig.rpmnew." In many cases, your old config will work just fine, and you needn't do anything. But you should at least compare the old and the new (try "diff oldfile newfile") to see if there are major syntax changes and such. In the case of Apache, you definitely have to use the new file, or the server will not start. Check the readmes for the program in question.

These are some of the config files I had to merge with new rpm versions:

	/etc/httpd/conf/httpd.conf 
	/etc/httpd/conf.d/php.conf
	/etc/php.ini
	/etc/inittab
	/etc/group
	/etc/aliases
	/etc/ssh/sshd_config
	/etc/rndc.conf
	/etc/pam.d/other
	/etc/pam.d/system-auth
	/etc/mail/sendmail.cf

(In the case of sendmail, I moved my old sendmail.mc and reinstalled the sendmail-config package which made a brand-new sendmail.mc. I then edited that for my server's needs, and ran m4 sendmail.mc > sendmail.cf to make a new sendmail.cf.)

Try

	cd /etc
	find . -iname *.rpmnew

to find others.

Other notes/tips/caveats:

Give yourself plenty of time for this process. It took me the better part of a day, plus a bit of time to resolve some fallout discovered later. It's a tedious, iterative process, and interruptions make it worse. But in the end it worked nicely.

When you're all done installing packages and tweaking configs, run /sbin/chkconfig (or manually check the /etc/rc.d/rc3.d/ entries) to make sure all the right services are set to start automatically (or not) the way you want them to. I didn't expect the upgrade to mess with those, but it did. httpd and spamassassin were among the ones which got set to NOT start up on boot, and of course I didn't find out til much later, the hard way. :\

Apache v. 2 changed some config file formats and locations. They make more sense now -- each web-related doodad (mailman, php, squirrelmail, etc.) has its own config file under /etc/httpd/conf.d/, and /etc/httpd/conf/httpd.conf is really just about Apache itself -- but it's different enough that you will need to manually merge whatever customizations you've done with your old webserver setup. In my case, that entailed copying and pasting the VirtualHosts sections from the old conf to the new one -- the syntax for that seems to be unchanged. Also check things like settings for whether to display directory indexes, users' ~/public_html dirs if you use those, access controls, etc.

This may have been the case before, but I only noticed after the upgrade: Sendmail REALLY CARES about the file modes on users' .forward files. If the mode is anything other than 644, the .forward file will be ignored. If you've got a lot of users, you can do this (as root):

cd /home/
find . -iname ".forward" | xargs chmod 644

This next one was probably due to me being sloppy, but my /etc/ftpaccess file somehow got reset to defaults, such that ftp users only had access to their home dirs. I had to re-comment-out the "guestuser *" line. Leave it alone, though, if you do in fact WANT your users chrooted to their home directories. See the built-in comments in the ftpaccess file for more info.

If you're running Mailman, it also underwent some changes; now there is a mailman daemon. I didn't know that, and spent a good two or three hours fiddling with mailman's config files, modes, permissions, symlinking everything to everything else, uninstalling, reinstalling, etc. ad nauseam to try to get it working, THEN noticed /etc/init.d/mailman. Heh.

I had at some point fouled up modes/perms on the /var/spool/whatever directories related to mail (probably while trying to do the sendmail upgrade under 7.3). The end result was that the local "mail" function would fail constantly. In the end, I erased sendmail, deleted relevant directories (backed up users' inboxes first), and reinstalled sendmail, and that seemed to do the trick. I'm pretty sure this was my own unique screwup, but if you have any trouble with command-line mail after following any of these howtos, PLEASE let me know (mbates _at_ whoopis _dot_ com) and we'll fix your settings -- and put a big red warning in the howto. :)

Another thing which may be totally unrelated to this upgrade: My log rotation got fubar'd. Still working on fixing that. But look out for 200 MB logfiles filling up your root partition. NOTE: Turns out that this is not a RedHat thing but rather a UML filesystem thing. If you're not using UML, or you have no idea what UML is, read no further. :) If you ARE though, then here's a bandaid (make sure you have the "crontabs" package installed):
- Move the logrotate script OUT of /etc/cron.daily and INTO /etc/cron.weekly.
- Edit the logrotate script such that it says:
```
/usr/sbin/logrotate --force /etc/logrotate.conf
```
  This will cause your logfiles to be rotated once a week regardless of their size. If your logs fill up too fast for that, then you will need to find another aolution or modify this one.