Fighting Crackers at Christmas

This marks the third Christmas week in ten years where I spent an absurd amount of time fighting off bad guys on my server(s). It makes me miserable to have to spend time doing this when I could be out caroling, or skating, or socializing.

My "good will to all men" today excludes spammers, who all deserve to rot in hell, after first being drawn, quartered, and spitted by a lynch mob of angry sleep deprived system administrators.

I first noticed my box relaying spam email a few weeks back for about an hour before I caught it. Ironically I had been listening to a cybersecurity show on NPR featuring expert gary warner at the time.I updated it, rebooted, tracked my logs for a while, and hoped whatever vulnerability it was, had been fixed by the nearly 1GB of updates (I had been remiss on this very stable server on keeping it updated)

All was quiet until a few days ago when all hell broke loose, and zillions of emails started emanating from my server for about 15 minutes. I caught it and basically shut the box down while I enjoyed some Christmas cheer. I'm glad my users are tolerant and understand that I am not a youtube or commercial hosting site.

I've spent most of the morning trying to trace down the point of entry into my system, and have found extensive logs that show it to have been an apache centered attack, most likely php (not certain)

I've had to shut down my email entirely. I've left the web server running as more or less a spam trap, and as a stopgap (it doesn't appear like they got root) for the sites I host.

I see that another attacker HAD got root, briefly, back in july 24th (some sort of asterisk vulnerability), but I'd stopped that one cold, and there doesn't seem to be any relation to the attack I'm fighting now.

Still, I figure I have been beaten in at least 3 places and that the only safe thing for me to do is to shut down the server involved and rebuild from scratch. I'm doing that now. Should only take 2-3 days of my time. !@#!@#!@#!@!!

On the bright side, I never accept public keys from public servers on my private boxes, so I'm (and my clients) are safe that way (the attack can't be spread via ssh), but I am disgusted with the state of the internet today. I have enjoyed being a part of it for a long time, but the hassle of running my own server(s) far outweighs the benefit, particularly while I'm sitting here on Christmas day, boggling at the problem.

It's one thing to get p0wnd and used as a spam relay, although that hurts a lot, but according to my logs, the initial attack, on December 3rd, also attempted to completely destroy my system. I have a lot of precious things on this box, including all my songs, and I'd have hated losing all that.

--16:23:03--  http://www.euphoria.gr/forum/files/test.pl
Resolving www.euphoria.gr... 89.234.44.185
Connecting to www.euphoria.gr|89.234.44.185|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32719 (32K) [text/plain]
Saving to: `test.pl.1'

     0K .......... .......... .......... .                    100% 72.4K=0.4s

16:23:04 (72.4 KB/s) - `test.pl.1' saved [32719/32719]

rm: cannot remove `/var/log/lastlog': Permission denied
rm: cannot remove `/var/log/wtmp': Permission denied
... a zillion recursive rm commands that would have destroyed my box elided ...

Whatever "test.pl" was, it's no longer on the host from where it was obtained, and not obvious if the site it was obtained from was implicated, or merely another victim.

Swimming through the fetid sea that email has become is one of the saddest, most disturbing things I've ever had to do on any holiday. It was bad a few years ago, now it is outright horrible. This is the kind of stuff my email server rejects out of hand:

Dec 16 04:18:39 ns1 postfix/smtpd[15044]: warning: 200.103.97.142: hostname 200-103-97-142.gnace300.ipd.brasiltelecom.net.br verification failed: Name or service not known
Dec 16 04:18:39 ns1 postfix/smtpd[15044]: connect from unknown[200.103.97.142]
Dec 16 04:18:41 ns1 postfix/smtpd[15044]: NOQUEUE: reject: RCPT from unknown[200.103.97.142]: 504 5.5.2 : Helo command rejected: need fully-qualified hostname; from= to= proto=ESMTP helo=
Dec 16 04:18:41 ns1 postfix/smtpd[15044]: lost connection after DATA from unknown[200.103.97.142]
Dec 16 04:18:41 ns1 postfix/smtpd[15044]: disconnect from unknown[200.103.97.142]

I have a friend who has the luxury of just arbitrarily rejecting mail from most places outside the US...

Spam is one thing, attacks against your web servers are another. I mean, I have just about every single form of security turned on on the web server, so I catch and hopefully defeat attacks, but half the time I'm not even sure what the heck they are doing. Take this one for example:

admin.transconf.net-access_log.2:217.33.12.98 - - [15/Dec/2007:14:46:10 -0500] "
POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0" 406 342
admin.transconf.net-access_log.2:217.33.12.98 - - [15/Dec/2007:14:46:10 -0500] "
CONNECT http://lti-mail01.ltinetworks.com:25 HTTP/1.0" 400 309
admin.transconf.net-error_log.2:[Sat Dec 15 14:46:10 2007] [error] [client 217.3
3.12.98] mod_security: Access denied with code 406. Pattern match "^$" at HEADER
("USER-AGENT") [severity "EMERGENCY"] [hostname "lti-mail01.ltinetworks.com"] [u
ri "http://lti-mail01.ltinetworks.com:25/"]
audit_log.1:Request: lti-mail01.ltinetworks.com 217.33.12.98 - - [15/Dec/2007:14
:46:10 --0500] "POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0" 406 342 "-"
 "-" - "-"
audit_log.1:POST http://lti-mail01.ltinetworks.com:25/ HTTP/1.0

And the various rbl filters enabled stop an extraordinary amount of spam, but still quite a few get through...

This (below) looks to be one of the problems. The php script in question is definately an re-emailer, but how it's getting executed on my system is a question....

www.teklibre.com-error_log:[Sun Dec 23 15:53:37 2007] [error] [client 82.128.20.
28] PHP Notice:  Undefined variable: emaillist in http://chelseacharms.com/cart/
small/longthing.txt?.inc on line 160
www.teklibre.com-error_log:[Sun Dec 23 15:54:23 2007] [error] [client 82.128.20.
28] PHP Notice:  Undefined variable: emaillist in http://chelseacharms.com/cart/
small/longthing.txt?.inc on line 160
www.teklibre.com-error_log:[Tue Dec 25 01:23:58 2007] [error] [client 82.128.18.
1] PHP Notice:  Undefined variable: emaillist in http://chelseacharms.com/cart/s
mall/longthing.txt?.inc on line 160
www.teklibre.com-error_log:[Tue Dec 25 01:24:05 2007] [error] [client 82.128.18.
1] PHP Notice:  Undefined variable: emaillist in http://chelseacharms.com/cart/s
mall/longthing.txt?.inc on line 160
www.teklibre.com-error_log.1:[Sun Dec 16 08:00:35 2007] [error] [client 83.229.5
.133] PHP Notice:  Undefined variable: emaillist in http://chelseacharms.com/car
t/small/longthing.txt?.inc on line 160
www.teklibre.com-error_log.1:[Sun Dec 16 08:01:36 2007] [error] [client 82.128.2
0.85] PHP Notice:  Undefined variable: emaillist in http://chelseacharms.com/car
t/small/longthing.txt?.inc on line 160
www.teklibre.com-error_log:[Tue Dec 25 01:24:05 2007] [error] [client 82.128.18.
1] PHP Notice:  Undefined variable: emaillist in http://chelseacharms.com/cart/s
mall/longthing.txt?.inc on line 160

Aha, I think I found the issue in teklibre's include statement that is doing the code injection, but is that the real source of the problem? Gah... fixed. Now do I just clean out the 4000+ spam mails waiting to be sent and restart the mail server? Dunno.

The spam that is being attempted to be forwarded seems to be an all-out phishing attack against two banks, RegionsNet Online Banking, and Halifax Online Banking.

And with that much documented, and postfix turned off, the patient is still sick, but will live a while, it's time for some christmas dinner, and I'm going to rethink my commitment to staying alive on the internet. It's a full time job that I'd rather someone else did.

Update: I get worried when I start seeing really odd queries from clusters like that of Amazon.com. It's bad enough when there are bots on small nets, but on the backbone is a real problem...

[client 67.202.18.57] ModSecurity: Access denied with code 400 (phase 2). Match of "rx ^[a-z]{3,10}\\\\s*(?:\\\\w{3,7}?\\\\:\\\\/\\\\/[\\\\w\\\\-\\\\.\\\\/]*)??\\\\/[\\\\w\\\\-\\\\.\\\\/~%:@&=+$,;]*(?:\\\\?[\\\\S]*)??\\\\s*http\\\\/\\\\d\\\\.\\\\d$" against "REQUEST_LINE" required. [id "960911"] [msg "Invalid HTTP Request Line"] [severity "CRITICAL"] [hostname "www.transconf.net"] [uri "?"] [unique_id "WI27AAyhxKUAACKHIpIAAAAK"]


57.18.202.67.in-addr.arpa       name = ec2-67-202-18-57.compute-1.amazonaws.com.

But this one has nothing to do with the other issues I've seen today. If you don't do security every day, and then leap into it, then you start jumping at shadows. IF you don't do security every day, and you haven't had dinner yet, you start getting hungry.

Outta here.

Update 2: Back from dinner and dessert. I re-enabled mail on the box. If you got bounces or rejects from me, now you know why. I spent some time reading the CyberSecurity Enhancement Act of 2007, which federalizes spam related crime rather dramatically. I don't think this is the answer, as spam is more of an international problem.

Letters of Marque and Reprisal, now THAT's something I'd dig my teeth into.

Labels: spam

  ¶ 12:16 PM 1 comments

  A Merry Christmas from Achmed

I like to think that laughter is the most healing emotion. Laughter lets us confront our fears and those things that make us fearful. Laughter can give us courage, and ultimately, laughter can drive the bad guys from the room, at least long enough for a little love to show. That's why Achmed and his sidekick, Jeff Dunham, are my men of the year.

Labels: laughter

  ¶ 9:29 PM 0 comments