On 29/06/2015 the server was detected to be sending spam. Even though we have been take measures from updating our server security link. We don’t like spam and we don’t like sending spam. Our provider blocked the port 25 to prevent further spam being sent. For us, whilst we don’t like to be blocked, we don’t want to be sending spam either. In this case our provider only blocked the port rather than the entire server, this means that our website would continue to operate and allowing us to check through the server logs to resolve the issue.
This website only has the following ports open prior to the port 25 block:
PORT STATE SERVICE 21/tcp open ftp 22/tcp open ssh 80/tcp open http 443/tcp open https
This leads to the question, how can a server be sending out spam if the mail port of 25 isn’t even open in the fist place? Logically for spam to be sent and to be detected by our un-managed provider, they would notice this from the network level as they don’t have root access to the server. This means our first point of reference would be the network logs to see its current activity, in addition to this we have been provided with some logs on the spam itself from our provider.
When we checked the network information, we noticed some unusual activity:
netstat -tanpu Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 1 220.127.116.11:32998 18.104.22.168:25 SYN_SENT 6514/apache2 tcp 0 1 22.214.171.124:60185 126.96.36.199:25 SYN_SENT 6514/apache2 tcp 0 1 188.8.131.52:46344 184.108.40.206:25 SYN_SENT 6514/apache2 tcp 0 1 220.127.116.11:42946 18.104.22.168:25 SYN_SENT 6514/apache2 tcp 0 1 22.214.171.124:49332 126.96.36.199:25 SYN_SENT 6514/apache2 tcp 0 1 188.8.131.52:36927 184.108.40.206:25 SYN_SENT 6514/apache2 tcp 0 1 220.127.116.11:57274 18.104.22.168:25 SYN_SENT 6514/apache2 tcp 0 1 22.214.171.124:33001 126.96.36.199:25 SYN_SENT 6514/apache2 tcp 0 1 188.8.131.52:59820 184.108.40.206:25 SYN_SENT 6514/apache2 tcp 0 1 220.127.116.11:56587 18.104.22.168:25 SYN_SENT 6514/apache2 tcp 0 1 22.214.171.124:33848 126.96.36.199:25 SYN_SENT 6514/apache2 tcp 0 1 188.8.131.52:33854 184.108.40.206:25 SYN_SENT 6514/apache2 tcp 0 1 220.127.116.11:33012 18.104.22.168:25 SYN_SENT 6514/apache2 tcp 0 1 22.214.171.124:36932 126.96.36.199:25 SYN_SENT 6514/apache2 tcp 0 1 188.8.131.52:56595 184.108.40.206:25 SYN_SENT 6514/apache2
The apache2 service should be running on port 80 for HTTP or port 443 for HTTPS. We can see that on the logs it suggest that the program is running with the process ID of 6514, we then looked into the process ID to see the list of open files associated to this ID.
lsof -p 6514 apache2 6514 www-data cwd DIR 8,1 4096 214405 /var/www/wp-content/themes/[THEME NAME]/css
The output was quiet extensive, however the most interesting line is the one above, where by it shows the theme CSS directory is currently running a process. We looked into this directory and found php script inside a CSS directory! Afterwards we checked the creation date of this php script and compared it against the server apache2 logs. We was able to identify the source of the issue and cleaned up the server.
Whilst we have resolved the spam issue, we decided to look into the spam emails to see how many spam emails there was and to see their contents.
Total number of spam emails
exim -bpc 124262
We can see that the server tried to send a total of 124,262 email emails.
exim -bp 89m 1.6K 1ZBIi4-0006pW-Bo <> *** frozen *** firstname.lastname@example.org
The message tried to be sent 89 minutes ago with the message size of 1.6k. The message ID is: 1ZBIi4-0006pW-Bo and it attempted to be sent from email@example.com, this particular email account doesn’t actually exist either.
The “frozen” state means the server tried to send an email address but it was unsuccessful. After a few attempts the emails are set in the frozen state an added to the mail queue.
Viewing message header
exim -Mvh 1ZBIi4-0006pW-Bo Debian-exim 101 103 <> 1435998124 0 -ident Debian-exim -received_protocol local -body_linecount 32 -max_received_linelength 103 -allow_unqualified_recipient -allow_unqualified_sender -frozen 1435998124 -localerror XX 1 firstname.lastname@example.org 159P Received: from Debian-exim by example.com with local (Exim 4.80) id 1ZBIi4-0006pW-Bo for email@example.com; Sat, 04 Jul 2015 09:22:04 +0100 046 X-Failed-Recipients: firstname.lastname@example.org 029 Auto-Submitted: auto-replied 060F From: Mail Delivery System <Mailer-Daemon@example.com> 034T To: email@example.com 059 Subject: Mail delivery failed: returning message to sender 049I Message-Id: <E1ZBIi4-0006pW-Bo@example.com> 038 Date: Sat, 04 Jul 2015 09:22:04 +0100
Here we can see the header of the email which tried to be sent.
exim -Mvb 1ZBIi4-0006pW-Bo 1ZBIi4-0006pW-Bo-D This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: firstname.lastname@example.org Mailing to remote domains not supported ------ This is a copy of the message, including all the headers. ------ Return-path: <email@example.com> Received: from www-data by example.com with local (Exim 4.80) (envelope-from <firstname.lastname@example.org>) id 1ZBIi4-0006pL-AF for email@example.com; Sat, 04 Jul 2015 09:22:04 +0100 To: firstname.lastname@example.org Subject: Best Popular Vagra web store X-PHP-Originating-Script: 33:dir.php From: "Sherri Odom" <email@example.com> Reply-To:"Sherri Odom" <firstname.lastname@example.org> X-Priority: 3 (Normal) MIME-Version: 1.0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: 8bit Message-Id: <E1ZBIi4-0006pL-AF@example.com> Date: Sat, 04 Jul 2015 09:22:04 +0100 <div> Best Popular Vagra web store – <a href="http://SpamWebsite.com/">check it out</a> </div>
As we can see in the email, the contents are definitely from a spamming nature. This is also the typical email in which you may receive within your inbox from time to time. In this case, the emails was blocked from being sent in the first place, this means this spam never reached your inbox.
Removing the emails from the system
exim -bp | exiqgrep -i | xargs exim -Mrm Message 1ZBIi4-0006pW-Bo has been removed
The issue for the email spam has been resolved, however we still have the issue where by the server currently still holds 124,262. We will look to clear the mail queue.