A rootkit has been infecting a number of Linux servers through the month of February. An early trend indicated servers running RedHat/CentOS and CloudLinux as well as cPanel could be vulnerable to whatever exploit was being leveraged to access the servers. One of the tell-tale signs was the detection of /lib/libkeyutils.so.1.9 or /lib64/libkeyutils.so.1.9. The rootkit was believed to be stealing passwords, SSH keys, /etc/shadow from the servers, was able to be used as a backdoor to access the server, and would send spam. While investigation into the vulnerabilities continued, an exploit of SSHd was considered along with other bugs. It was recommended to firewall SSHd and restrict it to select IPs.
Threads on Reddit, cPanel, and WebHostingTalk spread the discussion since February 8th. Yesterday, it seems the investigation found a possible source. While not responsible for all of the infections, cPanel sent out an email to clients that opened a support ticket within the last 6 months. cPanel disclosed in an email to those clients that one of the servers in their technical support department had been compromised. As a result, this apparently provided account logins and root passwords to malicious users to access these servers and then place the rootkit.
Web hosting company WiredTree took precautionary measures to block port 22 until more about the issue was known.
Dear Valued WiredTree Customer,
I am writing you tonight to inform you that we have disabled access to port 22 (default SSH port) on your server as temporary precautionary security measure. Our security team has good reason to believe there is a root-level exploit in the wild for RedHat/CentOS servers as compromises have been reported on WebHostingTalk, Reddit, as well as on our own network and at other providers we have talked to. There have
been a number of similarities in the attacks and that is why we have decided it is best to block this port temporarily until the attack vector is determined. If you require SSH or SFTP access, we can set it up for
you on an alternate port if you open a Grove ticket. We understand this sudden change is an inconvenience and interruption to your work flow, but we believe it is in the best interest of your server’s security at this time.We are watching this issue closely and will be taking further precautionary or preventative measures if needed. Again, we are deeply sorry for the inconvenience. If you require SSH or SFTP access and you were
using it on port 22 (if you never changed it, this is what it would be) we can change the port and restore access for you if you open a ticket. We will roll out updates as soon as they are available.
Yesterday, the company removed the port 22 block when more details were known.
We recently emailed you to inform you that we temporarily disabled access to port 22 (default SSH port) on your server as a precautionary security measure. This block has now been lifted.
Our security team had been following some wide spread reports of root level compromises over the course of a couple of weeks. As time went on more and more were being reported, and we saw a handful on our network. One thing all of the servers compromised had in common was that SSHd was enabled with password authentication. We blocked SSHd temporarily as a precautionary measure, however we have since learned that SSHd was not the actual culprit.
We have been informed by cPanel that one of their servers in their Technical Support department was compromised and after further investigation, we have found that servers that were compromised had a cPanel ticket opened at one point where root level SSH access was given to cPanel Support so they could log in from their support offices. This extends back as far back to tickets being opened with cPanel support in October 2012.
We understand that having this port blocked caused issues for some of our customers and we are deeply sorry that you had to endure such disruption in your work flow. We wouldn’t have gone this route unless it was absolutely in the best interest of our customers. While it was not the actual cause, the block did prevent any additional servers from being compromised.
WiredTree is going to be revisiting our internal policies immediately in how we deal with our partners and their access to client servers in order to minimize the impact of a similar situation in the future.
If you are still having issues connecting to the default SSH port (22) please open a ticket in Grove and we will help you out as soon as possible. We still urge you to change the default SSHd port so please do not hesitate contact our support so they can do this for you.
The email that cPanel sent out yesterday which provided the new information:
From: no-reply@cpanel.net
Sent: Friday, February 22, 2013 12:48 AMSubject: Important Security Alert (Action Required)
Salutations,
You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with “sudo” or “su” for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.
As we do not know the exact nature of this compromise we are asking for customers to take immediate action on their own servers. cPanel’s security team is continuing to investigate the nature of this security issue.
–cPanel Security Team
It is recommended to change the password of all accounts on a server that has contacted cPanel support.
Understanding the source of these infections provides a little relief but it just exemplifies the heightening reliance on hosted services and outsourced support. These elements then introduce new security risks and data to manage outside of your own grasp.