It seems to happen this way – Easter weekend (or any other long weekend break) – some of us plan activities, some plan to move home and some…well, you know, relax?? But any of these were the last things on our minds all thanks to a dangerous bug aptly named on social media, DRUPALGEDDON2. In Drupal’s own words “This potentially allows attackers to exploit multiple attack vectors on a Drupal site, which could result in the site being completely compromised,” the group noted in a blog post.
The bug’s official identifier is CVE-2018-7600 and it is one of the biggest patch releases from the organisation since 2014. Having a web interface that is Drupal-based meant that we picked up on this and started preparing for the release from Thursday 29th of March to Friday when the patch was issued. We have 100’s of servers around the world (some dedicated) and thousands of customers. To pull this together was going to take some organisation.
If you are a Rocketseed customer you will know that there are two things we hold critical:
- The integrity of your data and its security
- Unbeatable customer service and support
And being a company with a global presence that reaches from Australia through EMEA and onto the USA, there are multiple time zones to coordinate and manage and it takes time to prepare for the level of patching that had to be done. To say all hands on deck, would be somewhat of an understatement.
And just in case you’re thinking to yourself ‘SIMPLE contact the IT admins or heads and let them know that patches were being applied’ – Whoa there pony! You forget some crucial elements…
First and foremost, timing. We found out the severity first thing Thursday morning. Yes, we were aware that there ‘may’ be a patch coming but not at this scale. To ready hundreds of servers and apply patch of this magnitude was going to take a lot of work for an engineering team, support, comms and account management. Believe it or not, these amazing Rocketseeders have families and plan holidays, bolt-on days before or after breaks etc!
Secondly, there are logistics – getting the VPN access and the Change Control coordination requirements, setting up support teams (this is email technology, things can and do go wrong), getting the infrastructure right and finally making sure that the teams worldwide were able to access elements of the servers globally while customers could be away or otherwise engaged over the break. Ultimately, keeping it as smooth as possible.
Thirdly, communication. How do we update on this? (We are and remain completely transparent with our customers), how do we get a front foot on this without creating panic? Is the vulnerability as severe as they are saying? Thankfully we’re able to ensure communications with touch points globally using centralised distribution lists.
Finally, execution – test the patch, apply the patch, monitor the patch and ensure mail delivery.
…NOT SO SIMPLE.
Did things go absolutely perfectly? No. Could we have done better? Yes. But the fact is that perfection is a rarity, particularly where humans are involved! That being said, plans and protocol will help you get there.
Ultimately Rocketseed as a business was tested to the absolute max this last weekend and from it lessons have been learned…tweak this, tighten that, we could have done that better, why didn’t ansible work as it should etc. As our chairman said, ‘When you’re close to the fire, you don’t see the positives that emerge down the line’. As it turns out he was right. Customer feedback so far has centred around gratefulness; grateful that Rocketseeders pulled together to protect their customers’ interests and security.
Email signatures and branding involve very nuanced technology that requires a big stack to make them work. That being said, what really makes it work is the people behind it – developers, engineers, sys ads, client services, account management and support. As the CEO of this business, I would be remiss if I didn’t say that I learnt from this experience. Why? I simply couldn’t be more proud of the way that teams from across the globe pulled together (over the break), put in place protocol and delivered for thousands of customers out there. Not one complaint, simply absolute focus on keeping the integrity of customers’ security intact before DrupalGeddon2 could run arbitrary code on the CMS core component.
Ultimately, we took a front foot approach and chose transparency with our customers and we believe that is always the right approach to take. Thanks to all of the customers who came back to us. We value your feedback and apply it constructively in every instance. A huge thank you to the Rocketseed staff around the globe who pulled together without fail, it was exemplary and epitomises what makes Rocketseed the company it is today.
