News & Blog

Security Patching Automation

News & Blog

Selecting a hosting provider that supports security patching automation

It can be difficult to maintain your Content Management System (CMS) with the latest and greatest security patches while developing new features and publishing new content. The manual process works, but it is not scalable and cumbersome.  But what does it take to move from manual to fully automated or even just semi-automated patching? And even when semi-automated patching can be done, how can we least impact the system, the users, and minimizing downtime?  Setting up maintenance windows and hoping for the best during each patching process is no longer affordable.

At Web Teks, our DevOps team started by investigating various hosting providers, just to name a few, Pantheon, WP Engine, HostGator, GoDaddy, BlueHost, and LiquidWeb.

After some initial investigation, we further investigated Pantheon.   By utilizing its Multi-Dev feature, we were able to replicate the production site, including code, content, and database. This replica environment is an exact duplicate of the production environment and is completely isolated. From this point on, any actions we take with patching can be completed without worrying about the impact on customers’ traffic and data.

In the manual process, we typically log in to a site and check its CMS Core version. We then install a new release that has an impact on all files and folders. Consequently, it may impact the database records. When that process starts, it will be difficult to restore without a good backup and recovery solution.  It is extremely important to have a successful backup solution, in case of upgrading failure.

We use version control to manage our code and configuration. By utilizing Github upstream repository (managed by Pantheon), we are able to easily upgrade the CMS version by merging the changes, i.e. patching upgrade along with any code updates to the repository.   We also have the ability to rollback based on version control.  Once the code is committed, we automatically deploy using git hooks and restart the services with no downtime.

These are examples of Pantheon’s upstream repositories:

After the service has restarted, we typically then login to the site again and run any necessary update scripts.

But these scripts will now be executed automatically after restart.  In addition, we are now able to update plugins and themes together all automatically in one process.  At this point, we have a functional CMS with no downtime.

Upgrade Failures

Some common pitfalls which may impact the success of an upgrade.

  • Plugins / Themes are no longer maintained
  • Custom code within Core / Plugin / Themes
  • Update scripts did not execute properly for database
  • Caching issue within a plugin or hosting provider

How do you check that the site is functioning properly? Checking them manually would require time and ensure attention to detail. You would have to log in and create a new blog post or comment on an existing post. You may also have to check web-server logs to see if anything changed with warnings or errors. You may also have to determine whether your high traffic pages are still delivering appropriately and responsively.

At Web Teks, we automate this process by using a combination of open-source tools to ensure the service is still functioning after a release. After validation we release it (patched environment) to the customer’s production environment.

These are 3 major items during our verification process:

  • Confirming that the latest core version is deployed
  • Verifying that plugins and themes are up to date and enabled
  • Determining if any security vulnerabilities still exist

Useful Tools

The following are various tools that we utilized in our automated testing process:

behat

A PHP framework for auto testing behavior-driven development.  It is a great tool to do functional testing.

https://docs.behat.org/en/latest/

The example above visits the homepage of the site defined in the behat.yaml and checks that the response code is 200, which means a successful response. We can perform all types of interaction with the CMS by interfacing with the extensions.

Provides an interface between Behat and WordPress to enable functionality testing of common components.  If Wordhat does not support a scenario it’s possible to write your own to automate the test.  https://wordhat.info/

Drupal Extension

Enables Drupal-specific functionality for Behat.

https://www.drupal.org/project/drupalextension

For customers, it is very important to verify that the data entry process (posting / commenting)  is still functional. Entering a new post and commenting to an existing post manually would require time and effort. In addition, some organizations may not have enough Quality Assurance Engineers to accomplish the testing process for every site and every release. Once configured, behat testing can be executed quickly and results of pass/fail are known immediately. These test cases can be as simple as data entries or as complex as connectivities to other workflows.

Enables automated visual regression testing by taking screenshots and comparing them against a baseline.

Visual testing can have a great impact on the validation of upgrades. Comparing a minor version change should show very little changes with appearance. Visual testing where we compare pixel for pixel, the patched environment compared to the production environment. This page comparison can be accomplished for multiple versions of the page such as mobile and desktop. If the pixel changes exceed a threshold, we flag it as a failure.

Some common issues that we have seen.

https://github.com/garris/BackstopJS

  • Page with custom forms may fail to load
  • Shift with text that causes the content to no longer line up
  • Invalid text or page layouts

If we identify that the site changes significantly since the patch routine, we need to review them before proceeding with the release. If changes are significant after the update, it could be a sign of issues with custom code or incompatibility issues with plugins/themes.

lighthouse logo

The Lighthouse report is a part of the Google Chrome audit tab and can be used to identify common problems that impact the site’s performance or user experience.

Lighthouse report

Measuring the performance of a site is a critical component of testing. This tells us what the user experiences. Users want quick response times and stability. Lighthouse lets us know whether we are in compliance by scanning the site and checking for best practices as well as giving us a performance score.

By collecting this information, we measure the performance over time and recommend changes to improve the score. We may hold off a release until we identify issues and resolve them. Lighthouse supports other metrics that can be useful, such as Accessibility and Search Engine Optimization (SEO).

https://github.com/GoogleChrome/lighthouse

Lighthouse report

Each of these components can be integrated into a Continuous Integration(CI) pipeline and execute them consistently. By measuring and collecting the data, we can identify discrepancies and resolve them. By storing performance metrics and patching information in a database, we can now identify top sites with security vulnerabilities as well as sites required performance improvement.

Executing those steps manually is time-consuming and error-prone. By creating an automated process, we can expeditiously and effectively patch sites.

In closing…

In summary, these procedures are now automated:

  1. Login to the site
  2. Backup the data
  3. Patch site
  4. Verify site updates both admin and user interface
  5. Should it fail, skilled developers will intervene and resolve issues.

Ability to automate this patching process enables scalability, especially when we have important patches, such as critical vulnerability. 

If you’re tired of managing your WordPress or Drupal instances or are worried about patching and upgrading to ensure security compliances,  contact us and let us help you focus on developing or publishing content for your users!

Our DevOps implement automation that reduces the time to deployment and integrations with open source software to validate changes. One of our Hosting providers, Pantheon facilitates automation and encourages CI/CD as part of the Platform. When applied properly and managed correctly, we can be confident of a successful release.

 

We take processes apart, rethink, rebuild, and deliver them back working smarter than ever before.