Crownpeak: Statement on Meltdown and Spectre Vulnerabilities

On January 3rd 2018, engineers from Google’s Project Zero team released details regarding a new class of vulnerabilities that exploit design flaws in nearly all superscalar CPUs. This document describes Crownpeak’s current and anticipated future response to these newly-announced vulnerabilities.

So far, 3 variants of the vulnerability are known. The first, referred to as Meltdown, is present in all Intel and some ARM chips, as recently announced by Apple. The second and third impact all CPUs that support speculative execution, which means that most CPUs manufactured over the last 20 years are vulnerable. The papers describing the technical details of the attacks can be found at https://meltdownattack.com. We highly recommend reading the papers in order to understand the details of the attacks.

Crownpeak has already launched efforts to mitigate these vulnerabilities. These include working with our cloud service partners to ensure that our entire stack is protected, since these vulnerabilities require not only operating system patches but also modifications to the hypervisors, which underpin all modern cloud service architectures.

Notwithstanding these efforts, however, we do not believe that the patches so far available will provide a complete mitigation (see Spectre FAQ below). It’s important to note that this situation is evolving rapidly, with new mitigations and attack vectors being announced regularly. As a result, Crownpeak may issue further guidance as more information becomes available. We are confident, though, that our response so far represents best-available industry practice.

Below is an FAQ that addresses some of the most common concerns expressed by our customers so far. We will update this list as the situation evolves.

As of January 8, 2018:

Frequently Asked Questions:

What has Crownpeak done to protect its systems and my data/website?

Crownpeak has already updated all production systems to include the recent Intel microcode update and hypervisor updates where applicable. This ensures separation between customer environments and prevents one customer from potentially reading the data of another customer via a hypervisor exploit. It also makes the new microcode instructions available for additional software mitigations as necessary.
We are in the process of applying corresponding operating system updates and expect to complete this exercise within the next ten (10) days.

What action do I need to take?

None, with one exception (see below). Crownpeak provides a fully-managed Software-as-a-Service (SaaS), which will require no customer action during the remediation exercise.
Exception: Crownpeak DXM includes web hosting as part of its subscription. For customers taking advantage of this bundled web hosting service, no action is necessary, since this will be remediated as part of Crownpeak’s overall remediation program. However, Crownpeak DXM does not compel customers to use the bundled web hosting service. Some customers elect to use a third-party web hosting service or provide web hosting from their own data center facilities. In those circumstances, customers must conduct their own impact analysis and remediation arrangements, since Crownpeak has no responsibility for, or access to, that infrastructure.

Will I (or my customers) experience any down time?

No, with two exceptions (see below). Crownpeak services are deployed as highly-available shared-nothing configurations that allow maintenance activities such as this to be executed as rolling remediations, in which redundant tiers of infrastructure are updated in sequence, thus ensuring continuous service availability throughout the entire process.
Exception: For Crownpeak DXM customers using third-party or their own in-house web hosting infrastructure (see above), it will be necessary to consult with those operators to determine the likelihood or necessity of down time.
Exception: The web hosting for a small number of long-term Crownpeak DXM customers is deployed using an older architecture that has since been superseded with a more resilient design. For those customers, an upgrade to the newer architecture will be required before a downtime-free remediation can be carried out. Crownpeak will be contacting impacted customers to discuss preferences and options.

I’ve read that Spectre is much more difficult to mitigate than Meltdown. How is Crownpeak affected and what are you doing about it?

Spectre attacks involve exploiting timing differences in cached vs un-cached data after a speculatively-executed operation loads data from memory. To avoid Spectre-style attacks, therefore, software must either take precautions to ensure that speculative execution does not leave memory fetches in the cache after a rollback, or prevent speculative execution altogether.
Crownpeak has completed an analysis of our systems and has determined that there may be a possible vector for a Spectre-type attack, although actual execution of such an attack would likely not be feasible due to the high and variable load on the affected systems, making exploitation impractical under real world conditions. Even so, we are working to mitigate this to ensure that there is not even a theoretical threat to our systems.
Specifically, we are working with our vendors to get patched compilers that will allow us to mitigate these attacks via the “retpoline” mitigation shared by Google, and that implement the new microcode instructions released by Intel. In the meantime, we will be ensuring that access to high resolution timing information (implicit and explicit) is restricted within the affected environment. By blocking access to high resolution timing information, execution of a Spectre-type attack will be rendered impossible.

I’ve read that mitigating these vulnerabilities will impose big performance penalties. What slowdown can I expect?

Crownpeak has already identified which parts of the Crownpeak infrastructure are most likely to see performance impairment as a result of these remediations, and we have deployed additional hardware to compensate for any issues. We do not foresee any performance degradation as a consequence of this remediation.

What if I want more information?

For more information about the technical nature of the announced threats, please consult your own trusted security resources. Aside from explaining what we’re doing to rectify the issue, we cannot provide specific advice, opinion or guidance on the nature of these threats or how customers should respond.

For more information about how Crownpeak is executing its remediation plan, and the plan’s potential impact on your Crownpeak service, please contact your Customer Success Manager.