Live Patching Linux Kernel Vulnerabilities with AWS and KernelCare

By Jim Jackson, President and CRO at CloudLinux

Linux kernel live patching can automatically fix critical security issues in a running kernel without rebooting the server or instance. It protects kernels from vulnerabilities, improves service availability, and reduces sysadmin overhead.

In this post, I will explain how live patching works and how enterprises building complex, Linux-powered service stacks on Amazon Web Service (AWS) can use it to strengthen security and elevate availability.

Keeping your Linux kernel fully up-to-date and safe from exploits is not just for mainstream Linux flavors. KernelCare is an independent live-patching solution that works on many kinds of Linux distributions and kernels, including Amazon Linux 1 and 2, under both virtualized and bare-metal varieties.

KernelCare was developed by CloudLinux, an AWS Partner Network (APN) Advanced Technology Partner.

Why Does the Linux Kernel Change?

The Linux kernel is software in flux:

• In 2018, 4,449 programmers contributed 83,247 individual changes to five major versions (4.15 to 4.20).
• Between January 2016 and December 2018, security researchers discovered 847 vulnerabilities in the kernel.
• From 2008 to 2019, the Linux kernel more than doubled in size, from 10 million to 24 million lines of code.

Every release of the Linux kernel code after the first in 1991 seeks to better deliver what users demand of a system: improved performance, lower running costs, and tighter security.

The Linux kernel is open source—a product of the voluntary contributions of tens of thousands of designers, programmers, and testers.

Kernel code changes for two reasons:

• Choice: Users and developers want new and improved features.
• Obligation: Developers continually strive to remove bugs and close security loopholes.

Figure 1 – Linux Kernel Source Lines of Code from 2008-2019.

The Linux kernel development community tracks and coordinates changes, grouping them to form releases of two kinds: major (4.19 to 4.20) and minor (4.13.15 to 4.13.16).

The time between releases varies according to the focus and direction of development, and the nature and severity of exposed software vulnerabilities. Security patches to resolve them become part of your system’s kernel when your distribution’s vendor releases a stable version of your kernel. But you may wait weeks or months for a patch after the vulnerability has become public knowledge.

Alternatively, if you have technical expertise in Linux development, and can test a self-built kernel to your own satisfaction, you can download the kernel source code, compile, and install it.

Or, there is a third way. It’s called live patching.

What is Live Patching?

Live patching is a technique for fixing security issues in a Linux kernel without having to wait for a kernel release, and without the need to shut down and restart servers. Live patching fast-tracks fixes for severe kernel bugs, bypassing official release schedules.

KernelCare is an independent live-patching solution that works on many kinds of Linux distributions and kernels, including Amazon Linux 1 and 2, under both virtualized and bare-metal varieties.

How Did Live Patching Originate?

Although the concept traces back to the earliest days of programmable computation, the story basically begins in 2006 at MIT, where Jeff Arnold is a student and volunteer Linux system administrator. The system needs updating, but he doesn’t want to disturb his colleagues by taking the server offline. Before he can update the machine, though, somebody hacks it, forcing a full system rebuild.

This experience inspired Arnold’s master’s thesis on Ksplice, which is software for patching Linux kernels without restarting them. By 2010, Ksplice Inc. was a thriving Linux live-patching business.

A year later, Oracle bought it, making the service available only for its own brand of Linux. It took another three years for the two other large Linux vendors at the time, Red Hat and SUSE, to come out with their own live-patching implementations. Each company took a different approach.

As with Oracle, they tied their solutions to support contracts for their own Linux offerings. This left an opportunity for KernelCare to offer cost-effective live patching for a wide range of Linux kernels.

What Problem Does Live Patching Solve for EC2 Customers?

System administrators strive to avoid unplanned downtime. They know that server restarts directly affect business. Downtime compromises customer service and jeopardizes company revenue. They know it takes time to negotiate system maintenance windows with other business units. Meanwhile, servers wait, unpatched and vulnerable.

This is a real security risk, and often a violation of compliance policies and best practices. Live patching is the only way to keep Linux kernels updated with the latest security patches without suffering downtime.

Customer Use Case: Affinity Water

Enterprises look to lower costs and increase flexibility with AWS—Affinity Water is one of those companies. They are a UK-based utility that installed KernelCare on a few dozen of their AWS-hosted virtual machines (VMs).

Affinity Water has four different flavors of Linux underpinning the services they need to supply 3.6 million customers. An eager DevOps culture drove them to relieve the burden of routine kernel patching by automating it. They installed KernelCare, being the only live patching solution that works on all their distributions.

Customer Use Case: Efinity Insurance

Making their data center and application clusters more resilient and adaptable was why Efinity Insurance first chose AWS for their online “quote and bind” service offering. A KernelCare customer for over a year, they have grown exponentially on the back of their Java-based software solution deployed on AWS instances running CentOS.

When Efinity Insurance’s clients started asking about their SOC 2 compliance status, they saw that security and availability are two of the five trust services criteria met by using KernelCare to automate Linux kernel patching.

How Does Live Patching Work?

KernelCare’s approach to live patching has three components:

• A patch server stores patches for each kernel version and architecture type. This can be the central KernelCare server, or your own in-house server.
• A small agent program runs in the background, checking a patch server for patches at configurable intervals.
• When instructed by the agent, a kernel module performs the patching, handling the logic and mechanics of pausing and restarting the kernel’s processes.

Here is the process, simplified:

1. When the agent sees an available patch, it downloads it and gives it to the KernelCare kernel module. This module is a kernel plugin and acts as part of the kernel itself.
2. The module takes over, suspending all processing (except its own) to make an inventory of active processor threads.
3. The module loads the new binary code into privileged kernel space and remaps affected entries in the thread inventory.
4. The kernel restarts and processing continues.

From the perspective of a user or server process, the kernel never stops. Patching happens so quickly that applications are unaware of the change.

What Are the Disadvantages of Live Patching?

Live patching is for critical security problems, or for when you have to fix vulnerabilities in isolated portions of code without affecting functionality elsewhere. If a fix affects many parts, or alters data structures, live patching is difficult or best avoided.

Not all kernels support live patching, because each live-patching vendor implements their solution differently. Some use core kernel functions, while others use a mix of open- and closed-source approaches. There is no generic, multi-platform solution for all possible Linux variants.

Live patching is easy; making patches is not. The tools and source codes are freely available. Anyone can build and install live patching software for their chosen Linux platform.

For all live-patching approaches, writing patches is technically difficult. You need a detailed knowledge of the kernel source code and its programming models and conventions. It is up to you to test patches thoroughly before using them.

Summary

Automatic software patching is not yet seen as a security benefit, even though out-of-date software is the cause of many recent cybersecurity failures.

As companies and their staff become more security-aware and, in some cases, legally accountable for security breaches, system administrators are starting to see automated Linux kernel live patching as the missing link in their system security profiles.

Despite the disadvantages, a system with live patching is safer than one without.

The content and opinions in this blog are those of the third party author and AWS is not responsible for the content or accuracy of this post.

.

CloudLinux – APN Partner Spotlight

CloudLinux is an APN Advanced Technology Partner. KernelCare is an independent live-patching solution built by CloudLinux that works on many kinds of Linux distributions and kernels, including Amazon Linux 1 and 2, under both virtualized and bare-metal varieties.

Contact CloudLinux | Solution Overview | AWS Marketplace

*Already worked with CloudLinux? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.