Just as intrusion protection can minimize but not eliminate risk, rapid detection and automated responses can minimize, but not completely eliminate, the impact to affected systems and data. A comprehensive plan against cyber attacks – whether malware, ransomware, virus outbreak, or other form of malicious activity – must necessarily include steps to minimize the risk of an outbreak in the first place, and a rapid-response plan to limit the scope and impact when an attack does occur. Beyond that, there must be measures in place to ensure that any impacted systems and data can be recovered.
Data never stands still in an enterprise environment, and a single Qumulo cluster can host petabytes of unstructured data. In a relatively large enterprise where daily change rates are less than 1%, a 5PB Qumulo cluster could see up to 50TB of new and altered data per day. Since data is constantly changing, a data-recovery plan needs to minimize the amount of data that can be lost to a malware outbreak.
As such, a recovery strategy can be complex. It should include all the systems and data that might be affected by a cyber attack, and have mitigation plans in place that address all possible impacts.
It is beyond the scope of this paper to provide a complete recovery plan. Since every enterprise operates under unique circumstances – data volume, change rates, retention and recovery objectives, and other considerations and constraints – every recovery strategy will also be unique.
This section of the document provides an overview of the Qumulo controls and features that can be leveraged to ensure timely recoverability of any impacted data.
Snapshots
Snapshots on a Qumulo cluster can be used in several ways to protect the cluster’s data. They can be used alone for quick and efficient data protection and recovery. A snapshot of the live data on one Qumulo cluster can be replicated to a secondary cluster or to S3 storage. Alternatively, a Qumulo snapshot can be paired with a third-party backup solution to provide effective long-term protection (with more robust version control for changed files) against data loss.
Snapshot basics
By itself, a Qumulo snapshot is a very efficient method of protecting data. A snapshot can be taken at any point in time, either according to a fixed schedule, or on-demand as needed. Once taken a snapshot consumes no space initially.
A snapshot preserves everything in the file system – file data, directory entries, creation and modification times, permissions, etc. As files within the snapshot change over time, new data is written alongside the original version, and new entries are written in the file system identifying each version of the same file.
If a file or directory needs to be rolled back to a previous version, it can be recovered from a snapshot. Since each snapshot is immutable (read-only), a potential malware or ransomware will not be able to change the data within a snapshot. Snapshots can be further protected by locking, which prevents a snapshot from being deleted until a set period of time has passed.
Snapshot policy management
In order to automate a scheduled snapshot, a storage administrator must first define a named snapshot policy defining the parameters and lifecycle of the snapshot. These include:
- Target directory – this can exist at any hierarchical level within the file system.
- Schedule – can be configured using a wide range of settings, including hourly or daily; on one or more days per week, or more/less frequently as needed.
- Naming format – used to determine how to ensure each snapshot version has a unique yet identifiable name to facilitate recovery later if needed
- Lifecycle – determines whether a snapshot is allowed to expire automatically, or must be manually deleted.
- Locking – if enabled, prevents a snapshot from being deleted before expiring.
On-demand snapshots
Besides scheduled snapshots, Qumulo also supports the use of on-demand snapshots, which can be taken at any time from anywhere in the file system. As with scheduled snapshots, an on-demand snapshot can be set either to expire automatically, or to be retained (with optional locking) until an administrator deletes it.
As mentioned earlier in the document, on-demand snapshots can be automated using the Qumulo API, and programmed into the SIEM or IDP platform in response to a detected security event. For more information on managing scheduled, on-demand, and automated snapshots, please refer to the Snapshot management section of the Appendix at the end of this document.
Snapshot Locking
While a snapshot is, by definition, immutable – i.e. it provides a historical reference for the state of the data at a particular point in time, rather than the actual data itself – it is not invulnerable to external tampering. A rogue administrator, or an external actor using a compromised user account with elevated privileges and access to the web UI, CLI or API, can either change the snapshot’s expiration time so it’s purged prematurely, or delete the snapshot outright.
To provide added protection, administrators can apply a “lock” as part of the password’s policy settings. If enabled, snapshot locking prevents the alteration or deletion of a snapshot – even by a storage administrator – prior to its expiration time.
Enabling snapshot locking
To use the snapshot locking feature, an administrator must first generate an asymmetric cryptographic key pair using a Qumulo-supported key-management service (KMS), either through their preferred KMS or open-source tool, or using the Qumulo CLI to generate one directly from the cluster. The public key is installed on the cluster – manually if the key was generated externally, or automatically if the key was created through the CLI – and the private key must be secured using the customer’s own established security practices.
Managing locked snapshots
Under ordinary operating circumstances, locked snapshots will be allowed to expire on their own, and administrators can leverage them as necessary in the event of a cyberattack, secure against even administrative tampering.
If a customer wants to delete a locked snapshot prior to its expiration date – e.g. to reclaim space on a nearly-full cluster – they will need to generate a request on a separate system using the private key, then upload the signed and authorized request to the cluster before the locked snapshot can be adjusted.
Snapshots with replication
For additional data protection, snapshots can be paired with Qumulo’s replication services to ensure that at least one copy of all data is hosted in secondary storage from which it can be recovered if data on the primary storage instance is lost.
This secondary storage instance can be another Qumulo cluster – whether on-premises, at a second data center, or hosted in one of the public cloud providers (Amazon Web Services, Google Cloud, or Microsoft Azure) – or an S3 target.
Qumulo replication
Qumulo’s built-in replication service can copy data at scale between any two Qumulo storage instances. Besides protecting data against cyber attack, a secondary location with another Qumulo cluster can also serve as failover storage in the event of a site-level outage.
Since Qumulo storage can be deployed anywhere – in a second data center or in any of the public-cloud platforms (Amazon Web Services, Google Cloud, and Microsoft Azure) – and delivers the same services regardless of location, replication can be configured to run in any direction between any two Qumulo endpoints.
Continuous replication
This form of replication simply takes a snapshot of the data on the source Qumulo cluster and copies it to a directory on a target cluster. As long as the replication relationship is active, the system scans any modified files to identify and copy only the specific changes to the target.
While this does create a copy of the primary dataset on a second cluster, it is not recommended for data protection; since any data corruption or loss on the primary source will also propagate to the target.
Snapshot-based replication
With snapshot-based replication, snapshots are also taken of the target directory on the secondary cluster. Once a replication job has been completed, a new snapshot of the target directory is created, ensuring consistent data on both clusters, as well as maintaining a change log for each file on the target.
These snapshots can be set to use the same expiration as the source snapshot policy, or can be configured with longer expiration times to provide a longer recovery window in the event of a malware attack.
Replication to/from S3 storage
Where the native replication service provides the ability to copy snapshots between two Qumulo clusters, Qumulo can also copy data in either direction between the local file system and an AWS S3 bucket.
Within the AWS S3 bucket, all files are written in native S3 format, with no gateway required. Subsequent copies are incremental so that only changed files are copied – another reliable way to protect Qumulo data offsite and away from an on-premises attack.
Depending on the workload or data change rates it is possible to copy consistent snapshots to an S3 bucket and use AWS versioning to track the multiple versions of files in the bucket. Once data has been written successfully to AWS S3 storage, intelligent tiering can be leveraged to move older files to AWS Glacier or Glacier Deep Archive Storage, providing a cost-effective data storage solution for data that is not in active use.
Like all other tasks on Qumulo storage, data movement to and from AWS S3 storage can be automated by using the API or CLI.
External backup integration
For longer-term data protection, and the ability to maintain a longer version history for critical files, Qumulo integrates with any file protocol-based1 backup solutions.
Some vendors, such as CommVault and Atempo, use the Qumulo API to compare snapshots and identify changes, enabling them to take instantaneous incremental backups without the need for a tree walk. Another advantage to file-based backups is that the backup image is platform agnostic, meaning that if necessary, data can be restored to any storage target.
This blog post has provided a high-level overview of Qumulo’s own features, and its interoperability with 3rd-party data protection solutions to minimize the impact of a malware attack. A holistic protection strategy needs additional strategies and practices to ensure that any attack that occurs can be quickly discovered and contained, and that any lost data or impacted services can be quickly restored.
Learn more
- Part 1: How to Use Qumulo’s Built-in Security Controls for Data Protection
- Part 2: Detecting Ransomware Attacks in Real Time with Qumulo
- Download the latest Security Architecture and Practices white paper
Contact us
- Click here to schedule a meeting.
- Subscribe to the Qumulo blog for customer stories, technical insights, and product news.
