Whitepaper 'FinOps and cost management for Kubernetes'
Please consider giving OptScale aStar on GitHub, it is 100% open source. It would increase its visibility to others and expedite product development. Thank you!
Webinar: Introducing the Hystax Global Partner Program. Register now
Webinar: Introducing the Hystax Global Partner Program. Register now
How much is your company actually spending on AI this month?
Cut it by 60%. Govern every AI prompt and agent. Learn more on optscale.ai
Ebook 'From FinOps to proven cloud cost management & optimization strategies'
OptScale FinOps
OptScale — FinOps
FinOps overview
Cost optimization:
AWS
MS Azure
Google Cloud
Alibaba Cloud
Kubernetes
MLOps
OptScale — MLOps
ML/AI Profiling
ML/AI Optimization
Big Data Profiling
OPTSCALE PRICING
cloud migration
Acura — Cloud migration
Overview
Database replatforming
Migration to:
AWS
MS Azure
Google Cloud
Alibaba Cloud
VMWare
OpenStack
KVM
Public Cloud
Migration from:
On-premise
disaster recovery
Acura — DR & cloud backup
Overview
Migration to:
AWS
MS Azure
Google Cloud
Alibaba Cloud
VMWare
OpenStack
KVM

Disaster recovery testing: types of tests, how to build a DR test plan, and what to measure

Table of contents

Disaster recovery testing is the practice of exercising a DR plan, from a walkthrough up to a real failover into a recovery environment, to confirm that recovery meets the declared RTO (recovery time objective) and RPO (recovery point objective). DR testing shows whether the plan works as written, how long recovery really takes, and how much data is lost along the way.

This guide covers six types of tests, a testing schedule by tier, a ten-step disaster recovery test plan, the metrics that turn a test into evidence, and failover testing automation.

What disaster recovery testing is and why an untested plan is not a plan

A disaster recovery plan is a written set of steps, owners, and targets for restoring IT systems after an outage. IT environments change fast: applications gain new dependencies and service accounts expire. The documented boot order stays the same while the systems change, and everything looks fine until a real failover. Veeam’s Data Trust and Resilience Report 2026, a survey of more than 900 IT, security, and risk leaders, found that 90% were confident they could recover from a cyber incident within their RTOs. In the same report, only 28% of the organizations whose operations or data were hit by ransomware fully recovered all affected data. A test replaces an estimate such as “we should recover in four hours” with a measured number.

Types of disaster recovery tests

Tests range from plan reviews to full production shutdowns. NIST SP 800-34 Rev. 1 (2010) separates two kinds of exercises. Tabletop exercises are discussion-based and require no equipment, while functional exercises have staff perform recovery duties in a simulated environment. The six types below follow common industry practice. A hands-on test is often called a disaster recovery drill.

Test type What it checks Effort Risk to production What it proves
Plan review/walkthrough Contacts, steps, and dependencies; read line by line Hours None PThe document is complete and current
Tabletop exercise A disaster scenario discussed by IT and business owners Half a day None People know their roles and escalation paths
Simulation test Selected recovery steps performed against a scripted scenario 1-2 days Low Teams can run the procedures within a set time
Parallel test Recovery systems started and run next to production Several days Low to medium Systems start and work on the recovery side
Full interruption test Production shut down, business runs from the recovery site Weeks of planning High End-to-end recovery, including failback
Non-disruptive test failover Workloads recovered into an isolated network while production runs Hours once automated Very low, if the test network is isolated Actual RTO, boot order, and data consistency
DR test types plotted by proof of recovery and risk to production

A parallel test and a non-disruptive test failover are close. The test failover is an automated form of the parallel test that modern DR tools provide. They differ in where recovery runs: a parallel test runs next to production, so shared data feeds and networks need careful management, while a test failover runs on an isolated network cut off from production.

For the business side, see how business continuity and disaster recovery strategies fit together.

How often should a disaster recovery plan be tested?

The usual floor is one test a year for every critical system, and another after any major change. Tier 1 systems are tested more often.

A test failover is cheap to repeat and does not touch production when the test network is isolated, so quarterly runs are realistic.

The table is a starting schedule that teams adapt.

Tier Example workloads Test type Frequency Unplanned test after
Tier 1, critical Core databases, payments, customer-facing apps, and identity Test failover and tabletop, plus a full interruption test where the business allows it Test failover quarterly; tabletop and full interruption test yearly Major change, platform migration, failed test, or internal audit finding
Tier 2, important ERP reporting, internal business apps, and file services Non-disruptive test failover Twice a year Change to the application or its dependencies
Tier 3, deferrable Dev and test, archives, and tools that tolerate days of downtime Plan review/walkthrough Yearly Move to new storage or a new hypervisor

Test schedule by workload tier

Some sectors face legal requirements. DORA, applied since January 17, 2025, requires EU financial entities to test their ICT business continuity plans and their response and recovery plans at least yearly, and again after substantive changes to systems behind critical or important functions. SOC 2 (where the Availability criteria are in scope) and ISO 22301 require testing recovery plans, and NIS2 lists disaster recovery among its required measures. Few organizations test often enough. In Quest Software’s March 2026 State of ITDR survey of 650 IT and security leaders, only 24% tested identity recovery every six months, the minimum Quest describes as widely recommended, and another 24% never tested.

Testing during a VMware exit or upgrade

Broadcom lists the end of general support for vSphere 8.0 as October 11, 2027. That date forces a decision: upgrade to VCF 9 or VVF 9, or leave VMware. Either path changes the environment the DR plan was written for, so boot order, network mappings, and licenses must be checked again. For workloads that leave in a data center migration, test failovers before and after cutover show whether the recovery path survived.
See how Hystax Disaster Recovery tests your DR plan without touching production

infrastructure, auto-testing configured in the plan, and failback to production

Thank you for your request!

We will be in touch soon.

We respect your privacy. See our Privacy Policy. You can unsubscribe at any time.

How to build a disaster recovery test plan

A solid disaster recovery test plan defines four things: what is tested, how success is judged, who does what, and where findings go.

The ten step titles double as a DR test plan template and checklist.

  1. Define scope and test level. Pick the systems, their tier, and a test type from the tables above. Add at least one ransomware scenario, with recovery from a clean copy.
  2. Set success criteria from RTO and RPO. Write a pass criterion for each system, for example, running within 60 minutes with at most 15 minutes of data lost.
  3. Map application dependencies and boot order. List what each system needs, such as databases, directory services, DNS, license servers, and firewall rules, and the order in which systems start.
  4. Write the disaster recovery runbook. A DR runbook is the step-by-step recovery procedure. It names owners, expected results, and the place where credentials are kept, and someone who did not write it should be able to follow it.
  5. Prepare an isolated test network. Build a sandbox with no route to production, so duplicate IP addresses and hostnames cannot collide with live systems.
  6. Assign roles. You need a test lead and an owner for each system, plus an observer to log times and a business representative to sign off.
  7. Run the test. Follow the runbook exactly as written, and log start and finish times, every deviation, and every manual fix.
  8. Record the results. The test report covers scope, date, targets, measured values, failures, and open risks.
  9. Close the gaps. Fix the runbook, configuration, credentials, or licenses, and assign an owner and due date to every finding.
  10. Retest. Repeat whatever failed, update the plan, and schedule the next test.

What to measure: RTO, RPO, and other test results

RTO and RPO, the recovery objectives, are only targets until a test measures them. The difference between RTO and RPO has its own article, so this section covers measurement. Six metrics are enough.

How actual RPO, technical RTO and business RTO are measured in a DR test
Metric How it is measured in the test Pass criterion Typical failure
Actual RTO Time from the failover command until the application answers a test request (technical RTO); the business RTO also counts detection and decision time Technical RTO plus detection and decision time is at or below the business RTO Slow disk restore, wrong boot order, manual steps
Actual RPO Age of the newest data in the recovered system, compared with the moment the test failover was started At or below the target RPO Replication lag, skipped schedule, older recovery point than assumed
Data integrity check Checksums, row counts, or queries compared with the source No corrupted or missing records Crash-consistent copy of a busy database; application-consistent snapshots avoid it
Application-level health check Login, a key transaction, and an API call, end to end All critical transactions pass The machine boots, but the app cannot reach DNS, the database, or the license server
Failback time Time to return workloads and changed data to production Within the planned maintenance window Failback never tested, large volume of changes
Manual interventions Number of steps done outside the runbook or automation Every step documented Fixes known only to one engineer

Metrics for a DR test, with pass criteria and typical failures

Failover vs failback: test both directions

Failover moves workloads to the recovery site, and failback returns them once the primary site is repaired, with the data changed in the meantime. Failover testing is routine, but failback is easy to skip, so the first real failback goes untested. The yearly full interruption test is the natural place to include it and time it. Where production cannot be shut down, the closest substitute is a test failover and failback of the full stack in an isolated environment.

Why DR tests fail

Recovery often misses its targets. In Veeam’s 2024 Data Protection Trends survey of 1,200 IT leaders, respondents reported that only 58% of servers met their recovery SLA in the last large-scale attack. Seven causes are common.

  • The recovery site differs from production. Changes reach production but not the recovery site.
  • Dependencies stay unverified. DNS, directory services, license servers, and third-party APIs are assumed to work.
  • There is no isolated network. Teams fear collisions with live systems and keep postponing the test, although a sandbox reduces the risk.
  • Credentials and licenses expire. Service accounts, certificates, and license keys run out between tests.
  • The runbook belonged to someone who left. Nobody else knows the unwritten steps.
  • Success criteria were never set. A server that boots counts as a pass.
  • Only infrastructure is tested. The virtual machine starts, but nobody checks that the application works.

Reversing these causes gives the core of disaster recovery testing best practices.

Automating and orchestrating DR testing

Manual testing works for a few machines. With more machines, the runbook grows long, and people run it slowly and differently each time. Disaster recovery automation moves individual steps into software: replicating data, starting machines, and checking the result.

Disaster recovery orchestration coordinates those steps in a recovery sequence: which machines start first and which wait, which network each one joins, and which health check completes a step.

The same sequence later drives the real failover. In Veeam’s 2024 Data Protection Trends survey, only 13% of organizations used orchestrated workflows in their disaster recovery processes. 

Automation also makes the non-disruptive test failover practical. Replication keeps a copy of each protected machine at the recovery site. At test time, the tool builds machines from a chosen recovery point or snapshot inside an isolated test network. Production keeps running, and the test machines are deleted when the checks finish.

How a non-disruptive test failover works in an isolated test network

In tools that support scheduled tests, a test failover can run unattended every quarter, and each run produces a report with times, the measured RTO, and failed steps for auditors to review.

Failover testing on private clouds and virtualization platforms

Hystax Disaster Recovery, part of Hystax Platform, supports two replication modes: 

  • Agentless replication works at the platform level through an external Hystax agent, with nothing installed in the guest OS. It is available for VMware, OpenStack, and oVirt. 
  • Agent-based replication uses a Windows or Linux agent inside the guest OS and covers other hypervisors, VMs in public clouds, and physical servers. 

 

DR plans are generated from the replicated infrastructure, and auto-testing can be set up within the plan.

On Red Hat OpenShift Virtualization, for example, teams can validate plans in an isolated environment at any time without affecting production workloads. For the ransomware scenario from step 1 of the test plan, the built-in ransomware protection flags an abnormal volume of changed data during replication. 

The platform then places the machine in a Suspended state: it stops creating new restore points and does not apply the retention policy, so earlier clean restore points stay available.

Recovery targets include OpenStack, VMware, KVM, Red Hat OpenShift Virtualization, Virtuozzo, oVirt, OpenNebula, and KubeVirt, plus AWS, Azure, Google Cloud (GCP), and Alibaba Cloud. Failback to the primary site is built in. Recovery can also run across clouds.

Adam Data Center, an independent Spanish IaaS provider with data centers in Madrid and Barcelona, added disaster recovery to its OpenStack-based cloud with Hystax. According to the March 2026 case study, destination systems boot individually or in groups while respecting dependencies, and contingency environments can be deployed and validated without stopping production.

Test recovery on your platform

Talk to Hystax engineers about testing recovery plans on the stack you use

Thank you for your request!

We will be in touch soon.

We respect your privacy. See our Privacy Policy. You can unsubscribe at any time.

Frequently asked questions

What is disaster recovery testing?

Disaster recovery testing means exercising a DR plan to check recovery against the declared RTO and RPO. The test can be a team discussion or a real failover. It finds outdated dependencies, expired credentials, and missing steps before a real outage does.

What are the types of DR tests?

There are six common types: plan review, tabletop exercise, simulation test, parallel test, full interruption test, and non-disruptive test failover. The range runs from discussion to a full production shutdown. Most teams mix several, exercising safer tests often and heavy tests yearly.

How often should a DR plan be tested?

The baseline is one test a year per critical system, plus one after major changes such as a platform migration. Critical workloads can be tested quarterly with a test failover, because with an isolated test network it does not affect production. DORA requires annual testing for EU financial entities.

What is a non-disruptive test failover?

A non-disruptive test failover recovers workloads from replicated data into an isolated test network while production keeps running. Production users are not affected. The test machines are removed afterward, so teams can measure real recovery time and verify applications as often as needed.

Who is responsible for DR testing?

The IT or infrastructure owner leads the test, system owners run their recovery steps, and a business representative signs off the result. Management owns the recovery targets. With DRaaS, the provider handles much of the testing, but the customer still validates applications and approves the results.

What is the difference between failover and failback?

Failover moves workloads from the primary site to the recovery site, in a real outage or during a test. Failback returns them once the primary site is repaired, along with the data changed in the meantime. Both directions need testing, because only a tested failback shows whether changed data returns intact.

What should a DR test report contain?

A DR test report records scope, date, and test type, target and measured RTO and RPO, data and application check results, failback time, manual interventions, failed steps, and open risks. Each finding gets an owner and a due date. It also serves as audit evidence.

A recovery plan is only as good as its last successful test. Walkthroughs and real failovers expose gaps and missing dependencies, so the plan is reviewed after every test and major change.

For test types, frequency, and a step-by-step plan, see our guide on how to test a disaster recovery plan.

Enter your email to be notified about new and relevant content.

Thank you for joining us!

We hope you'll find it usefull.

You can unsubscribe from these communications at any time. Privacy Policy

News & Reports

OptScale AI – AI Governance Platform

Hystax has announced the release of OptScale AI, extending its OptScale FinOps platform with capabilities designed to help organizations manage and optimize AI usage across teams, models, and AI agents.

FinOps, cloud cost optimization and security

Discover our best practices: 

  • How to release Elastic IPs on Amazon EC2
  • Detect incorrectly stopped MS Azure VMs
  • Reduce your AWS bill by eliminating orphaned and unused disk snapshots
  • And much more deep insights

Optimize RI/SP usage for ML/AI teams with OptScale

Find out how to:

  • see RI/SP coverage
  • get recommendations for optimal RI/SP usage
  • enhance RI/SP utilization by ML/AI teams with OptScale