Before You Read This Chapter
The Recovery Plan has already been created.
The Tabletop Exercise carried out previously also went well.
The team believed this meant they were ready.
However, when they actually tried using the recovery tools and procedures for the first time, unexpected problems turned up.
This raises a question:
"Why aren't what is written down and what can actually be done the same thing?"
This chapter thinks through this difference.
16.1 What Is Written Down and What Can Actually Be Done Are Not the Same
In Chapter 15, we saw that Restore having finished and Recovery having succeeded are not the same thing. This chapter takes that idea one step further.
A Recovery Plan being written down, or a Tabletop Exercise having gone well in the past, do not, by themselves, prove that Recovery capability actually exists.
A written plan, and a past discussion-based confirmation, are each useful, but they are separate from something actually confirmed by hands-on work.
This chapter considers how to test and exercise Recovery capability, and how to turn the problems found there (Gaps) into improvements.
16.2 What Do Test / Exercise Confirm?
Test and Exercise each confirm a different property.
Confirming whether a system or component actually operates, and confirming plan, role, and readiness (plan/role/readiness), must not be conflated.
For example, even if a given Test confirms a system's technical operation, that does not mean it has also confirmed whether the responsible personnel are in a position to actually carry out the procedure. The reverse is also true.
When looking at the results of a Test or Exercise, you need to make clear exactly what it confirmed.
16.3 Tabletop Exercise and Functional Exercise
There are several kinds of Test and Exercise, but here we take up the difference between two of them.
A Tabletop Exercise is a discussion-centered method of confirmation. The people involved gather and talk through an anticipated situation, confirming the flow of the response.
A Functional Exercise is a method of confirmation in which people actually perform roles and carry out operations within a simulated environment. It calls for not just discussion, but actual operation and role performance.
These two differ greatly in nature.
A Tabletop Exercise going well does not, by itself, prove that Recovery is technically possible. Confirming, in discussion, that "this seems like it will work" and confirming it by actually carrying out the operations are separate things.
| Type | What it confirms | What it does not, by itself, prove |
|---|---|---|
| Tabletop Exercise | The understanding and agreement of those involved regarding the flow of the response | Whether Recovery is actually technically possible |
| Functional Exercise | Actual operations and role performance in a simulated environment | Whether it reaches a state actually usable for business |
16.4 Recovery Tools / Procedures Are Also Objects of Verification
The Recovery tools and procedures you will actually rely on also need to be tested and exercised themselves, in advance.
Even if a Recovery Plan calls for using a given tool or procedure, if it has never actually been used, you cannot know whether it will work without problems when it matters.
For example, a Restore script, a Recovery access tool, and the operating steps written in a procedure document are all things it is considered desirable to try out at least once before actually relying on them.
How to design or implement these tools, or what automation mechanisms to use, is not covered in this chapter.
16.5 Finding Gaps Through Testing
One purpose of carrying out Test and Exercise is to find problems — Gaps — that could not have been known without actually trying.
A test that only confirms scenarios that go well — a so-called happy-path-only test — does not provide sufficient confirmation of Recovery under conditions different from normal, such as a situation where some dependency is unavailable, or an unexpected obstacle arises.
That said, this chapter does not present a fixed list of scenarios specifying exactly what situations must always be assumed for testing. What situations to assume differs depending on the target system and organization.
16.6 Feeding Gaps Back Into Improvement, and Retesting
Once a Gap is found, it needs to be connected, in some form, to an improvement.
In this book, we organize this overall flow as the following cycle:
Plan → Test / Exercise → Gap → Improve Plan / Dependency / Procedure → Retest
The results of a Test or Exercise should be reflected in improvements to the Recovery Plan or Procedure. The targets of improvement can include not just the Plan itself, but also information related to dependencies, and procedures as well.
After making an improvement, testing again (Retest) whether it actually works can also be considered part of this cycle.
This cycle is something this book has organized by putting together multiple perspectives covered so far; it is not a standard taxonomy or an official cycle defined by NIST or CISA.
16.7 Do Not Over-Generalize From a Single Success
Finally, there is one point worth noting.
A single Test or Exercise going well does not mean that Recovery capability as a whole has been proven. Different kinds of Test and Exercise each confirm a different property, and a single result cannot be extended to cover a different property that was not confirmed.
Also, what was confirmed in testing does not necessarily hold in exactly the same way in an actual incident. A test is, at most, a confirmation result under that particular time and those particular conditions.
Test and Exercise can be material that increases confidence in whether a Recovery Point can actually be used. However, that result alone does not prove that a particular Recovery Point is safe under every condition — what can be confirmed is limited to the time, scope, and conditions that were actually tested.
16.8 Summary
This chapter organized how to test and exercise Recovery capability, and how to turn that into improvement.
The important points are these:
- A Recovery Plan being written down, or a past Tabletop Exercise having succeeded, do not, by themselves, prove Recovery capability.
- Test and Exercise each confirm a different property, and must not be conflated.
- Tabletop Exercise and Functional Exercise differ greatly in nature. A successful Tabletop Exercise does not prove technical Recoverability.
- Recovery tools and procedures also need to be tested and exercised themselves before you actually rely on them.
- A Gap found through testing should be connected to improving the Plan, Dependency, and Procedure, and retested as needed. This cycle is this book's own organization, not a standard NIST/CISA cycle.
- The success of a single Test or Exercise must not be over-generalized to a different, unconfirmed property, or to success in an actual incident.
Starting with the next chapter, Chapter 17, we shift perspective to cover Archive and Retention, which have a different purpose from Backup, and then, in Chapter 18, look at their relationship to law, regulation, Privacy, and Contract.
Questions to Consider After Reading Chapter 16
Thinking about the systems you are responsible for, can you answer the following questions?
- Why does a written Recovery Plan, or a past Test having succeeded, not by itself prove Recovery capability?
- What does each of Tabletop Exercise and Functional Exercise confirm?
- Why do Recovery tools and procedures themselves need to be tested and exercised in advance?
- Why is the cycle Plan → Test / Exercise → Gap → Improve Plan / Dependency / Procedure → Retest not a standard NIST cycle?
- Why does a single Test succeeding not guarantee success in an actual incident?
You do not need to be able to answer all of them.
Rather, if there is a point where you get stuck, that is exactly the point this chapter is meant to help you check.