Chapter 14 — Recovery Sequence and Playbook

14.1 The Need to Think About Recovery Sequence

In Chapter 12, we saw that Recovery is a problem of dependencies. In Chapter 13, we considered the path for beginning Recovery from a state where the normal administrative path cannot be used.

This chapter considers, once Recovery can be begun, what actual order to proceed with Recovery in.

If there is only one Recovery target, there is not much need to worry about order. But when there are multiple Recovery targets that need to be brought back in stages, the problem of "what to bring back, in what order" arises.

14.2 Business Priority ≠ Technical Recovery Order

From the business side's perspective, it is natural to want the most important service recovered first.

However, the order that can actually be executed technically does not necessarily follow business priority.

Business Criticality can influence Recovery Priority, but that does not automatically determine Technical Recovery Order as-is.

In this book, we organize this relationship as follows:

Business Priority ≠ Technical Recovery Order

This is an organization used to aid understanding in this book, not a standard rule established by NIST or CISA.

Because of technical dependencies, it may be necessary to bring back a foundational element that is not very conspicuous from a business standpoint, before a business-critical Application. For example, if the highest-priority business Application depends on a Database, then even if you want to bring back the Application first, the Database's Recovery may need to happen first.

This is only one hypothetical example. What dependencies actually exist differs depending on the target system.

14.3 Dependency Constrains Recovery Order

As we saw in Chapter 12, Recovery is affected by various dependencies. This dependency information can be kept not just as something understood in the abstract, but also maintained in a visualized form, such as a diagram or map. That is because dependencies affect the order of Recovery.

Understanding dependencies gives you material for thinking about "what needs to happen first to bring this service back." That said, this chapter does not re-list Chapter 12's inventory of dependencies. Nor does it re-explain individual elements such as identity, DNS, network, or storage here.

What matters here is the idea itself that dependencies can affect Recovery order.

14.4 A Single Fixed Order Cannot Be Decided

From what we've covered so far, you might be tempted to think the following:

"Then if we base it purely on dependencies, wouldn't the correct Recovery order be automatically determined?"

However, this book does not take that view.

Recovery Priority and the Technical Recovery Order based on dependencies are related, but they are not the same thing. Nor does technical dependency alone determine a single correct order common to every situation.

For that reason, this book does not support fixed rules such as the following:

• Identity must always be recovered first
• DNS must always be recovered first
• Storage must always be recovered first
• there exists a single correct Recovery order common to every organization

The actual order needs to be considered according to the situation at hand, taking into account both business priority and technical dependency.

14.5 A Recovery Plan Is Not a List of Restore Commands

So far, we have thought about the "order" of Recovery. Next, let's consider, including that order, how to keep a record of the content of Recovery.

A Recovery Plan is not merely a list of Restore commands.

A Recovery Plan needs to take into account, at minimum, elements such as the following:

• people
• process
• technology
• external resources
• dependencies

Even if you write down only the procedure for executing commands, if the people who actually carry it out, the necessary process, external partners, and other dependent elements are not taken into account, it can turn out to be insufficient for actual Recovery.

14.6 Information to Keep in a Playbook

As something for organizing the information needed to actually carry out Recovery, this book also covers the Playbook.

A Playbook should, at minimum, be considered to keep information such as the following:

• actionable processes
• responsible personnel
• invocation conditions / authority
• notification paths
• milestones
• completion criteria

In addition to this, the dependencies and external resources covered in the previous section can also be considered for inclusion in the Playbook.

Organizing this kind of information for each target system or service makes it easier to actually carry out Recovery.

Let's organize this information into a simple table.

This table is only one example, from this book, for organizing information worth keeping in a Playbook. It is not an official template defined by a particular standards body, and the "Target / Process" column, too, is a convenient category for organizing information. The actual format differs by organization and target.

14.7 Completion Criteria and Connecting to the Next Chapter

One piece of information worth keeping in a Playbook is completion criteria.

Having "carried out" a piece of Recovery work and that work having actually "been completed" are not the same thing. Merely attempting some operation does not necessarily mean it can be called complete. A Playbook needs to make clear, in advance, what counts as completion.

Also, restoring data, recovering a system, recovering a service, and resuming business operations each represent a different confirmation result. Having "come back" at one level does not necessarily mean it has automatically "come back" at another level as well.

That said, this chapter does not cover concretely how to verify these, or through what tests or exercises to confirm them. That is covered again in a later chapter.

14.8 Summary

This chapter organized what order to proceed with Recovery in, and how to keep a record of its content.

The important points are these:

• Business importance (Business Priority) and the technically executable Recovery order (Technical Recovery Order) are not the same thing (Business Priority ≠ Technical Recovery Order).
• Technical dependency can create the need to bring back an element that is not conspicuous from a business standpoint first.
• Neither dependency alone, nor business priority alone, automatically determines a single correct Recovery order.
• A Recovery Plan is not a list of Restore commands; it needs to take into account people, process, technology, external resources, and dependencies.
• A Playbook should keep information such as responsible personnel, invocation conditions/authority, notification paths, milestones, and completion criteria.
• Making completion criteria explicit lets you distinguish between having "carried out" something and having "completed" it.

Questions to Consider After Reading Chapter 14

Thinking about the systems you are responsible for, can you answer the following questions?

1. Why are Business Priority and Technical Recovery Order not the same thing?
2. Why can it happen that you want to bring back a business-critical Application first, but cannot?
3. Why does this book not present a single Recovery order as a standard?
4. Why is a Recovery Plan insufficient if it is only a list of Restore commands?
5. What is the point of making Completion Criteria explicit in a Playbook?

You do not need to be able to answer all of them.

Rather, if there is a point where you get stuck, that is exactly the point this chapter is meant to help you check.