Before You Read This Chapter
At one organization, multiple Backup copies were stored in multiple locations.
Copies existed not only in one data center, but also in another data center, and in the cloud as well.
On the surface, this looked sufficiently distributed.
However, all of these copies could be operated from the same administrator account, under the same administrative authority.
One day, the person in charge stopped to think: what would happen if this administrator account itself were compromised?
They realized that, even with multiple geographically separated copies, if the authority to operate them was concentrated in one place, then when that authority was put at risk, multiple copies could be put at risk at the same time.
"Copies existing in multiple locations" and "those copies not being lost from the same cause at the same time" are not necessarily the same thing.
This chapter organizes this difference.
7.1 Diversifying Copy Count and Location Alone Is Not Enough
In the chapters so far, we have looked at Backup capture methods (Chapter 3) and the difference between business priority and technical recovery order (Chapter 6).
This chapter considers the "placement" of Backups.
Holding multiple copies in multiple locations helps reduce the risk of depending on a single copy or a single location. This is a widely known and useful idea.
However, simply increasing the number of copies or distributing their locations can sometimes not be enough. As in the opening scenario, even when copies are physically and geographically separated, if the authority to operate them is concentrated in one place, multiple copies can be put at risk at the same time when that authority is compromised.
This chapter organizes this kind of "hard-to-see shared weakness."
7.2 The Idea of 3-2-1
3-2-1 is one widely known idea for thinking about how to place Backups.
It is generally explained as: three copies, two different types of media, and one off-site copy.
Underlying this idea is the notion of not relying on a single copy, a single storage method, or a single location alone. By increasing the number of copies, distributing the media or methods used to store them, and placing at least one in a separate location, it aims to reduce the chance that all copies are lost at once from a single cause.
There is a point worth noting here. While 3-2-1 is a widely known guideline, within the range of evidence that could be confirmed for this book, the numbers "3," "2," and "1" themselves are not officially established as the one correct answer for every situation.
Therefore, this book treats 3-2-1 not as "an official rule that must be followed without exception," but as "one accessible guideline for the idea of distributing copies and storage methods."
7.3 Off-site / Offline / Isolated / Immutable — Properties That Look Similar but Differ
When you try to put an idea like 3-2-1 into practice, you encounter terms such as the following:
- Off-site
- Offline
- Isolated
- Immutable
These are all terms related to the safety of a Backup, but they do not mean the same thing. Satisfying one does not necessarily mean the others are also satisfied.
Roughly speaking, each refers to the following property:
- Off-site: the copy is placed in a physical location different from the production system.
- Offline: the copy or mechanism is, under normal conditions, unreachable over the network.
- Isolated: the copy or mechanism is separated from the production environment or the normal administrative path.
- Immutable: it has the property that, under certain conditions, changes or deletion are restricted.
For example, even if a copy is placed in a different building (Off-site), if it is constantly connected to the network and can be changed or deleted at any time from the normal administrator account, it is neither Offline nor Isolated.
The detailed technical meaning of each property, and how to actually verify it, are covered again in the next chapter. For this chapter, first hold onto the point that these are separate properties.
Organizing the Four Properties
Let's organize what we've covered so far into a simple table.
| Property | What mainly changes | What it helps separate | What it alone does not prove |
|---|---|---|---|
| Off-site | The physical location of the copy | Separation from a disaster/failure at the same site | Whether administrative authority or network paths are also separated |
| Offline | The connection state under normal conditions | Separation from attacks that exploit an always-on connection | Whether the copy is current, verified, or administratively protected |
| Isolated | Separation of the path/authority from the production environment | Which domain is separated needs to be checked | Which specific dimension — network, identity, administration, keys, storage, location — is actually separated |
| Immutable | Restriction on change/deletion | Protection against tampering/deletion under certain conditions | Absolute safety, or protection from every administrative authority |
This table, too, is only a guide for organizing ideas; it does not present a numerically scored measure of safety, or a rating. Detailed verification methods for each property are covered in the next chapter.
7.4 Separation of Location and Separation of Administration Are Different Problems
This is the single most important point in this chapter.
Even when copies are physically and geographically separated, if they can be operated from the same administrator account or the same administrative authority, the Failure Domain is not necessarily separated administratively.
This is exactly what the opening scenario illustrates. The copies existed in multiple locations, but the authority to operate them was concentrated in one place. In this case, if that authority is compromised, multiple geographically separated copies can be put at risk at the same time.
There are implementation examples that address this problem by introducing an additional approval, or a separate administrative boundary, to protect critical operations. For example, mechanisms such as Microsoft Azure Backup's Resource Guard, or AWS Backup's multi-party approval, are product-specific implementation examples that can be configured to require an approval separate from normal administrative authority for destructive operations.
Mechanisms like these are one option, not the only solution. This chapter also does not cover the concrete configuration steps. What matters is the idea itself that separating location alone does not automatically get you administrative separation as well.
7.5 The Idea of Failure Domain
In this book, we use the term Failure Domain to think about the scope within which things could fail together, or be compromised at the same time, from the same cause.
Backup and recovery mechanisms themselves can each have their own Failure Domain. For example, the following kinds of elements can end up sharing a Failure Domain with the production system:
- where the copy is located
- the storage or network it depends on
- credentials or identity
- administrative authority
- the keys used for encryption
- the recovery tooling itself
These are examples for organizing ideas in this book, not an official list covering every situation. What elements can actually end up sharing a Failure Domain depends on the system's configuration.
7.6 Considering Isolation From Multiple Perspectives
As we have seen, the word "isolated" cannot be captured from a single dimension alone.
As one example of how this book organizes the idea of isolation, let's look at it broken down into several dimensions.
- Geographic: is the physical location separated?
- Network: is it cut off from the normal network path?
- Storage: is the underlying storage itself not shared with production?
- Identity: is the management of authentication principals or credentials separated from the production environment?
- Administrative: are the principals or authority able to perform administrative operations separated from the production environment's administrators?
- Authorization: does the approval mechanism or authorization path for critical operations avoid depending solely on normal operating authority?
- Cryptographic: is the management of the keys used for encryption separated?
To repeat, this is only one example of how this book organizes the idea, not a classification defined by an industry standard. Nor is it necessarily the case that this covers every dimension.
What matters is the idea that whether something is "isolated" should not be judged from a single dimension alone, but needs to be confirmed from multiple perspectives.
7.7 Summary
This chapter organized the placement and isolation of Backups.
The important points are these:
- Holding multiple copies in multiple locations is useful, but it can sometimes not be enough on its own.
- 3-2-1 is a widely known guideline, but this book does not treat it as an official rule that applies to every situation.
- Off-site, Offline, Isolated, and Immutable are each different properties and do not mean the same thing.
- Physical/geographic separation alone does not automatically get you administrative separation as well.
- Backup and recovery mechanisms can also share a Failure Domain with the production system.
- Isolation needs to be considered from multiple perspectives, including geography, network, storage, Identity, administrative authority, authorization, and cryptographic keys.
The safety of a Recovery copy needs to be considered by combining these multiple properties, not just the number of copies or their locations.
In the next chapter, we look in more detail at how to verify each of the properties Off-site, Offline, Isolated, and Immutable.
Questions to Consider After Reading Chapter 7
Thinking about the systems you are responsible for, can you answer the following questions?
- What is the purpose of the idea of 3-2-1?
- Do Off-site and Offline mean the same thing?
- Why can placing Backups in multiple locations alone not guarantee administrative separation?
- What does Failure Domain mean, as this chapter uses the term?
- What weakness can exist in a configuration where all Backup copies can be operated from the same administrator account?
You do not need to be able to answer all of them.
Rather, if there is a point where you get stuck, that is exactly the point this chapter is meant to help you check.