In this section, we will discuss the behavior of the vSAN Stretched Cluster when various failures occur.
Understanding failure scenarios requires a thorough grasp of component location. In a Stretched Cluster Scenario, the components of a vSAN Object are placed as shown in the image.

The virtual machine's virtual disk (VMDK) has one component placed in the Preferred Site, one component placed in the Secondary Site, and a Witness component in the Tertiary Site that houses the vSAN Witness Host. The illustration shows a storage policy that will protect across sites, but not within a site. This is the default protection policy used with vSAN Stretched Clusters for versions 6.1 through 6.5.
Below is a vSAN 6.6 cluster with data protection (mirroring) across sites, as well as local data protection (mirroring in this case) within each of the data sites. vSAN Stretched Clusters can support up to a single complete site failure and a policy-based maximum of host failures within a site.

If a site has more failures than the local protection policy will allow, then the site is considered failed. It is important to remember that the vSAN Witness Host, residing in the Tertiary site, is only a single host. Because of this, a failure of the vSAN Witness Host is also considered a site failure.
1. Individual Drive Failure
Let's first discuss what happens when a single drive fails. In this scenario, vSAN will mark the component as absent. However the VM will continue to run. If the policy includes Local Protection, reads will be serviced by the remaining components within the same site.
The component will be rebuilt within the same site after 60 minutes if there is an additional host available and the failed host does not come back online.
If there are no additional hosts available within the site, the component will only be rebuilt if the failed/isolated host comes back online.
In cases where multiple hosts that contain the components fail or are isolated, reads will be services across the inter-site link. This can be a significant amount of traffic on the inter-site link depending on the amount of data on the failed host.

If the policy does not include Local Protection, reads will be serviced across the inter-site link.
This can be a significant amount of traffic on the inter-site link depending on the amount of data on the failed host.
The component will be rebuilt within the same site as the failed/isolated hosts after 60 minutes if there is an alternate.
If there are no additional hosts available, or hosts are at capacity, the component will only be rebuilt if the failed/isolated host comes back online.

2. Individual Host Failure or Network Isolation
What happens when a host fails or is network isolated? vSAN will mark the component absent. The VM will continue to run or it will be rebooted by vSphere HA if the VM was running on the host that went offline.
If the policy includes Local Protection, reads will be serviced by the remaining components within the same site.
The component will be rebuilt within the same site after 60 minutes if there is an additional host available and the failed host does not come back online.
If there are no additional hosts available within the site, the component will only be rebuilt if the failed/isolated host comes back online.
In cases where multiple hosts that contain the components fail or are isolated, reads will be services across the inter-site link. This can be a significant amount of traffic on the inter-site link depending on the amount of data on the failed host.

If the policy does not include Local Protection, reads will be serviced across the inter-site link. This can be a significant amount of traffic on the inter-site link depending on the amount of data on the failed host.
The component will be rebuilt within the same site as the failed/isolated hosts after 60 minutes if there is an alternate.
If there are no additional hosts available, or hosts are at capacity, the component will only be rebuilt if the failed/isolated host comes back online.

3. Site Failure or Network Partitions
a. Preferred Site Failure or Completely Partitioned
In the event the Preferred Site fails or is partitioned, vSAN powers the virtual machines running in that site off. The reason for this, is because the virtual machine's components are not accessible due to the loss of quorum. The vSAN Stretched Cluster has now experienced a single site failure. The loss of either site in addition to the witness is two failures, will take the entire cluster offline.

An HA primary node will be elected in the Secondary Site, which will validate which virtual machines are to be powered on. Because quorum has been formed between the vSAN Witness Host and the Secondary Site, virtual machines in the Secondary Site will have access to their data, and can be powered on.

b. Secondary Site Failure or Partitioned
In the event the Secondary Site fails or is partitioned, vSAN powers the virtual machines running in that site off. The reason for this, is because the virtual machine's components are not accessible due to the loss of quorum. The vSAN Stretched Cluster has now experienced a single site failure. The loss of either site in addition to the witness is two failures, will take the entire cluster offline.

The HA primary node on the Preferred Site, will validate which virtual machines are to be powered on. Virtual machines which have been moved to the Preferred Site will now have access to their data, and can be powered on.

c. vSAN Witness Host Failure or Partitioned
Virtual machines running in both of the main sites of a Stretched Cluster are not impacted by the vSAN Witness Host being partitioned. Virtual machines continue to run at both locations. The vSAN Stretched Cluster has now experienced a single site failure. The loss of either site in addition to the witness is two failures, will take the entire cluster offline.

In the event the vSAN Witness Host has failed, the behavior is the same as if the vSAN Witness Hosts were partitioned from the cluster. Virtual machines continue to run at both locations. Because the vSAN Stretched Cluster has now experienced a single site failure, it is important to either get the vSAN Witness Host back online, or deploy a new one for the cluster.

When the existing vSAN Witness Host comes back online or a new vSAN Witness Host is deployed, metadata changes are resynchronized between the main Stretched Cluster sites and the vSAN Witness Host. The amount of data that needs to be transmitted depends on a few items such as the number of objects and the number of changes that occurred while the vSAN Witness Host was offline. However, this amount of data is relatively small considering it is metadata, not large objects such as virtual disks.

Only single failure possibilities have been considered in the scenarios thus far. To get more information on what happens if there are failures at multiple levels, please refer to this article from VMware.
With this I'll conclude this post here.
Thank you for reading!
*** Explore | Share | Grow ***
Comments