Scope: BBS firmware 5.1 and above.
Running my Barracuda backup results in warnings that the CBT is disabled and sometimes it is but, most times it is not when I check after the backup.
CBT is disabled during the backup only. Vmware API maybe resetting CBT.
“Use Changed Block Tracking (CBT) to perform incremental backups on virtual machines running on VMware ESX/ESXi. CBT identifies and tracks block changes since the last backup, and stores these changes in log form, greatly reducing the number of backup windows and improving backup window times. Additionally, subsequent replication efficiency and speed are improved.” (Barracuda Networks, 2017)
“Changed Block Tracking (CBT) is a VMware feature that helps perform incremental backups. VMware Data Recovery uses this technology and so can developers of backup and recovery software.
Virtual machines running on ESX/ESXi hosts can track disk sectors that have changed. This feature is called Changed Block Tracking (CBT). On many file systems, CBT identifies the disk sectors altered between two changes set IDs. On VMFS partitions, CBT can also identify all the disk sectors that are in use.
Virtual disk block changes are tracked from outside virtual machines, in the virtualization layer. When software performs a backup, it can request transmission of only the blocks that changed since the last backup, or the blocks in use. The CBT feature can be accessed by third-party applications as part of the vSphere APIs for Data Protection (VADP). Applications call VADP to request that the VMkernel return blocks of data that have changed on a virtual disk since the last backup snapshot.
For CBT to identify altered disk sectors since the last change ID, these items are required:
The host must be ESX/ESXi 4.0 or later.
The virtual machine owning the disks to be tracked must be hardware version 7 or later.
I/O operations must go through the ESX/ESXi storage stack. So NFS is supported, as is RDM in virtual compatibility mode, but not RDM in physical compatibility mode. VMFS is supported, whether backed by SAN, iSCSI, or local disk.
CBT must be enabled for the virtual machine (see below).
Virtual machine storage must not be (persistent or non-persistent) independent disk, meaning unaffected by snapshots.
For CBT to identify disk sectors in use with the special "*" change ID, these items are required:
The virtual disk must be located on a VMFS volume, backed by SAN, iSCSI, or local disk.
The virtual machine must have zero (0) snapshots when CBT is enabled, for a clean start.
In some cases, such as a power failure or hard shutdown while virtual machines are powered on, CBT might reset and lose track of incremental changes. In vSphere 4.1 and prior, Cold Migration (but not Storage vMotion) could reset but not disable CBT. In vSphere 5.x versions earlier to vSphere 5.5 Update 2, storage vMotion resets CBT. For more information, see Changed Block Tracking is reset after a storage vMotion operation in vSphere 5.x (2048201).” (Vmware B, 2017)
“When running virtual machine backups using tools such as VMware Data Recovery (VDR), vSphere Data Protection (VDP), or any third-party backup tool which utilizes Changed Block Tracking (CBT) to perform incremental virtual machine backups while running on ESXi 5.x hosts, you may experience these symptoms:
Virtual machine backups are larger than usual
Incremental backups take as long and as much space as a full backup
Snapshot removal tasks fail because the backup job is still running or the backup window exceeds
CBT files grow large, although there are no major changes in the virtual machine.
This issue occurs because CBT is reset during virtual disk migration using Storage vMotion. This results in the backup tool being unaware which blocks have changed since the last backup. An incremental virtual machine backup is not possible and a full backup is required.
This is a known issue affecting ESXi 5.0.
This issue is resolved in:
ESXi 5.5 Update 2, available at VMware Downloads. For more information, see the VMware ESXi 5.5 Update 2 Release Notes. (see link below)
ESXi 5.1 Update 3, available at VMware Downloads. For more information, see the VMware ESXi 5.1 Update 3 Release Notes. (see link below)
To work around this issue, do not use Storage vMotion or Storage DRS on virtual machines to backup.
Additional cause: Backing up a virtual machine with Changed Block Tracking (CBT) enabled fails after upgrading to VMware ESXi 6.0.x on host
This issue occurs due to heap exhaustion. It can occur when attempting to enable Changed Block Tracking (CBT). If a virtual machine with a large number of virtual disks reached an upper threshold, enabling CBT fails because of heap exhaustion. This issue also occurs with multiple virtual machines with CBT enabled. In case of Windows virtual machines with VSS enabled, taking a quiesced snapshot creates double the amount of memory overhead. Finally, if the heap is close to exhaustion, performing a vMotion can also cause this issue as that process involves taking snapshots as well.
Note: The virtual disks can be spread across virtual machines or can be in a single virtual machine.” (Vmware C, 2017)
“This issue is resolved in VMware ESXi 6.0 Build 2715440, available at “VMware Downloads”. For more information, see VMware ESXi 6.0, Patch Release ESXi600-201505001 (2116125).” (Vmware A, 2017)
ADDITIONAL information found here:
Barracuda Networks. (2017). Understanding Changed Block Tracking on Virtual Machines. Retrieved from Barracuda Campus.com: https://campus.barracuda.com/product/backup/article/BBS/UnderstandCBT/
Vmware A. (2017). Backing up the virtual machine fails when CBT is enabled (2114076). Retrieved from Vmware.com: https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&externalId=2114076&sliceId=1&docTypeID=DT_KB_1_1&dialogID=505260733&stateId=0%200%20505274901
Vmware B. (2017). Changed Block Tracking (CBT) on virtual machines (1020128). Retrieved from Changed Block Tracking (CBT) on virtual machines (1020128)mware.com: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1020128
Vmware C. (2017). Changed Block Tracking is reset after a storage vMotion operation in vSphere 5.x (2048201). Retrieved from Changed Block Tracking is reset after a storage vMotion operation in vSphere 5.x (2048201)mware.com: https://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2048201
Vmware D. (2017). Vmware ESXI 5.5 update 2 release notes. Retrieved from Vmware.com: https://www.vmware.com/support/vsphere5/doc/vsphere-esxi-55u2-release-notes.html
Link to This Page: