Machines replication#

Machines replication consists of several steps:

Installing agents and configuring replication
Getting consistent machines snapshots
Detecting changes on machines between two snapshots
Sending deltas to the target platform with regard to WAN optimization and deduplication
Storing of incremental snapshots with regard to the snapshot storage policies

Retrieving machines snapshots and detecting changes#

Depending on the initial virtualization platform, it is possible to obtain consistent snapshots of machines in one of the following ways:

in case of replicating machines on VMware, getting snapshots of Windows and Linux machines is done with standard tools of VMware (VMware CBT API), leveraging a chain of internal API calls and snapshots as well as the means of the machine's OS itself.
in the case of replicating machines from oVirt, getting snapshots of the machines is done using standard oVirt tools. oVirt takes a snapshot at the start of replication, at the end of replication it deletes the snapshot from the previous replication, leaving a new one, an incremental replica is taken based on the definition of new blocks between two snapshots. In this regard, it is not recommended to delete the latest snapshot of the source machines. The oVirt API is used to create snapshots, and the oVirt-imageio API is used to read data.
in the case of replicating machines from OpenStack, getting instance images with volume snapshots on each disk is done using OpenStack API. Old images (agent-created only), except the latest one from successful replication, will be removed. Reading and getting changed areas will be performed by using the Ceph RBD client for each volume snapshot. In this regard, it is not recommended to delete related images and the latest agent-created volume snapshot of the source machines to continue the replication process as incremental restore points.
for Windows machines, consistent snapshots are acquired by means of Volume Shadow Storage (VSS) calls and tracking changes between two Application-consistent snapshots
for Linux machines, consistent snapshots are acquired by implementation of the VSS snapshot analog for block devices and sending changes to the snapshot store.

Warning

In case of a VMware infrastructure replication, VMware Tools must be installed on your virtual machines.

Sending deltas to storage. Data deduplication#

Deltas are the accumulated changes since the last replication.

Sending deltas to the target platform is done using customer agent secure HTTPS protocol. This constitutes the deduplication process - the customer application only sends data that is not yet stored for the given customer allowing to significantly save the amount of transferred items, reduce load on the network and, at times, accelerate replication of machines.

In addition to WAN deduplication, optimization also includes compression of the network traffic.

Creating a group of machines#

Once a customer dashboard has been created as part of product's initial configuration for replicating machines, navigate to the main page and select machine groups. Grouping machines allows to merge them by functionality or common parameters of replication schedule.

See Groups actions for detailed information about groups operations.

Setting replication schedule#

Once machines have appeared in the list and replication has been enabled for the specified machines, it is necessary to set replication schedules. This is described in the section Edit Replication schedule.

Warning

Please note that settings can be changed globally for a customer, for each individual group or for individual machines.

Once these steps are completed, the machines will be replicated based on schedule and rules that have been set during the configuration.

Machines replication and distribution between groups#

Once groups have been created, you can proceed to replicate machines. To do this, follow the machine replication process (Section: ACP - Replicating machines) directly from the menu or through the group settings using the Register new platform menu item. The second option will automatically select a group for the machine during the relevant migration process step.

Select the type of a replicated platform.

protect flow

Further steps for working with the Replication Agent Download Wizard are available at the following links:

Replication agent installation and configuration#

VMware Agent#

VMware Agent is an external replication agent type that is deployed as a separate instance in the source environment.

Requirements#

To ensure the replication process is successful, it is necessary to set the appropriate permissions. The permissions can be set at a high level of the vCenter hierarchy or at least at the host/cluster level where the replication agent is deployed.

Note

If you set permissions at the host/cluster level, you should also add a read-only permission (empty role) at the datacenter level within the vCenter hierarchy. This permission should be set to propagate to children for HotAdd to work or without propagation for NBD mode, though some functionality may be limited (e.g., retrieving networks, storage information such as free space, etc.).

Additionally, set permissions if you want to use HotAdd transport mode.

VMware Replication Agent requires the following user permissions in vSphere (Role "VMware Consolidated Backup user" in vCenter):

Datastore → Browse
Datastore → FileManagement
Virtual machine → Change Configuration → Disc Lease
Virtual machine → Interact → Power On
Virtual machine → Interact → Power Off
Virtual machine → Provisioning → Allow read-only disk access
Virtual machine → Provisioning → Allow virtual machine files upload
Virtual machine → Provisioning → Allow virtual machine download
Virtual machine → Snapshot management → Create snapshot
Virtual machine → Snapshot management → Remove Snapshot

Permission to access CBT is necessary for the correct performance of the application. To enable CBT:

Virtual machine → Change Configuration → Disc change tracking

Additionally, it is recommended to include the following global permissions:

Global → Disable methods
Global → Enable methods
Global → Licenses

A list of permissions for vCenter version 8.x:

Table 1: A list of permissions for vCenter version 8.x.
Privilege/Permission Name in the API	Privilege/Permission Name in the vSphere Client	Description
Datastore.Browse	Browse datastore	Allows browsing files on a datastore.
Datastore.FileManagement	Low level file operations	Allows performing read, write, delete, and rename operations in the datastore browser.
VirtualMachine.Config.DiskLease	Change Configuration → Acquire disk lease	Allows disk lease operations for a virtual machine.
VirtualMachine.Interact.PowerOn	Interaction → Power On	Allows powering on a powered-off virtual machine, and resuming a suspended virtual machine.
VirtualMachine.Interact.PowerOff	Interaction → Power Off	Allows powering off a powered-on virtual machine. This operation powers down the guest operating system.
VirtualMachine.Provisioning.DiskRandomRead	Provisioning→Allow read-only disk access	Allows opening a disk on a virtual machine for random read access. Used mostly for remote disk mounting.
VirtualMachine.Provisioning.PutVmFiles	Provisioning →Allow virtual machine files upload	Allows write operations on files associated with a virtual machine, including vmx, disks, logs, and nvram.
VirtualMachine.Provisioning.GetVmFiles	Provisioning →Allow virtual machine download	Allows read operations on files associated with a virtual machine, including vmx, disks, logs, and nvram.
VirtualMachine.State.CreateSnapshot	Snapshot management → Create snapshot	Allows creation of a snapshot from the virtual machine’s current state.
VirtualMachine.State.RemoveSnapshot	Snapshot management → Remove Snapshot	Allows removal of a snapshot from the snapshot history.
VirtualMachine.Config.ChangeTracking	Change Configuration → Toggle disk change tracking	Allows activating or deactivating of change tracking for the virtual machine's disks. Switching it on is necessary to access CBT.
Global.DisableMethods	Disable methods	Allows servers for vCenter Server extensions to deactivate certain operations on objects managed by vCenter Server. We highly recommend turning it on.
Global.EnableMethods	Enable methods	Allows servers for vCenter Server extensions to activate certain operations on objects managed by vCenter Server. We highly recommend turning it on.
Global.Licenses	Licenses	Allows viewing installed licenses and adding or removing licenses. We highly recommend turning it on.

Note

In case of using vCloud, vCenter user requires one extra permission to operate: Profile-driven storage → Profile-driven storage view

Ports for correct agent work:

Acura host for Live Migration - tcp/80, tcp/443
vSphere host - tcp/443
ESXi host(s) - tcp/udp/902
Send logs to Acura cluster - udp/12201

VMware Replication Agent uses VMware snapshots and VMware CBT API in order to create consistent replicas of machines' data.

This implies the following considerations regarding the VMware storage:

VMware snapshots consume storage to retain copy-on-write buffer, so it is recommended to have at least 10% free space available on VMware storage.
VMware puts additional load on storage while creating snapshots or running machines with existing snapshots.

Please consider that storage performance warning thresholds need to be adjusted in order to meet this increased load during replication.

Warning

The source machine must have VMware Tools installed manually prior to any replication procedures for it to display its network information in the target VMware ESXi correctly.

HotAdd transport mode#

Permissions#

Add permissions for using HotAdd transport mode either at the hierarchy level where the agent is deployed or directly on the agent itself, ensuring they are at the required level or higher:

Virtual machine → Change Configuration → Add existing disk
Virtual machine → Change Configuration → Advanced configuration
Virtual machine → Change Configuration → Remove disk

Preparing for HotAdd mode usage#

Note

Each VMware replication agent can open up to 60 disks simultaneously via HotAdd. If you plan to replicate a large number of VMs in parallel, it is recommended to increase VMware agent virtual hardware — to 16 vCPU cores and 32 GB of RAM. Additionally, consider scaling the number of services on Acura.

Follow the instructions to get ready for and use HotAdd transport mode:

1. Set the permissions. Use the instructions.

2. Edit an existing source cloud: in ACP, select the customer → Manage Clouds. In the Actions menu of the desired cloud, select Edit. Locate the VDDK transport mode field in the Edit Cloud form and select the appropriate variant. Save changes.

3. Remove the replication agent for this cloud (if it was deployed earlier), download it again, and deploy it on the source side. Use the instructions.

4. By default, VDDK uses the following pipe timeouts:

"pipe_start_timeout": 60,   // in seconds
"pipe_stop_timeout": 30

To customize timeouts, for example, to allow the agent to attempt opening disks via HotAdd for up to 10 minutes before switching to the next transport mode (if configured), use the following settings:

"pipe_start_timeout": 600,
"pipe_stop_timeout": 600

To apply these settings:

Open the agent’s console.
Run: sudo nano /hx/cabrio/cabrio.conf.
Add a new line with the settings, separated by commas.
Validate that the configuration remains valid JSON before saving the file.
Restart agent services to apply new settings.

Installation#

The first two steps that must be completed to successfully install the VMware replication agent are described in the section Machines replication and distribution between groups.

Step 3. Agent options#

In Step 3, select the group that will contain the protected machines and configure the VMware cloud access settings. For group details, see Creating a group of machines. Then click Next.

protect flow3 vmware

If the platform is not listed, select New VMware vSphere instead of Registered VMware vSphere. Enter the values described in the table below; select ? in the wizard for field-specific help. Click Next. The wizard validates the values before opening Step 4.

protect flow3 vmware new

Table 2: New VMware vSphere fields description.
Field	Description	Example
Cloud name	The name of the cloud which will be shown in the UI. The name must be unique.	cloud
Endpoint	Endpoint to connect to	cloud.example.org
Login	User login	username
Password	Password to access the target cloud	password
Use enforce quiesce strategy	By default, the agent will try to quiesce the machine, create non-quiesce snapshot in case if quiesce is falling. When the checkbox is checked, non-quiesce snapshot will not be created.	unchecked
Use cluster level discover	Enable devices discover on cluster level. If unchecked, devices discover will be performed on host level. Details.	unchecked
On replication start	Whether to change source machine power state on replication start. Details	No action
On replication finish	Whether to change source machine power state on replication finish. Details	No action
Storage threshold per machine disk	Run application on if datastore has free space X percent of total disk size (0 – unlimited)	0
Free space storage threshold	Run application on if datastore has X percent of total disk size (0 – unlimited)	0
VDDK Compression	Use VDDK compression. Select the option from the list: No compression, zlib, fastlz, skipz.	No compression
Host/Cluster name	Name of the Host or Cluster (multiple allowed, separated by commas). Enables advanced discover mode. This field is optional. Details.	Empty
VDDK transport mode	Select the VDDK transport mode.	NBD:NBDSSL
Parallel disks uploading	Enable parallel disk uploading. By default, source disks will be uploaded sequentially. If checked, source disks will be uploaded in parallel. Enabling this setting increases the RAM size requirements for the replication agent.	unchecked
Filter VMs by name prefixes	Check to discover only VMs with names that match the provided prefixes. Details.	unchecked
Name prefixes filter	Active only when the ‘Filter VMs by name prefixes’ option is enabled. A comma-separated list of name prefixes. Details.	WIN_,TEST_
Filter VMs by custom attributes	Check to discover only VMs that match the provided custom attributes. Note that custom attributes are not available on ESXi: vCenter server endpoint is required. Details.	unchecked
Custom attributes filter	Active only when the ‘Filter VMs by custom attributes’ option is enabled. A comma-separated list of custom attributes in key=value format. Details.	TYPE=PROD,TYPE=TEST

Machine discovery levels#

In the source cloud or replication agent settings, you can configure the level at which virtual machines are discovered for migration. To do this, two fields are provided: Use cluster level discover and Host/Cluster name.

The machine discovery level is determined by the Use cluster level discover setting. By default, this setting is disabled, which means that machine discovery is performed at the host level. When enabled, discovery is performed at the cluster level.

In the Host/Cluster name field, specify the hosts or clusters within which the search is performed. Rules:

Leave the field empty if only machines from the host or cluster where the replication agent is deployed should be included in the protection list.
If necessary, specify multiple hosts or clusters; use a comma as a separator.
Do not assign the same hosts or clusters to different replication agents.
To perform discovery and replication, the user specified in the source cloud settings must have sufficient permissions. Learn more.

On replication start and On replication finish#

Use On replication start and On replication finish to power the source virtual machine off or on when replication starts or ends. The default value is No action, which leaves the machine state unchanged.

protect flow3 vmware onstartstop

Filter fields#

Use Filter VMs by name prefixes and Filter VMs by custom attributes to limit the machines included in discovery. You can enable either or both filters.

Typical cases and rules for using filters:

If you need to discover VMs with multiple prefixes, you can specify them as a comma-separated string.
If no filters are used (both checkboxes are unchecked), all VMs will be discovered according to the other settings.
If at least one filter is enabled but its input field is empty, no VMs will be discovered.
Filters cannot contain special characters such as commas (","), slashes ("/"), etc., and must not end with spaces.
Enter custom-attribute keys and values exactly as they are configured in vCenter.
In the case of custom attributes, an exact match of both keys and values is required.
If both filters (prefixes and custom attributes) are set, only VMs that meet both criteria (i.e., in the intersection of the filtered sets) will be discovered.
The replication agent VMware retrieves updated settings for the corresponding source cloud on the next heartbeat after saving (approximately every 30 seconds by default).

To enable VM filtering by attributes, set them for the source VMs in vCenter:

1. Open the VM page.

2. Go to Actions → Tags & Custom Attributes → Edit Custom Attributes.

3. If the attribute already exists, set or change its value. Otherwise, add a new attribute (type: Virtual machine) by specifying its name and value.

Static settings#

Additionally, it is possible to specify static network settings. Set Agent network settings to Static, then enter the network values described in the table below.

protect flow3 vmware settings

Table 3: Agent network settings static fields description.
Field	Description	Example
Agent IP	The IP address of the agent	172.22.6.230
Subnet mask	The subnet mask of the agent	255.255.0.0
Gateway	Agent gateway	172.22.1.2
DNS servers	DNS of the agent’s server. You can specify multiply addresses separated by commas.	1.1.1.1, 2.2.2.2

Step 4. Protection instructions#

In Step 4, download and deploy the agent as follows:

Select Download agent to download the OVA file, then deploy it on each ESXi host in the VMware cluster that you want to protect.
Launch deployed VMs (agents) to protect machines on each host.

protect flow4 vmware

Machines will be detected and shown in a target group in a few minutes after agent deployment and launch. By default, all discovered machines have Unprotected status. Select the machines to protect, then select Actions → Start Protection.

Alternatively, select Download Replication Agent on the Manage Clouds page.

A agent's instance must have at least 2 vCPUs and 4 GB RAM and must be deployed on each of the ESXi hosts that have machines intended for replication.

Note

VMware vSphere can issue a warning about the presence of an unknown configuration parameter for the virtual machine. The service parameter "hvragent" is added deliberately and it is not a security risk.

VMware Agent configuration#

Use the shared external replication agent TUI to configure network, controller, host, and service settings.

VMware agent logs are stored in /var/log/cabrio/cabrio.log.