Machines replication#

Machines replication consists of several steps:

Installing agents and configuring replication
Getting consistent machines snapshots
Detecting changes on machines between two snapshots
Sending deltas to the target platform with regard to WAN optimization and deduplication
Storing of incremental snapshots with regard to the snapshot storage policies

Retrieving machines snapshots and detecting changes#

Depending on the initial virtualization platform, it is possible to obtain consistent snapshots of machines in one of the following ways:

in case of replicating machines on VMware, getting snapshots of Windows and Linux machines is done with standard tools of VMware (VMware CBT API), leveraging a chain of internal API calls and snapshots as well as the means of the machine's OS itself.
in the case of replicating machines from oVirt, getting snapshots of the machines is done using standard oVirt tools. oVirt takes a snapshot at the start of replication, at the end of replication it deletes the snapshot from the previous replication, leaving a new one, an incremental replica is taken based on the definition of new blocks between two snapshots. In this regard, it is not recommended to delete the latest snapshot of the source machines. The oVirt API is used to create snapshots, and the oVirt-imageio API is used to read data.
in the case of replicating machines from OpenStack, getting instance images with volume snapshots on each disk is done using OpenStack API. Old images (agent-created only), except the latest one from successful replication, will be removed. Reading and getting changed areas will be performed by using the Ceph RBD client for each volume snapshot. In this regard, it is not recommended to delete related images and the latest agent-created volume snapshot of the source machines to continue the replication process as incremental restore points.
for Windows machines, consistent snapshots are acquired by means of Volume Shadow Storage (VSS) calls and tracking changes between two Application-consistent snapshots
for Linux machines, consistent snapshots are acquired by implementation of the VSS snapshot analog for block devices and sending changes to the snapshot store.

Warning

In case of a VMware infrastructure replication, VMware Tools must be installed on your virtual machines.

Sending deltas to storage. Data deduplication#

Deltas are the accumulated changes since the last replication.

Sending deltas to the target platform is done using customer agent secure HTTPS protocol. This constitutes the deduplication process - the customer application only sends data that is not yet stored for the given customer allowing to significantly save the amount of transferred items, reduce load on the network and, at times, accelerate replication of machines.

In addition to WAN deduplication, optimization also includes compression of the network traffic.

Creating a group of machines#

Once a customer dashboard has been created as part of product's initial configuration for replicating machines, navigate to the main page and select machine groups. Grouping machines allows to merge them by functionality or common parameters of replication schedule.

See Groups actions for detailed information about groups operations.

Setting replication schedule#

Once machines have appeared in the list and replication has been enabled for the specified machines, it is necessary to set replication schedules. This is described in the section Edit Replication schedule.

Warning

Pay you attention that settings can be changed globally for a customer, for each individual group or for individual machines.

Once these steps are completed, the machines will be replicated based on schedule and rules that have been set during the configuration.

Machines replication and distribution between groups#

Once groups have been created, you can proceed to replicate machines. To do this, follow the machine replication process (Section: ACP - Replicating machines) directly from the menu or through the group settings using the Register new platform menu item. The second option will automatically select a group for the machine during the relevant migration process step.

Select the type of a replicated platform.

pd_protect_flow

Further steps for working with the Replication Agent Download Wizard are available at the following links:

Replication agent installation and configuration#

VMware Agent#

VMware Agent is an external replication agent type that is deployed as a separate instance in the source environment.

Requirements#

To ensure the replication process is successful, it is necessary to set the appropriate permissions. The permissions can be set at a high level of the vCenter hierarchy or at least at the host/cluster level where the replication agent is deployed.

Note

If you set permissions at the host/cluster level, you should also add a read-only permission (empty role) at the datacenter level within the vCenter hierarchy. This permission should be set to propagate to children for HotAdd to work or without propagation for NBD mode, though some functionality may be limited (e.g., retrieving networks, storage information such as free space, etc.).

Additionally, set permissions if you want to use HotAdd transport mode.

VMware Replication Agent requires the following user permissions in vSphere (Role "VMware Consolidated Backup user" in vCenter):

Datastore - Browse
Datastore - FileManagement
Virtual machine - Change Configuration -- Disc Lease
Virtual machine - Interact -- Power On
Virtual machine - Interact -- Power Off
Virtual machine - Provisioning -- Allow read-only disk access
Virtual machine - Provisioning -- Allow virtual machine files upload
Virtual machine - Provisioning -- Allow virtual machine download
Virtual machine - Snapshot management -- Create snapshot
Virtual machine - Snapshot management -- Remove Snapshot

Permission to access CBT is necessary for the correct performance of the application. To enable CBT:

Virtual machine - Change Configuration - Disc change tracking

Additionally, it is recommended to include the following global permissions:

Global - Disable methods
Global - Enable methods
Global - Licenses

A list of permissions for vCenter version 8.x:

Privilege/Permission Name in the API	Privilege/Permission Name in the vSphere Client	Description
Datastore.Browse	Browse datastore	Allows browsing files on a datastore.
Datastore.FileManagement	Low level file operations	Allows performing read, write, delete, and rename operations in the datastore browser.
VirtualMachine.Config.DiskLease	Change Configuration → Acquire disk lease	Allows disk lease operations for a virtual machine.
VirtualMachine.Interact.PowerOn	Interaction → Power On	Allows powering on a powered-off virtual machine, and resuming a suspended virtual machine.
VirtualMachine.Interact.PowerOff	Interaction → Power Off	Allows powering off a powered-on virtual machine. This operation powers down the guest operating system.
VirtualMachine.Provisioning.DiskRandomRead	Provisioning→Allow read-only disk access	Allows opening a disk on a virtual machine for random read access. Used mostly for remote disk mounting.
VirtualMachine.Provisioning.PutVmFiles	Provisioning →Allow virtual machine files upload	Allows write operations on files associated with a virtual machine, including vmx, disks, logs, and nvram.
VirtualMachine.Provisioning.GetVmFiles	Provisioning →Allow virtual machine download	Allows read operations on files associated with a virtual machine, including vmx, disks, logs, and nvram.
VirtualMachine.State.CreateSnapshot	Snapshot management → Create snapshot	Allows creation of a snapshot from the virtual machine’s current state.
VirtualMachine.State.RemoveSnapshot	Snapshot management → Remove Snapshot	Allows removal of a snapshot from the snapshot history.
VirtualMachine.Config.ChangeTracking	Change Configuration → Toggle disk change tracking	Allows activating or deactivating of change tracking for the virtual machine's disks. Switching it on is necessary to access CBT.
Global.DisableMethods	Disable methods	Allows servers for vCenter Server extensions to deactivate certain operations on objects managed by vCenter Server. We highly recommend turning it on.
Global.EnableMethods	Enable methods	Allows servers for vCenter Server extensions to activate certain operations on objects managed by vCenter Server. We highly recommend turning it on.
Global.Licenses	Licenses	Allows viewing installed licenses and adding or removing licenses. We highly recommend turning it on.

Note

In case of using vCloud, vCenter user requires one extra permission to operate: Profile-driven storage -> Profile-driven storage view

Ports for correct agent work:

Acura host for Live Migration - tcp/80, tcp/443
vSphere host - tcp/443
ESXi host(s) - tcp/udp/902
Send logs to Acura cluster - udp/12201

VMware Replication Agent uses VMware snapshots and VMware CBT API in order to create consistent replicas of machines' data.

This implies the following considerations regarding the VMware storage:

VMware snapshots consume storage to retain copy-on-write buffer, so it is recommended to have at least 10% free space available on VMware storage.
VMware puts additional load on storage while creating snapshots or running machines with existing snapshots.

Please consider that storage performance warning thresholds need to be adjusted in order to meet this increased load during replication.

Warning

The source machine must have VMware Tools installed manually prior to any replication procedures for it to display its network information in the target VMware ESXi correctly.

Permissions for HotAdd#

Add permissions for using HotAdd transport mode either at the hierarchy level where the agent is deployed or directly on the agent itself, ensuring they are at the required level or higher:

Virtual machine - Change Configuration -- Add existing disk
Virtual machine - Change Configuration -- Advanced configuration
Virtual machine - Change Configuration -- Remove disk

Installation#

The first two steps that must be completed to successfully install the VMware replication agent are described in the section Machines replication and distribution between groups.

Step 3. Agent options#

For successful installation, on the third step just select the group to which the protected machines will be added and click 'Next'. Details about creating groups of machines are described here.

Configure cloud access settings for VMware protection. When all of the above is done, you can move further and click 'Next'.

pd_protect_flow3_vmware

If the platform is not on the list, it must be registered. Just select 'New VMware vSphere' instead of 'Registered VMware vSphere' and fill in all fields. Click on the '?' for hints about fields. Click the 'Next' button, when finish. The installation wizard will check the correctness of the entered data. If correct, goes to the next step.

pd_protect_flow3_vmware_new

Field	Description	Example
Cloud name	The name of the cloud which will be shown in the UI. The name must be unique.	cloud
Endpoint	Endpoint to connect to	cloud.example.org
Login	User login	username
Password	Password to access the target cloud	password
Use enforce quiesce strategy	By default, the agent will try to quiesce the machine, create non-quiesce snapshot in case if quiesce is falling. When the checkbox is checked, non-quiesce snapshot will not be created.	unchecked
Use cluster level discover	Enable devices discover on cluster level. If unchecked, devices discover will be performed on host level.	unchecked
On replication start	Whether to change source machine power state on replication start	No action
On replication finish	Whether to change source machine power state on replication finish	No action
Storage threshold per machine disk	Run application on if datastore has free space X percent of total disk size (0 – unlimited)	0
Free space storage threshold	Run application on if datastore has X percent of total disk size (0 – unlimited)	0
VDDK Compression	Use VDDK compression. Select the option from the list: No compression, zlib, fastlz, skipz.	No compression
Host/Cluster name	Name of the Host or Cluster. Enables advances discover mode. This field is optional.	Empty
VDDK transport mode	Select the VDDK transport mode.	NBD:NBDSSL
Parallel disks uploading	Enable parallel disk uploading. By default, source disks will be uploaded sequentially. If checked, source disks will be uploaded in parallel. Enabling this setting increases the RAM size requirements for the replication agent.	unchecked
Filter VMs by name prefixes	Check to discover only VMs with names that match the provided prefixes.	unchecked
Name prefixes filter	Active only when the ‘Filter VMs by name prefixes’ option is enabled. A comma-separated list of name prefixes.	WIN_,TEST_
Filter VMs by custom attributes	Check to discover only VMs that match the provided custom attributes. Note that custom attributes are not available on ESXi: vCenter server endpoint is required.	unchecked
Custom attributes filter	Active only when the ‘Filter VMs by custom attributes’ option is enabled. A comma-separated list of custom attributes in key=value format.	TYPE=PROD,TYPE=TEST

Pay special attention to the 'On replication start', 'On replication finish' fields. This settings allow you to power off or power on the source virtual machine when replication starts/ends. The default value is 'No action': the state of the machine will not be changed.

pd_protect_flow3_vmware_onstartstop

Use the 'Filter VMs by name prefixes' and 'Filter VMs by custom attributes' checks to filter machines. These filters are useful when you need to limit the number of replicating devices. You can use them either together or separately.

Typical cases and rules for using filters:

If you need to discover VMs with multiple prefixes, you can specify them as a comma-separated string.
If no filters are used (both checkboxes are unchecked), all VMs will be discovered according to the other settings.
If at least one filter is enabled but its input field is empty, no VMs will be discovered.
Filters cannot contain special characters such as commas (","), slashes ("/"), etc., and must not end with spaces.
Make sure to set keys and values correctly to get the expected result.
In the case of custom attributes, an exact match of both keys and values is required.
If both filters (prefixes and custom attributes) are set, only VMs that meet both criteria (i.e., in the intersection of the filtered sets) will be discovered.
The replication agent VMware retrieves updated settings for the corresponding source cloud on the next heartbeat after saving (approximately every 30 seconds by default).

To enable VM filtering by attributes, set them for the source VMs in vCenter:

1. Open the VM page.

2. Go to Actions → Tags & Custom Attributes → Edit Custom Attributes.

3. If the attribute already exists, set or change its value. Otherwise, add a new attribute (type: Virtual machine) by specifying its name and value.

Additionally, it is possible to specify static network settings. Switch 'Agent network settings' to 'Static' and fill in the fields.

pd_protect_flow3_vmware_settings

Field	Description	Example
Agent IP	The IP address of the agent	172.22.6.230
Subnet mask	The subnet mask of the agent	255.255.0.0
Gateway	Agent gateway	172.22.1.2
DNS servers	DNS of the agent’s server. You can specify multiply addresses separated by commas.	1.1.1.1, 2.2.2.2

Step 4. Protection instructions#

On the fourth step there is a 'Download agent' button and protection instructions. To successfully install the agent, carefully follow the instructions on the screen:

Download agent OVA file using the 'Download agent' button above and deploy it on each ESXi host in your VMware cluster you want to protect.
Launch deployed VMs (agents) to protect machines on each host.

pd_protect_flow4_vmware

Machines will be detected and shown in a target group in a few minutes after agent deployment and launch. By default, all discovered machines will have 'Unprotected' status. Select machines you want to protect and use 'Actions -> Start Protection' to start protecting them.

Alternativaly use Download Replication Agent item on Manage Clouds page to get the agent's file.

A agent's instance must have at least 2 CPUs and 4 GB RAM and must be deployed on each of the ESXi hosts that have machines intended for replication.

Note

VMware vSphere can issue a warning about the presence of an unknown configuration parameter for the virtual machine. The service parameter "hvragent" is added deliberately and it is not a security risk.

VMware Agent configuration#

Use TUI to change network settings, IP addresses and configuration files. On the main window agent ID (in format AGENT_ID (CLOUD_ID)@CUSTOMER_ID), statuses and logs are shown. There are To console and Agent options buttons at the bottom of the window.

ao_tui_cabrio_main

Select To console if you prefer to configure via command line, otherwise press Agent options.

Note

We highly recommend to configure agent via Agent options.

In addition you can access TUI via ssh. Use user user and password px%96jA-8zX{ to log in. You can also use SSH key (if set in the cloud). The third button Exit console will appear. Use it to close ssh session.

Console#

After pressing To console the console will launch. The list of available commands will appear on the screen. Use it to configure parameters. To exit the console enter exit.

ao_tui_cabrio_console

To edit machine hostname use command sudo vi /etc/hostname or sudo nano /etc/hostname.

Note

Restart the machine to apply the changes.

Agent logs are stored in "/var/log/cabrio/cabrio.log".

Agent options#

Note

We highly recommend to use this option to configure agent.

The menu will appear on the screen.

ao_tui_cabrio_agentoptions

Warning

Changes perform here may automatically restart agent services, interrupting all running tasks. Configure the agent when it's idle.

1. Restart agent services restarts the services just right after the confirmation. The success of the restart can be tracked by the logs of the main window, which will appear immediately after the operation is confirmed.

2. Enable/disable debug logging adds/removes debug information to/from log. By default debug information is disabled.

3. Edit network. NetworkManager TUI will be launched when select this menu.

ao_tui_cabrio_NM

4. Edit hosts starts text editor GNU nano to edit data stored into /etc/hosts. Use it to specify DNS-names manually.

ao_tui_cabrio_hosts

5. Edit controller addresses is used to update IP addresses and ports. The menu will appear when clisk.

ao_tui_cabrio_addresses

Update IP address of the controller (Update all addresses at once):

ao_tui_cabrio_addresses_01

configure restAPI, receiver, report, stash, or logstash IP optionally (Update individual addresses):

ao_tui_cabrio_addresses_02

6. Edit agent common config starts text editor GNU nano to edit configuration file. If necessary update certificates, users or agents IDs. Controller addresses (can also be changed via 5. Edit controller addresses) are available for editing either.

ao_tui_cabrio_06

Warning

Do not update role and cloud_type fields.

7. Edit agent service config starts text editor GNU nano to edit service configuration file.

ao_tui_cabrio_07

Note

File content depends on the agent type.