Machines replication#

Machines replication consists of several steps:

Installing agents and configuring replication
Getting consistent machines snapshots
Detecting changes on machines between two snapshots
Sending deltas to a DC with regard to WAN optimization and deduplication
Storing of incremental backups in a DC with regard to the snapshot storage policies

Retrieving machines snapshots and detecting changes#

Depending on the initial virtualization platform, it is possible to obtain consistent snapshots of machines in one of the following ways:

in case of replicating machines on VMware, getting snapshots of Windows and Linux machines is done with standard tools of VMware (VMware CBT API), leveraging a chain of internal API calls and snapshots as well as the means of the machine's OS itself.
in the case of replicating machines from oVirt, getting snapshots of the machines is done using standard oVirt tools. oVirt takes a snapshot at the start of replication, at the end of replication it deletes the snapshot from the previous replication, leaving a new one, an incremental replica is taken based on the definition of new blocks between two snapshots. In this regard, it is not recommended to delete the latest snapshot of the source machines. The oVirt API is used to create snapshots, and the oVirt-imageio API is used to read data.
in the case of replicating machines from OpenStack, getting instance images with volume snapshots on each disk is done using OpenStack API. Old images (agent-created only), except the latest one from successful replication, will be removed. Reading and getting changed areas will be performed by using the Ceph RBD client for each volume snapshot. In this regard, it is not recommended to delete related images and the latest agent-created volume snapshot of the source machines to continue the replication process as incremental restore points.
for Windows machines, consistent snapshots are acquired by means of Volume Shadow Storage (VSS) calls and tracking changes between two Application-consistent snapshots
for Linux machines, consistent snapshots are acquired by implementation of the VSS snapshot analog for block devices and sending changes to the snapshot store.

Warning

In case of a VMware infrastructure replication, VMware Tools must be installed on your virtual machines.

Sending deltas to storage. Data deduplication#

Deltas are the accumulated changes since the last replication.

Sending changes to the DC is done using the customer agent secure HTTPS protocol. This constitutes the deduplication process - the customer application only sends data that is not yet stored for the given customer allowing to significantly save the amount of transferred items, reduce load on the network and, at times, accelerate replication of machines.

In addition to WAN deduplication, optimization also includes compression of the network traffic.

Creating a group of machines#

Once a customer dashboard has been created as part of product's initial configuration for replicating machines, navigate to the main page and select machine groups. Grouping machines allows to merge them by functionality or common parameters of replication schedule.

See Groups actions for detailed information about groups operations.

Setting replication schedule#

Once machines have appeared in the list and replication has been enabled for the specified machines, it is necessary to set replication schedules and snapshot storage policies. This is described in the sections Edit replication schedule and Edit retention settings.

Machines replication and distribution between groups#

Once groups have been created, you can proceed to replicate machines. To do this, follow the machine replication process (Section: ACP - Replicating machines) directly from the menu or through the group settings using the Protect new platform menu item. The second option will automatically select a group for the machine during the relevant migration process step.

Select the type of a replicated platform.

pd_protect_flow

Further steps for working with the Replication Agent Download Wizard are available at the following links:

Replication agent installation and configuration#

VMware Agent#

Client for VMware (VMware agent) - a Linux machine, supplied as an OVA file, that is deployed to each ESXi host or cluster and that replicates machines on these hosts. Standard VMware tools are used to detect changes and retrieve deltas on virtual machines via the Changed Block Tracking (CBT) API. These tools, in turn, trigger the operating system's mechanisms (such as quiesce) to ensure data consistency within the replicas.

An agent instance must have at least 2 CPUs and 4 GB of RAM and must be deployed on each ESXi host or cluster containing machines designated for replication.

Note

An ESXi host or cluster detection depends on cloud configuration settings according to the discovery level.

Requirements#

VMware Replication Agent requires the following user permissions in vSphere (Role "VMware Consolidated Backup user" in vCenter):

Datastore - Browse
Datastore - FileManagement
Virtual machine - Configuration -- Disc Lease
Virtual machine - Interact -- Power On
Virtual machine - Interact -- Power Off
Virtual machine - Provisioning -- Allow read-only disk access
Virtual machine - Provisioning -- Allow virtual machine files upload
Virtual machine - Provisioning -- Allow virtual machine download
Virtual machine - Snapshot management -- Create snapshot
Virtual machine - Snapshot management -- Remove Snapshot

Permission to access CBT is necessary for the correct performance of the application. To enable CBT:

Virtual machine - Configuration - Disc change tracking

Additionally, it is recommended to include the following global permissions:

Global - Disable methods
Global - Enable methods
Global - Licenses

A list of permissions for vCenter version 8.x:

Privilege/Permission Name in the API	Privilege/Permission Name in the vSphere Client	Description
Datastore.Browse	Browse datastore	Allows browsing files on a datastore.
Datastore.FileManagement	Low level file operations	Allows performing read, write, delete, and rename operations in the datastore browser.
VirtualMachine.Config.DiskLease	Change Configuration → Acquire disk lease	Allows disk lease operations for a virtual machine.
VirtualMachine.Interact.PowerOn	Interaction → Power On	Allows powering on a powered-off virtual machine, and resuming a suspended virtual machine.
VirtualMachine.Interact.PowerOff	Interaction → Power Off	Allows powering off a powered-on virtual machine. This operation powers down the guest operating system.
VirtualMachine.Provisioning.DiskRandomRead	Provisioning→Allow read-only disk access	Allows opening a disk on a virtual machine for random read access. Used mostly for remote disk mounting.
VirtualMachine.Provisioning.PutVmFiles	Provisioning →Allow virtual machine files upload	Allows write operations on files associated with a virtual machine, including vmx, disks, logs, and nvram.
VirtualMachine.Provisioning.GetVmFiles	Provisioning →Allow virtual machine download	Allows read operations on files associated with a virtual machine, including vmx, disks, logs, and nvram.
VirtualMachine.State.CreateSnapshot	Snapshot management → Create snapshot	Allows creation of a snapshot from the virtual machine’s current state.
VirtualMachine.State.RemoveSnapshot	Snapshot management → Remove Snapshot	Allows removal of a snapshot from the snapshot history.
VirtualMachine.Config.ChangeTracking	Change Configuration → Toggle disk change tracking	Allows activating or deactivating of change tracking for the virtual machine's disks. Switching it on is necessary to access CBT.
Global.DisableMethods	Disable methods	Allows servers for vCenter Server extensions to deactivate certain operations on objects managed by vCenter Server. We highly recommend turning it on.
Global.EnableMethods	Enable methods	Allows servers for vCenter Server extensions to activate certain operations on objects managed by vCenter Server. We highly recommend turning it on.
Global.Licenses	Licenses	Allows viewing installed licenses and adding or removing licenses. We highly recommend turning it on.

Note

In case of using vCloud, vCenter user requires one extra permission to operate: Profile-driven storage -> Profile-driven storage view

Ports for correct agent work:
- DR host - tcp/80, tcp/443
- vSphere host - tcp/443
- ESXi host(s) - tcp/udp/902
- Send logs to the Acura cluster - udp/12201

VMware Replication Agent uses VMware snapshots and VMware CBT API in order to create consistent replicas of machines' data.

This implies the following considerations regarding the VMware storage:

VMware snapshots consume storage to retain copy-on-write buffer, so it is recommended to have at least 10% free space available on VMware storage.
VMware puts additional load on storage while creating snapshots or running machines with existing snapshots.

Please consider that storage performance warning thresholds need to be adjusted in order to meet this increased load during replication.

Warning

The source machine must have VMware Tools installed manually prior to any replication procedures for it to display its network information in the target VMware ESXi correctly.

Installation#

The first two steps that must be completed to successfully install the VMware replication agent are described in the section Machines replication and distribution between groups.

Step 3. Agent options#

For successful installation, on the third step just select the group to which the protected machines will be added and click 'Next'. Details about creating groups of machines are described here.

Configure cloud access settings for VMware protection. When all of the above is done, you can move further and click 'Next'.

pd_protect_flow3_vmware

If the platform is not on the list, it must be registered. Just select 'New VMware vSphere' instead of 'Registered VMware vSphere' and fill in all fields. Click on the '?' for hints about fields. Click the 'Next' button, when finish. The installation wizard will check the correctness of the entered data. If correct, goes to the next step.

pd_protect_flow3_vmware_new

Field	Description	Example
Cloud name	The name of the cloud which will be shown in the UI. The name must be unique.	vcenter
Endpoint	Endpoint to connect to	vcenter.dts.loc
Login	User login	username
Password	Password to access the target cloud	password
Use enforce quiesce strategy	By default, the agent will try to quiesce the machine, create non-quiesce snapshot in case if quiesce is falling. When the checkbox is checked, non-quiesce snapshot will not be created.	unchecked
Use cluster level discover	Enable devices discover on cluster level. If unchecked, devices discover will be performed on host level.	unchecked
On replication start	Whether to change source machine power state on replication start	No action
On replication finish	Whether to change source machine power state on replication finish	No action
Storage threshold per machine disk	Run application on if datastore has free space X percent of total disk size (0 – unlimited)	0
Free space storage threshold	Run application on if datastore has X percent of total disk size (0 – unlimited)	0
VDDK Compression	Use VDDK compression. Select the option from the list: No compression, zlib, fastlz, skipz.	No compression
Host/Cluster name	Name of the Host or Cluster. Enables advances discover mode. This field is optional.	Empty

Pay special attention to the 'On replication start', 'On replication finish' fields. This settings allow you to power off or power on the source virtual machine when replication starts/ends. The default value is 'No action': the state of the machine will not be changed.

pd_protect_flow3_vmware_onstartstop

Additionally, it is possible to specify static network settings. Switch 'Agent network settings' to 'Static' and fill in the fields.

pd_protect_flow3_vmware_settings

Field	Description	Example
Agent IP	The IP address of the agent	172.22.6.230
Subnet mask	The subnet mask of the agent	255.255.0.0
Gateway	Agent gateway	172.22.1.2
DNS servers	DNS of the agent’s server. You can specify multiply addresses separated by commas.	1.1.1.1, 2.2.2.2

Step 4. Protection instructions#

On the fourth step there is a 'Download agent' button and protection instructions. To successfully install the agent, carefully follow the instructions on the screen:

Download agent OVA file using the 'Download agent' button above and deploy it on each ESXi host in your VMware cluster you want to protect.
Launch deployed VMs (agents) to protect machines on each host.

pd_protect_flow4_vmware

Machines will be detected and shown in a target group in a few minutes after agent deployment and launch. By default, all discovered machines will have 'Unprotected' status. Select machines you want to protect and use 'Actions -> Start Protection' to start protecting them.

Alternativaly use Download Replication Agent item on Manage Clouds page to get the agent's file.

A agent's instance must have at least 2 CPUs and 4 GB RAM and must be deployed on each of the ESXi hosts that have machines intended for replication.

Note

VMware vSphere can issue a warning about the presence of an unknown configuration parameter for the virtual machine. The service parameter "hvragent" is added deliberately and it is not a security risk.

VMware Agent configuration#

Use TUI to change network settings, IP addresses and configuration files. On the main window agent ID (in format AGENT_ID (CLOUD_ID)@CUSTOMER_ID), statuses and logs are shown. There are To console and Agent options buttons at the bottom of the window.

ao_tui_cabrio_main

Select To console if you prefer to configure via command line, otherwise press Agent options.

Note

We highly recommend to configure agent via Agent options.

In addition you can access TUI via ssh. Use user user and password px%96jA-8zX{ to log in. You can also use SSH key (if set in the cloud). The third button Exit console will appear. Use it to close ssh session.

Console#

After pressing To console the console will launch. The list of available commands will appear on the screen. Use it to configure parameters. To exit the console enter exit.

ao_tui_cabrio_console

To edit machine hostname use command sudo vi /etc/hostname or sudo nano /etc/hostname.

Note

Restart the machine to apply the changes.

Agent logs are stored in "/var/log/cabrio/cabrio.log".

Agent options#

Note

We highly recommend to use this option to configure agent.

The menu will appear on the screen.

ao_tui_cabrio_agentoptions

Warning

Changes perform here may automatically restart agent services, interrupting all running tasks. Configure the agent when it's idle.

1. Restart agent services restarts the services just right after the confirmation. The success of the restart can be tracked by the logs of the main window, which will appear immediately after the operation is confirmed.

2. Enable/disable debug logging adds/removes debug information to/from log. By default debug information is disabled.

3. Edit network. NetworkManager TUI will be launched when select this menu.

ao_tui_cabrio_NM

4. Edit hosts starts text editor GNU nano to edit data stored into /etc/hosts. Use it to specify DNS-names manually.

ao_tui_cabrio_hosts

5. Edit controller addresses is used to update IP addresses and ports. The menu will appear when clisk.

ao_tui_cabrio_addresses

Update IP address of the controller (Update all addresses at once):

ao_tui_cabrio_addresses_01

configure restAPI, receiver, report, stash, or logstash IP optionally (Update individual addresses):

ao_tui_cabrio_addresses_02

6. Edit agent common config starts text editor GNU nano to edit configuration file. If necessary update certificates, users or agents IDs. Controller addresses (can also be changed via 5. Edit controller addresses) are available for editing either.

ao_tui_cabrio_06

Warning

Do not update role and cloud_type fields.

7. Edit agent service config starts text editor GNU nano to edit service configuration file.

ao_tui_cabrio_07

Note

File content depends on the agent type.