Machines replication

Machines replication consists of several steps:

  1. Installing agents and configuring replication

  2. Getting consistent machines snapshots

  3. Detecting changes on machines between two snapshots

  4. Sending deltas to the AWS or KVM Platform with regard to WAN optimization and deduplication

  5. Storing of incremental snapshots on AWS or KVM Platform with regard to the snapshot storage policies

Retrieving machines snapshots and detecting changes

Depending on the initial virtualization platform, it is possible to obtain consistent snapshots of machines in one of the following ways:

  • in case of replicating machines on VMware, getting snapshots of Windows and Linux machines is done with standard tools of VMware (VMware CBT API), leveraging a chain of internal API calls and snapshots as well as the means of the machine’s OS itself.

  • for Windows machines, consistent snapshots are acquired by means of Volume Shadow Storage (VSS) calls and tracking changes between two Application-consistent snapshots

  • for Linux machines, consistent snapshots are acquired by implementation of the VSS snapshot analog for block devices and sending changes to the snapshot store.

Attention

In case of a VMware infrastructure replication, VMware Tools must be installed on your virtual machines.

Sending deltas to storage. Data deduplication

Deltas are the accumulated changes since the last replication.

Sending deltas to AWS or KVM Platform is done using customer agent secure HTTPS protocol. This constitutes the deduplication process - the customer application only sends data that is not yet stored for the given customer allowing to significantly save the amount of transferred items, reduce load on the network and, at times, accelerate replication of machines.

In addition to WAN deduplication, optimization also includes compression of the network traffic.

Creating a group of machines

Once a customer dashboard has been created as part of product’s initial configuration for replicating machines, navigate to the main page and select machine groups. Grouping machines allows to merge them by functionality or common parameters of replication schedule.

To create a group, click Add Group at the bottom of the Customer page.

cp_add_group_btn

It is possible to add any number of groups. By default, each group inherits global settings for a replication schedule and snapshot retention policies of a customer. Editing group settings will be described below. Note that group name must be unique within the customer.

cp_add_group_dialog

Each customer must have at least one group. The Default group, which includes all machines that have not been added to other groups, is created automatically.

To move machines between groups, select them and click Move to another Group in the Actions menu. Note that if replication schedule and snapshot storage policies have not been redefined exclusively, the new group’s parameters will apply after transfer.

Machines replication and distribution between groups

Once groups have been created, you can proceed to replicate machines. To do this, follow the machine replication process (Section: ACP - Replicating machines) directly from the menu or through the group settings using the Protect new platform menu item. The second option will automatically select a group for the machine during the relevant migration process step.

pd_protect_flow1

Select the type of a replicated platform.

pd_protect_flow2

Select a group where all the replicated machines will appear (machines can be redistributed between groups later). The last step is to download an agent, which is immediately ready to be installed on customer’s infrastructure.

Agent installation and configuration

VMware Agent

Requirements

Hystax VMware Replication Agent requires the following user permissions in vSphere (Role “VMware Consolidated Backup user” in vCenter):

  • Virtual machine - Configuration – Disc Lease

  • Virtual machine - Provisioning – Allow read-only disk access

  • Virtual machine - Provisioning – Allow virtual machine files upload

  • Virtual machine - Snapshot management – Create snapshot

  • Virtual machine - Snapshot management – Remove Snapshot

Permission to access CBT is necessary for the correct performance of the application. To enable CBT:

Virtual machine - Configuration - Disc change tracking

Additionally, it is recommended to include the following global permissions:

  • Global - Disable methods

  • Global - Enable methods

  • Global - Licenses

Note

In case of using vCloud, vCenter user requires one extra permission to operate: Profile-driven storage > Profile-driven storage view

Ports for correct agent work:

Hystax VMware Replication Agent uses VMware snapshots and VMware CBT API in order to create consistent replicas of machines’ data.

This implies the following considerations regarding the VMware storage:

  • VMware snapshots consume storage to retain copy-on-write buffer, so it is recommended to have at least 10% free space available on VMware storage.

  • VMware puts additional load on storage while creating snapshots or running machines with existing snapshots.

Please consider that storage performance warning thresholds need to be adjusted in order to meet this increased load during replication.

Attention

The source machine must have VMware Tools installed manually prior to any replication procedures for it to display its network information in the target VMware ESXi correctly.

Deploying VMware Agent

VMware Agent is an external replication agent type that is deployed as a separate instance in the source environment.

Use the “Download Agent” button at the last step of the Agent download wizard to get the agent’s .ova template.

pd_download_hvragent

A agent’s instance must have at least 2 CPUs and 4 GB RAM and must be deployed on each of the ESXi hosts that have machines intended for replication.

Note

VMware vSphere can issue a warning about the presence of an unknown configuration parameter for the virtual machine. The service parameter “hvragent” is added deliberately and it is not a security risk.

To deploy a new agent, follow the instructions below:

  1. Use VMware Web Client / vSphere Client / VMware fusion.

  2. Select the downloaded .ova template of the agent.

deploy1

  1. Select a storage that the agent will be uploaded to.

deploy2

  1. Select disk provisioning and network that have Internet access.

deploy3

  1. Start the agent when deployment process is complete.

After starting agents’ VMs, machines on the respective ESXi hosts will automatically appear in the list of customer machines in the selected group. The machines will be assigned with the “Discovered” state.

VMware Agent configuration

To edit the “hosts” file of a VMware replication agent:

  1. Go to the agent command line by pressing the “To console” button in the TUI.

  2. Edit /var/run/hosts file.

  3. Restart services for the changes to take effect by pressing the “Restart services” button in the TUI.

To change the hostname of a VMware replication agent, which is “hwragent” by default:

  1. Go to the agent command line by pressing the “To console” button in the TUI.

  2. Edit the /var/run/hostname file and change the hostname of the agent machine.

  3. Reboot the VMware VM gracefully for the changes to take effect.

Agent logs can be found by clicking the To console button in the agent’s UI. The logs are stored in “/logs/hvragent.log”

To get the current network settings, use the file “/var/run/network/current-settings*”.

To change the network settings (for example, if the agent can not get an IP address by DHCP), edit the file “/var/run/network/interfaces”. Once the settings have been changed, reboot the agent.

Linux Agent

Requirements

  • Replication Agent for Linux requires sudo permissions for installation

  • Hardware:

    • Memory: 500 MB RAM

    • Disk space: 100 MB required for product installation and not less than 15% free space of the disk size for snapshots creation

  • Ports needed for the correct work:

    • Send data to Acura host - tcp/443

    • Send logs to Acura - udp/12201

Pre-built and DKMS Agent types

Acura offers a choice between replication agent builds that depends on customer’s preferences and the actual use case.

The pre-built driver package (available both for Debian/Ubuntu and CentOS/RHEL distributions) requires no additional dependencies, but the supported Linux kernel list is limited. Use pre-built package for machines that are infrequently updated or have no Internet connection to install DKMS and other dependencies.

Click on “Show supported kernels” during Step 3 of the Agent download wizard, to see the full list of natively included versions.

pd_show_kernels

The DKMS driver package will build the driver on installation. It has a broad Linux kernel support and will rebuild the driver on kernel updates. It requires DKMS, build tools and kernel headers to be installed on the machine. Use DKMS package for machines that are frequently updated or not supported by the pre-built package.

Installation

Instructions for an agent with a pre-built package:

  1. Download a deb/rpm agent installer using the respective button of the Agent download wizard or copy the suggested wget command to run it in the source VM’s terminal.

pd_prebuilt

  1. Copy the package to a Linux machine that is intended for replication.

  2. Install the agent using the following commands (superuser privileges required):

for Ubuntu/Debian machines: dpkg -i /path_to_installer_package

for RHEL/CentOS machines: rpm -i /path_to_installer_package

The machine will be registered and shown in a target group in a few minutes after agent installation.

Instructions for an agent with DKMS:

  1. Install DKMS and build requirements on the machine (superuser privileges required):

for Ubuntu/Debian machines: apt-get update && apt-get install dkms perl make gcc libelf-dev

for RHEL/CentOS machines: add EPEL repository and run yum install dkms perl make gcc elfutils-libelf-devel

  1. Install Linux header files for current kernel and future updates. Use the following commands for stock kernels (superuser privileges required):

for Ubuntu machines: apt-get update && apt-get install linux-headers-$(uname -r) linux-headers-generic

for Debian machines: apt-get update && apt-get install linux-headers-$(uname -r) linux-headers-amd64

for RHEL/CentOS machines: yum install kernel-devel-$(uname -r) kernel-devel

  1. Download a deb/rpm agent installer using the respective button of the Agent download wizard or copy the suggested wget command to run it in the source VM’s terminal.

  2. Copy the package to a Linux machine that is intended for replication.

  3. Install the agent using the following command (superuser privileges required):

for Ubuntu/Debian machines: dpkg -i /path_to_installer_package

for RHEL/CentOS machines: rpm -i /path_to_installer_package

The machine will be registered and shown in a target group in a few minutes after agent installation.

Windows Agent

Requirements

  • Replication Agent for Windows requires administrative permissions for installation

  • Hardware:

    • Memory: 2 GB RAM

    • CPU: x64 processor

    • Disk space: 100 MB required for product installation and not less than 15% free space of the disk size for VSS snapshots creation

  • Software:

    • Microsoft .NET Framework 4.0

Attention

Windows replication agent must be installed by the System Administrator or using Administrator privileges. Otherwise, it will not have enough permissions to use API or create snapshots and the replication will fail.

Installation

  1. Download the zipped agent installer package using the link from the respective “Download agents” section in ACP.

  2. Copy the archive to a Windows machine that is intended for replication.

  3. Unzip the archive and run hwragent.msi to install replication services.

The machine will be registered and shown in a target group in a few minutes after the agent installation. By default, the discovered machine will have the “Discovered” status. To start replication, select the machine and use “Actions -> Start Replication”.

Setting replication schedule

Once machines have appeared in the list and replication has been enabled for the specified machines, it is necessary to set replication schedules (Sections: ACP - Edit replication schedule).

cp_edit_schedules_dialog

Settings can be changed globally for a customer, for each individual group or for individual machines.

Settings are inherited in the following way:

  • By default, global settings are applied to all groups and machines

  • Upon editing group settings, the new settings are applied to all machines (in the group or the ones added later) with the default settings, otherwise the settings of individual machines will still be valid.

  • Upon editing settings of a machine, only the one specific machine is affected, otherwise group settings or global settings apply.

Once these steps are completed, the machines will be replicated based on schedule and rules that have been set during the configuration.