Backup internal Acura DBs#

Overview#

Sometimes issues can occur with the machine hosting the Acura deployment. The problem can be solved by backups of etcd, Maria and Mongo databases, with the option to restore them on an external virtual machine. Backups must be performed regularly, which is why the process is automated using a cron job. It can be divided into two stages: creating backups and uploading the archive to external storage.

etcd and DBs Backup#

Cron job parameters are defined through the overlay. Below is an example with default values:

external_backup:
  suspend: false
  mountpoint: /acura/backup
  schedule: "@daily"
  keep: 5
  name: &external_backup_name
  startingDeadlineSeconds: 100
  image:
    repository: *external_backup_name
    tag: local

mountpoint – local backup storage path on the Acura machine
keep – maximum number of backups to retain
name – name of the external backup archive

The backup saves all etcd keys to acura_etcd-<timestamp>.dump. It also runs shell scripts to export MariaDB to acura_db_<timestamp>.sql.gz and the MongoDB events collection to acura_mongo_<timestamp>.json.gz. Only the last n files are retained (see the keep parameter).

External Backup#

The external backup copies the Acura DBs backup stored locally to the remove VM.

The external backup parameters are configured in etcd. If they are not specified, backups are assumed to be stored locally only. Example:

external_backup:
  type: object-storage 
  key: username
  secret: password
  url: http://172.22.5.88:9000
  bucket: some-bucket
  region: some-region (optional)
  backup_path: some/path/folder (optional)

type – expected values: object-storage or local
url – URL of the object storage
bucket – name of the bucket to store backups
region – storage region (optional parameter)
backup_path – path where the external backup archive will be stored (optional parameter)

If any parameter is missing (except for the region and backup_path), the external backup process fails. However, it's important to note that in case of any error, the individual local backups still remain on the machine.

The backup archives the latest backups from the etcd and dbs backup stage into a file named acura_backup_<timestamp>.tar.gz. If the appropriate parameters are set, the archive is uploaded to object storage (see the backup_path parameter). If no storage parameters are specified, the backups remain stored locally.

Recovery Process#

After the local backups of the Acura databases are uploaded to the remote VM, restore them using the alias: hx_restore_from_backup.