Backups and Restoration

Backups are available as part of the Helix TeamHub packages, and can be activated via configuration flags. The nature of backups changes from one Helix TeamHub component to another:

Component Method Schedule Description
Assets Archival Daily at 00:00 Attachments, Avatars, Logs, Configuration files
MongoDB Archival Daily at 00:00 Mongo dumps
Repositories Incremental sync Daily at 02:00 Repository type specific tools: git, svnsync, hg

Preparation

Helix TeamHub stores backups at /var/opt/hth/backups. For production deployments the recommendation is to mount a reliable external storage that has enough space to accommodate at least 1.5 times the Helix TeamHub data set (/var/opt/hth/shared).

Enabling backups

In order to have the backups running, activate them through the configuration flags as shown below. With replication enabled, backups can only be configured on master site.

Combo

Merge the following configuration to /var/opt/hth/shared/hth.json and make sure to add the backup settings under the existing keys if they already exist in the configuration.

{
  "backend": {
    "backups": true
  },
  "mongodb": {
    "backups": true
  },
  "repos": {
    "backups": true
  }
}

Finally, apply the changes by reconfiguring Helix TeamHub:

sudo hth-ctl reconfigure

Cluster and HA

In Cluster and HA deployment the Helix TeamHub services are distributed across the server roles. In order to enable backups activate the backup flags in /var/opt/hth/shared/hth.json on the appropriate servers. Make sure to add the backup settings under the existing keys if they already exist in the configuration.

Helix TeamHub DB

{
  "mongodb": {
    "backups": true
  }
}

Please notice that by default mailing is not configured for the DB node, so in order to get notifications in case of failures happen during DB backups, you'll need to add the following:

{
  "app": {
    "email": "support@acme.com"
  }
}

Helix TeamHub Web

{
  "backend": {
    "backups": true
  },
  "repos": {
    "backups": true
  }
}

Finally, apply the changes by reconfiguring Helix TeamHub on each server:

sudo hth-ctl reconfigure

Configuring how many backups to keep before oldest backup gets removed

Archival mode backups can be configured to prune old backups. The configuration below keeps the 30 latest backup archives and deletes the oldest when a new backup occurs.

{
  "backups": {
    "keep": 30
  }
}

Restoring Backups

Since Helix TeamHub backups are modular, when it comes to restoring the system from a backup it's important to consider the following:

See Backups and Restoration for more information on how to restore backups with replication enabled.

Stopping Services

Before starting the restoration process, it's a good idea to stop all the Helix TeamHub services:

sudo hth-ctl stop

Restoring Assets

SSH into one of the Helix TeamHub Web servers (or the Combo), switch to the hth user and grab the backend_backup.tar from backup storage, which is located at /var/opt/hth/backups/backend_backup/<date>/. Finally, extract the archives and restore:

sudo su - hth
tar xvf backend_backup.tar
cd backend_backup/archives; ls *.tar.gz | xargs -i tar xvf {} -C /

Restoring MongoDB Database

SSH into the Helix TeamHub DB server (or the Combo), switch to the hth user and grab mongodb_backup.tar from the backup storage /var/opt/hth/backups/mongodb_backup/<date>/. Finally, start mongodb, extract the archives and restore:

sudo su - hth
sudo hth-ctl start mongodb
tar xvf mongodb_backup.tar; cd mongodb_backup/databases/; tar xvf MongoDB.tar.bz2

For Combo deployment without MongoDB authentication, use the following command to restore the database:

mongorestore --port 4002 --db hth_production --drop MongoDB/hth_production/

For Cluster, HA, Replication or Combo deployments with MongoDB authentication, use admin credentials to restore the database:

mongorestore --port 4002 --db hth_production -u <admin-username> -p <admin-password> --drop MongoDB/hth_production/

Restoring Repositories

Repositories are stored under the /var/opt/hth/shared/companies/<company>/projects/<repo_type>/repositories directory. In order to restore backups from the backups store, SSH into one of the Helix TeamHub Web servers (or the Combo), switch to the hth user and run the following script:

#!/bin/bash

companies=`ls /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/`
# Loop through companies
for company in $companies; do
  # Loop through projects
  projects=`ls /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/$company/projects/`
  for project in $projects; do
    echo "Restoring repositories for project $project in company $company"
    repos_dest="/var/opt/hth/shared/companies/$company/projects/$project/repositories/"
    if [ ! -d $repos_dest ]; then
      # Company/Project may have been renamed
      echo "Creating $repos_dest"
      mkdir -p $repos_dest
    fi
    rsync -av --delete /var/opt/hth/backups/repos_backup/var/opt/hth/shared/companies/$company/projects/$project/repositories/ $repos_dest
  done
done

In order to restore the repositories hooks, execute the following commands to regenerate them:

cd /opt/hth/application/backend/current
rake hth:restore:all

Starting Helix TeamHub

Finally, start back all the Helix TeamHub services:

sudo hth-ctl start

Reconfiguring Helix TeamHub

Note that if there have been changes to the hth.json restored from backups, apply those changes by running reconfigure:

sudo hth-ctl reconfigure
Updated on: 17 November 2017