Setup and Manage Subversion Master-Slave Replication

This guide will go through the steps to setup and maintain master-slave sites for Subversion replication. Slave sites can be used to provide faster read access to repositories with HTTP access protocol (write operations will be proxied to master). SSH repository access is not supported on slave sites.

Note: Replication is available as a separate paid feature. Please contact Helix TeamHub Sales to upgrade your plan and to enable this feature.

Master Slave Replication

Contents

Prerequisites

Configure Master Site

  1. Install and setup master site normally using Helix TeamHub Admin and add replication license. Make sure correct hostname is set also for the system on all the nodes.
  2. Create a company admin bot for replication and take a note of the credentials.
  3. Create admin and application accounts for MongoDB by executing create_mongodb_users.sh as hth user on the master site. Use existing credentials if you have already created them.
  4. Generate a keyfile as hth user to be used with MongoDB replica set authentication. Place this file on DB node on cluster setups.

    cd /var/opt/hth/shared
    openssl rand -base64 741 > mongodb-keyfile
    chmod 600 mongodb-keyfile
    
  5. Cluster setups only: Edit hth.json on DB node on master site. Make sure to merge values for the existing keys.

    {
      // Setup site as master.
      "app": {
        "is_master": true
      }
    }
    
  6. Combo setups only: Edit hth.json on master site. Make sure to merge values for the existing keys.

    {
      // Add application user credentials for MongoDB.
      "mongodb": {
        "username": "<input db username here>",
        "password": "<input db password here>"
      }
    }
    
  7. Edit hth.json on Web node (or Combo) on master site. Make sure to merge values for the existing keys.

    {
      // Setup site as master.
      "app": {
        "is_master": true,
        "hostname": "master.helixteamhub.dev"
      }
    
      // Enable and add company admin bot credentials for replication.
      "replication": {
        "enable": true,
        "username": "<input sync username here>",
        "password": "<input sync password here>"
      }
    
      // Add application user credentials for MongoDB.
      "backend": {
        "db_username": "<input db username here>",
        "db_password": "<input db password here>"
      }
    }
    
  8. Reconfigure only DB node (or Combo) on master site.

    sudo hth-ctl reconfigure
    
  9. Connect to mongo console on master site (DB node or Combo) to initialize MongoDB replica set.

    Connect to MongoDB.

    mongo --port 4002
    

    Authenticate with admin account.

    use admin
    db.auth("<input admin username here>", "<input admin password here>")
    

    Initialize replica set (replace correct hostname for master).

    var config = {
      _id: "hth",
      members: [
        {
         _id: 0,
         host: "master.helixteamhub.dev:4002",
         priority : 1
        }
      ]
    }
    rs.initiate(config)
    

    Check status of replica set and wait until primary is available. You should see a member with "stateStr": "PRIMARY".

    rs.status()
    
  10. Cluster setups only: Reconfigure rest of the Web nodes on master site.

    sudo hth-ctl reconfigure
    
  11. Install Subversion replicator on the master site in /opt/hth/application/backend/current/bin/svnreplicator.sh.

  12. Login to Helix TeamHub Admin on master site and create the master site in the Sites section. Add sites

Configure Slave Site

  1. Install Helix TeamHub slave site as Combo and mark the instance as slave before configuring Helix TeamHub in step 2.

    touch /var/opt/hth/hth_slave
    sudo hth-ctl reconfigure
    

    Bootstrap the instance using Helix TeamHub Admin and add replication license. Make sure correct hostname is set also for the system.

  2. Copy mongodb-keyfile from master site and place it in /var/opt/hth/shared/mongodb-keyfile. Make sure it is owned by hth account and has correct permissions.

    cd /var/opt/hth/shared/
    scp root@master.helixteamhub.dev:/var/opt/hth/shared/mongodb-keyfile .
    chmod 600 mongodb-keyfile
    chown hth.hth mongodb-keyfile
    
  3. Edit hth.json on the slave site. Make sure to merge values for the existing keys.

    {
      // Setup site as slave.
      "app": {
        "is_slave": true,
        "hostname": "slave1.helixteamhub.dev"
      }
    
      // Add application user credentials for MongoDB and enable slave syncer.
      "backend": {
        "db_username": "<input db username here>",
        "db_password": "<input db password here>",
        "slave_syncer_enabled": true
      }
    
      // Disable unnecessary services.
      "resque": {
        "enable": false
      },
      "resque_scheduler": {
        "enable": false
      },
      "sangria": {
        "enable": false
      },
      "streamer": {
        "enable": false
      },
      "puma": {
        "enable": false
      },
      "maven": {
        "enable": false
      }
    }
    
  4. Reconfigure slave site.

    sudo hth-ctl reconfigure
    
  5. Connect to mongo console on master site (DB node or Combo) and add new slave to the replica set.

    Connect to MongoDB.

    mongo --port 4002
    

    Authenticate with admin account.

    use admin
    db.auth("<input admin username here>", "<input admin password here>")
    

    Copy current configuration to a variable and add new slave member with 0 votes and 0 priority and unique _id.

    var config = rs.conf()
    
    config.members.push({
      "_id" : 1,
      "host" : "slave1.helixteamhub.dev:4002",
      "votes" : 0,
      "priority" : 0
    })
    

    Verify modified configuration.

    config
    

    Reconfigure replica set with new config.

    rs.reconfig(config)
    

    Wait for the slave to sync up into secondary state. You should see a member with "stateStr": "SECONDARY".

    rs.status()
    
  6. Login to Helix TeamHub Admin on master and create a new slave site in the Sites section.

Replicate a Subversion Repository

See Replicate a Large Subversion Repository first before enabling replication.

Login to Helix TeamHub Client on master site as a company or project admin and select Replication tab from repository settings. Select slave sites where the repository should be relicated to. The time until the slave repository is available for use depends on the size of the repository.

Replication enabled

Replicate a Large Subversion Repository

Initial replication can take a long time for large repositories. Consider transferring large repositories to slaves by other means before enabling replication. One option is to rsync a hotcopy in order to avoid issues with ongoing commits:

  1. Create a hotcopy of the existing repository on master site as hth user.

    svnadmin hotcopy /var/opt/hth/shared/companies/$company/projects/$project/repositories/subversion/$repo /tmp/$repo
    
  2. Transfer the repository copy to slaves without hooks (make sure file and directory permissions remain for hth user).

    rsync -a --exclude hooks/* /tmp/$repo/ root@slave1.helixteamhub.dev:/var/opt/hth/shared/companies/$company/projects/$project/repositories/subversion/$repo
    
  3. Follow the steps above in Replicate a Subversion Repository.

Remove Replication of a Repository

Login to Helix TeamHub Client on master site as a company or project admin and select Replication tab from repository settings. Mark removed sites as disabled.

Replication disabled

Update Site Details

Login to Helix TeamHub Admin on master and update site details in the Sites section. Update hth.json and reconfigure the site when changing hostname or SSL (restart also slave_syncer sudo hth-ctl restart slave_syncer for a slave site). Update MongoDB replica set configuration after changing hostname for a site.

Admin sites

Remove Slave Site

  1. Login to Helix TeamHub Admin on master site and delete the targeted slave site in the Sites section.
  2. Connect to mongo console on master site (DB node or Combo) and remove slave site from replica set:

    Connect to MongoDB.

    mongo --port 4002
    

    Authenticate with admin account.

    use admin
    db.auth("<input admin username here>", "<input admin password here>")
    

    Copy current configuration to a variable and remove the targeted slave from members.

    var config = rs.conf()
    
    // Check the zero-based index of the slave.
    config.members
    
    // Remove the slave (index is the first argument).
    config.members.splice(1, 1)
    

    Verify modified configuration.

    config
    

    Reconfigure replica set with new configuration.

    rs.reconfig(config)
    

    Check status of replica set.

    rs.status()
    
  3. Shutdown and destroy the slave site instance.

Backups and Restoration

Backups can be configured normally on master site, but restoration follows different process.

  1. Remove slave sites by following the steps in Remove Slave Site.
  2. Follow steps in Restoring Backups on master site.
  3. Create slave sites by following the steps in Configure Slave Site.
  4. Export metadata about replicated repositories on master site as hth user.

    cd /opt/hth/application/backend/current
    rake hth:replication:export_metadata
    
  5. Optional. Consider transferring repositories to slaves by other means before proceeding to the next step. See Replicate a Large Subversion Repository for more information. Metadata of each slave is saved in /var/opt/hth/shared/slave_metadata/ as json files, which can be used to obtain a list of repositories.

    cat /var/opt/hth/shared/slave_metadata/slave1.helixteamhub.dev.json
    {
      "slave_host": "slave1.helixteamhub.dev",
      "repositories": [
        {
          "company": "hth",
          "project": "platform",
          "repository": "docs",
          "path": "/var/opt/hth/shared/companies/hth/projects/platform/repositories/subversion/docs"
        }
      ]
    }
    
  6. Run slave setup task on master site as hth user for each slave. Replace filename with correct metadata file.

    cd /opt/hth/application/backend/current
    rake hth:replication:setup_slave[/var/opt/hth/shared/slave_metadata/$slave_host_name.json]
    

Troubleshooting

Updated on: 5 October 2017