Server Migrations

Table of Contents
Intro
Prerequisites
Non-GreenArrow Files and Services
Migration Procedure
Post-Migration Configuration
Final Testing
Disable Old Server
Let Us Know You’ve Migrated

Intro

This document describes how to migrate a GreenArrow installation from one server to another. Please contact GreenArrow technical support if you would rather have GreenArrow migrate your GreenArrow installation.

Prerequisites

The migration procedure described on this page has some prerequisites:

An experienced Linux systems administrator who can execute the migration procedure.
The server that you’re migrating from must be running GreenArrow 4.200.0 or later. If you’re using an older release, then see these instructions.

Here’s a command you can run to check what version is installed:
```
rpm -q greenarrow 2> /dev/null || dpkg -l greenarrow 2> /dev/null || echo "GreenArrow is either not installed, or is older than version 4.200.0."
```
The server that you’re migrating to must run a Linux distribution from the same family as the server that you’re migrating from. Here are two examples:
1. If your existing server runs CentOS 8, then the migration must be made to a Red Hat-based distribution, such as CentOS 7 or 8.
2. If your existing server runs Ubuntu 18.04, then the migration must be made to a Debian-based distribution, such as Ubuntu 20.04 or Debian 11.
Most GreenArrow installations use PostgreSQL 9.5, which can run on all of our supported Linux distributions. Some installations use PostgreSQL 8.3, though, which comes with Linux distribution restrictions.

You can check which version of PostgreSQL your existing server is running by checking where the /var/hvmail/postgres/default symlink points to. If it points to /var/hvmail/postgres/9.5, then you’re running PostgreSQL 9.5. If it points to /var/hvmail/postgres/8.3, then you’re running PostgreSQL 8.3:
```
ls -l /var/hvmail/postgres/default
```
If you’re running PostgreSQL 8.3, then the following restrictions apply:
1. PostgreSQL 8.3 is only supported by Red Hat-based distributions, like CentOS 7 and 8.
2. PostgreSQL 8.3 can run on RedHat 8.x-based Linux distributions like CentOS 8, but will not support using TLS for incoming connections. If you wish to use both PostgreSQL 8.3 and TLS for incoming PostgreSQL connections, then you must run a RedHat 7.x or earlier Linux distribution, like CentOS 7.
Please contact GreenArrow technical support if you’d like us to migrate you from PostgreSQL 8.3 to 9.5.
The server should have enough disk space to hold the migrated data. The vast majority of GreenArrow’s data is located within the /var/hvmail directory and the migrated data will occupy approximately the same amount of disk space after the migration as it did before.

If the specs of the destination server are identical to or greater than the server that you’re migrating from, and you didn’t have any disk space issues before the migration, then you’re covered.
The /var/hvmail/control/UPGRADE_BLOCKER.txt file must not exist. This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software. Those same non-standard steps will need to be performed during the migration. Contact GreenArrow technical support to arrange to have those steps applied.

Non-GreenArrow Files and Services

These migration instructions do not take into account any non-GreenArrow files or services that you may also wish to migrate.

Migration Procedure

Some of the steps below are Linux distribution specific. For the sake of brevity, we refer to Red Hat-based Linux distributions, such as CentOS, Red Hat Enterprise Linux or Scientific Linux as “CentOS”, and Debian-based distributions, such as Ubuntu and Debian as “Debian”.

Here’s how to migrate GreenArrow to a new server. All steps should be completed on the new server unless otherwise indicated:

If you’re adding, removing or changing any IPs as part of the migration, then consider configuring their DNS records and feedback loop registrations in advance. Instructions for adding new IPs are here.
Authorize the new server’s root account to SSH into the old server as root.
Log in to the new server as the root user using the bash shell.
Install packages that are used by the migration process:

On CentOS systems, run:
```
yum install ruby rsync
```
On Debian systems, run:
```
apt-get install ruby wget gnupg rsync
```
Make an entry in the new server’s /etc/hosts file, pointing the oldserver hostname to an IP on the old server that has an SSH server running. For example:
```
echo "1.2.3.4 oldserver" >> /etc/hosts
```
This entry is needed because we’re going to run some commands later on which connect to oldserver.
Verify that you can SSH into the old server from the new server by running:
```
ssh oldserver
```
If a non-default port is used for ssh then you can tell the SSH client to use it by creating or updating /root/.ssh/config. For example:
```
Host oldserver
  Port 2222
```
Take a backup of the old server. A migration normally only uses some of the files that get created by a backup, but we recommend that you hold onto the entire backup directory to increase the odds of being able to recover if something goes wrong during the migration. For extra safety, you may want to copy this directory to some media that’s not attached to either server.
Copy the following files from the backup directory to the /var/migrate/ directory on the new server:
1. greenarrow-packages.txt
2. greenarrow-repository-access-key.txt
3. hvmail_reinstall_packages
4. postgresql-tablespaces.txt
Check the /var/migrate/postgresql-tablespaces.txt file for any tablespaces which specify a location.

Here’s an example of what this file might look like. In this example, the tablespace named “data2” is the only one which has a location (/media/data2) specified:
```
                     List of tablespaces
    Name    |   Owner    |              Location
------------+------------+------------------------------------
 data2      | greenarrow | /media/data2
 pg_default | postgres   |
 pg_global  | postgres   |
(3 rows)
```
If there are no tablespaces which specify a location, then you can move onto the next step.

If there are any tablespaces which specify a location, then re-create their directories on the new server. For example:
```
mkdir /media/data2
```
Also, rsync each of the directories identified above from the old server to the new server. You should both run each rsync invocation now, and record it so that it can be run again later after hvmail_server_migration has run to completion. For example:
```
rsync -a --delete oldserver:/media/data2/ /media/data2
```
View the contents of the /var/migrate/greenarrow-repository-access-key.txt file. This key is needed for the next step.
Complete the Prerequisites section of our Installation Guide. If you get to the “Install GreenArrow” step in the Installation Guide, you’ve gone too far.

Patch hvmail_reinstall_packages:

sed -i 's|yum -y --enablerepo=greenarrow install|yum -y --setopt=obsoletes=0 --enablerepo=greenarrow install|g' /var/migrate/hvmail_reinstall_packages

Install GreenArrow’s packages on the new server:
```
/var/migrate/hvmail_reinstall_packages /var/migrate/greenarrow-packages.txt
```
If the above command encounters an error that you’re able to correct, then it’s safe to run it again.
GreenArrow packages make some changes to environment settings. To activate the changes, either log out, then back into the server or run the following commands. If you ’re logged into the server in multiple windows, then you’ll need to complete this step in each window:
```
. /etc/profile.d/greenarrow-engine.sh
. /etc/profile.d/greenarrow-ruby.sh
. /etc/profile.d/greenarrow-studio.sh
. /etc/profile.d/greenarrow-passenger.sh
```
Run the migration script for the first time. This can safely be done while GreenArrow is still running on the old server:
```
/var/hvmail/libexec/hvmail_server_migration --no-exit-on-error
```
This migration script could take anywhere from minutes to hours to run, depending on how much data there is to transfer. It’s safe to run the script multiple times, as long as only one instance is running at any given point in time.

rsync is used for most of the data transfer, so all other things being equal, the closer together two invocations of this script are to each other, the faster the second invocation will run.

Stop GreenArrow on the old server:

systemctl stop greenarrow.service
systemctl disable greenarrow.service

Run the migration script again. This will synchronize any files which changed since the last time the script was run:
```
/var/hvmail/libexec/hvmail_server_migration
```
If the migration script completes without error, then its last line of output will read No errors were detected. If the last line contains anything else, review the remaining output for issues. Once any issues have been corrected, it’s safe to re-run hvmail_server_migration.
If you identified any additional rsync invocations to run while you were inspecting postgresql-tablespaces.txt earlier, run them now.
If there are any IP addresses to move from the old server to the new one, move them now.
This step is not needed for Let’s Encrypt certificates that are configured automatically.

If you’re using the Legacy Let’s Encrypt configuration method, then remove all of that method’s certificates from the HTTP TLS configuration file. These certificates are stored outside of GreenArrow’s directories, so they are not migrated by this procedure. Take note of which certificates you’ve removed in case you wish to add them back in later. There’s a step later in this procedure for doing that.

Start GreenArrow on the new server:

systemctl start greenarrow.service
systemctl enable greenarrow.service

On Debian systems, disable the GreenArrow package repos:

sed -i 's/^deb /#deb /' /etc/apt/sources.list.d/greenarrow.list

Check whether greenarrow.conf has dns_cache_service_run set:
```
grep dns_cache_service_run /var/hvmail/control/greenarrow.conf
```
1. If dns_cache_service_run is set to true or yes, then configure the /etc/resolv.conf file so that the only nameserver line is set to 127.0.0.1. For example:
```
echo "nameserver 127.0.0.1" > /etc/resolv.conf
```
2. If dns_cache_service_run is set to false or no, or is not specified, then configure the /etc/resolv.conf file to query one or more recursive DNS resolvers. For example, if you would like to query the Google and Cloudflare public DNS servers, you could populate the file with:
```
echo "nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 1.1.1.1" > /etc/resolv.conf
```
Verify that different nameservers aren’t configured outside of /etc/resolv.conf. If the commands below return any output, then verify that it matches your desired configuration:

On CentOS systems, run:
```
find /etc/sysconfig/network-scripts/ifcfg-* /etc/sysconfig/network -type f | xargs grep 'DNS=' | grep -v 'PEERDNS='
```
On Debian systems, run:
```
grep dns-nameservers /etc/network/interfaces
```
Check if the /var/hvmail/control/UPGRADE_BLOCKER.txt file exists:
```
cat /var/hvmail/control/UPGRADE_BLOCKER.txt
```
This file is only present on GreenArrow installations where non-standard steps are required to perform updates of GreenArrow’s software.

If the file is missing, then you can move onto the next step.

If the file is present, then contact GreenArrow technical support to have those non-standard steps applied.
Complete the Performance Tuning section of our Installation Guide.

Post-Migration Configuration

If you’re adding, removing or changing any IPs as part of the migration, then they should be added into GreenArrow’s configuration now. Instructions for adding new IPs are here.
This step is not needed for Let’s Encrypt certificates that are configured automatically.

Optionally, add Legacy Let’s Encrypt certificates to the HTTP Server’s TLS configuration.
If there are any other configuration changes desired as part of this migration, make them now.

Final Testing

Verify that some common misconfigurations are not present. Each test gets a simple pass/fail result:
```
hvmail_check_config
```
Verify that all services are running normally. Any services with an abnormal state are shown in red:
```
service greenarrow status
```
The hvmail-qmail-smtpd3 and hvmail-dnscache services are down by default. All other services should be reported as up unless you shut them down intentionally.
Send yourself a test email (replacing [email protected] with your actual email address):
```
date | /var/hvmail/bin/mailsubj "GreenArrow Test Message" [email protected]
```
Remove the entry that you made earlier for oldserver in the new server’s /etc/hosts file.

Congratulations, you’re finished migrating GreenArrow to the new server! As a next step, we recommend following up with GreenArrow technical support to request that we review the migrated GreenArrow installation for issues.

Continue to the next section to prevent the old server from coming back online, and attempting to send duplicate emails and/or use the same license key in two locations at the same time.

Disable Old Server

Remove GreenArrow’s licenses from the old server:

rm -f /var/hvmail/control/license_key /var/hvmail/control/bounce.boogie_studio_api_license_key

Delete the queued messages on the old server:

find /var/hvmail/qmail-ram/savedqueue -type f -delete
find /var/hvmail/qmail-bounce/savedqueue -type f -delete
cd /var/hvmail/qmail-disk/queue/
find info -type f -delete
find mess -type f -delete
find remote -type f -delete
find intd -type f -delete
find local -type f -delete
find todo -type f -delete
find bounce -type f -delete
find batches -type f -delete

Let Us Know You’ve Migrated

Contact GreenArrow technical support to update your server information. This is not required, but it helps prevent potential licensing issues.