GreenArrow Email Software Documentation

GreenArrow Updates

We release updates to GreenArrow periodically. You can see the currently available releases at any time by viewing the GreenArrow Changelog.

Updates for Cloud Customers

We update the majority of our Cloud Customers about once a month. The updates typically occur in the first few business days of the month.

Updates For On-Premise Customers

On-Premise customers can either ask GreenArrow technical support to perform updates for them or perform updates on their end by following the instructions in this document.

How to Find Your Currently Installed Versions

Recent GreenArrow installations make their version numbers available in the web interface.

GreenArrow Engine shows the version number in the footer of each page, next to the copyright statement. For example, this system is running Engine 4.200.0:

Copyright © 2012–2020 GreenArrow Email | Version 4.200.0

To view the version number in GreenArrow Studio, navigate to “Admin” => “System”, then view the “Version Number” field:

GreenArrow Studio Version

You can also SSH into your server and run the following command to view GreenArrow’s version:

rpm -q greenarrow 2> /dev/null || dpkg -l greenarrow 2> /dev/null || rpm -q greenarrow-engine 2> /dev/null || dpkg -l greenarrow-engine

The exact format of the above command’s output depends on what Linux distribution you’re using. As an example, here’s what the output looks like in CentOS:

greenarrow-4.200.0-0.x86_64

The above indicates that GreenArrow 4.200.0 is installed.

Operating System Updates

We recommend keeping non-GreenArrow packages up-to-date. When updating both GreenArrow and non-GreenArrow packages, the safest approach is to perform non-GreenArrow package updates first.

Update Prerequisites

The GreenArrow update procedure has the following prerequisites:

  1. An experienced Linux systems administrator who can execute the update procedure.

  2. The /var/hvmail/bin/greenarrow_blockers file must exist, and when executed, it must output text telling you that it’s safe to proceed with the update. This is described in more detail in step 2 of the update procedure below.

Update Procedure

Prerequisites

Verify that you’ve met all of the prerequisites listed above before proceeding.

This section describes how to update both GreenArrow Engine and Studio.

GreenArrow Engine and Studio have some shared code, so don’t be alarmed if you’re a GreenArrow Engine only customer, and see references to GreenArrow Studio in the output of the commands you run during the update.

GreenArrow has a lot of moving parts, so we recommend reading through the entire update procedure before proceeding, and considering whether the update should be scheduled. Depending on how old your current GreenArrow version is, what hardware is in use, and how much data is present, you can expect anywhere from a few seconds to a few minutes of downtime during most updates.

  1. SSH into the GreenArrow server as the root user, or a user with equivalent permissions.

  2. Check if it’s safe to proceed with the update, and only proceed if the greenarrow_blockers command outputs text telling you that it’s safe to proceed:

    greenarrow_blockers
    

    Stop here if greenarrow_blockers returns anything other than this:

    It is safe to upgrade this system by following the instructions at:
    https://www.greenarrowemail.com/docs/greenarrow-engine/Server-Management-and-Backups/GreenArrow-Updates
    

  3. Check what version of GreenArrow is installed:

    rpm -q greenarrow 2> /dev/null || dpkg -l greenarrow 2> /dev/null || rpm -q greenarrow-engine 2> /dev/null || dpkg -l greenarrow-engine
    

  4. Read through the Version-Specific Steps section for any extra manual steps which are needed as part of the update process. Complete them at the time indicated.

  5. Update GreenArrow’s packages.

    On Red Hat-based Linux distributions, such as CentOS, run:

    yum --enablerepo=greenarrow update greenarrow
    

    On Debian-based Linux distributions, such as Ubuntu:

    1. Enable GreenArrow’s package repository:

      sed -e '/greenarrow main/ s/^#*//' -i /etc/apt/sources.list.d/greenarrow.list
      

    2. Install the updated packages:

      apt-get update && apt-get --only-upgrade install greenarrow
      

    3. Disable GreenArrow’s package repository to prevent accidental updates in the future:

      sed -e '/greenarrow main/ s/^#*/#/' -i /etc/apt/sources.list.d/greenarrow.list
      

  6. Run automated post-update steps. This step normally takes anywhere from a few seconds to a few minutes to complete:

    greenarrow update
    

  7. Complete any post-update steps that you noted after reading through the the Version-Specific Steps section.

  8. Verify that some common misconfigurations are not present. Each test gets a simple pass/fail result:

    hvmail_check_config
    

  9. Verify that all services are running normally. Any services with an abnormal state are shown in red:

    hvmail_init status
    

  10. Send yourself a test email (replacing [email protected] with your actual email address):

    date | /var/hvmail/bin/mailsubj "GreenArrow Test Message" [email protected]
    

That’s it! You’re now running the latest GreenArrow releases.

If you run into a problem and need to downgrade, then see the GreenArrow Downgrades doc.

Version-Specific Steps

The following table is in ascending chronological order. Please begin reading through it at the first release listed which comes after your currently installed version and work your down. Here are two examples:

  1. If you have GreenArrow Engine 4.1.234-0 installed and you’re updating to GreenArrow Engine version 4.1.273-0 then there are no version-specific steps to complete.

  2. If you have GreenArrow Engine 4.1.217-0 installed, then you would begin with the upgrade instructions for 4.1.226-0.

All steps in this table should be completed before updating GreenArrow’s packages unless otherwise indicated:

4.1.216

April 18, 2018

This update introduces the greenarrow.conf configuration file. If GreenArrow’s Apache or PostgreSQL installation has a customized configuration, then one or more configuration directives may need to be moved into greenarrow.conf.

For this update, please:

  1. Complete all other upgrade steps.

  2. Run the following command, and examine the output:

    greenarrow_config validate
    
    If any conflicts are found, then a warning describing the conflict and how to resolve it are displayed as part of the output.

4.1.226

August 16, 2018

Starting with this update, the contents of the /var/hvmail/control/httpd.ssl.custom.conf file get included in alll HTTPS VirtualHost definitions. Previously, it was only included in the default HTTPS VirtualHost’s definition.

Before performing this update, please review the contents of your /var/hvmail/control/httpd.ssl.custom.conf file (which is blank by default) and verify that you’re okay with it being included in all HTTPS VirtualHost definitions.

If the new behavior presents a problem, then instead of defining VirtualHosts in /var/hvmail/control/httpd.ssl.listen, define them in /var/hvmail/control/httpd.custom.conf. See our HTTP Server documentation for more details on those two files.

4.1.227

August 31, 2018

This release lowers the dynamic defaults for apache_max_clients and passenger_max_requests_in_queue, reducing overall memory utilization to be closer to what it was before the April 2018 release of Engine 4.1.216.

If you’re running Engine 4.1.215 or earlier, then your release predates dynamic defaults, so no extra steps are required.

If you’re running Engine 4.1.216 or later, then we recommend the following:

  • Before upgrading, compare your existing configured or default limits (which you can get from the show_dynamic_defaults command) to the new defaults for these limits, which are described below. Most installations can use the new defaults for optimum performance.

  • After upgrading, you may want to monitor your utilization of these limits.

The new dynamic defaults are calculated as follows:

All systems start with the following:

apache_max_clients = 150
passenger_max_requests_in_queue = 100

If the system has more than 4GB of RAM and at least 4 CPU cores, those numbers are increased:

modifier = ( memory_in_gb - 4.0 ) * 32
apache_max_clients += modifier
passenger_max_requests_in_queue += 3 * modifier / 4

4.1.230

September 18, 2018

Before installing this update, please check if the /var/hvmail/control/domainkeys.legacy directory exists.

If the directory does exist, then you’re using the old command line DKIM configuration system that was replaced in this release. You’ll need to convert any DKIM keys in that directory to a Web or API based DKIM configurations before the upgrade or the new command line system after the upgrade.

If the directory does not exist, then no additional steps are required.

4.1.233

October 17, 2018

  1. The X-GreenArrow-Signing-Key-Filename and X-GreenArrow-DomainKeys-Enable headers are no longer supported, starting with this release. If you’re currently using them, then transition away from them before proceeding with this upgrade:

    1. The recommended replacement for X-GreenArrow-Signing-Key-Filename is the X-GreenArrow-DKIM header.
    2. There is no replacement for the X-GreenArrow-DomainKeys-Enable header. DomainKeys is obsolete. DKIM should be used instead.
  2. For new installations, SimpleMH now defaults to using URL suffixes of /click and /open for clicks and opens. Previously, both clicks and opens used /click.php

    If you wish to use the new defaults, perform the update as normal, then come back and do the following:

    1. Edit /var/hvmail/control/simplemh-config.
    2. Find the definition of $CLICKTHROUGH_URL.
    3. On that line, replace /click.php with simply /click.
    4. Restart SimpleMH by running svc -t /service/hvmail-simplemh*.
4.1.274

October 3, 2019

Search for non-standard SMTP and QMQP services by running the following command:

find /service/ -name "hvmail-qmail-smtpd*" -o -name "hvmail-qmail-qmqp*" | 
grep -v "/service/hvmail-qmail-smtpd[23]*$"

If the above command produces no output, then only default services are present, so you can ignore the rest of this version’s update instructions.

If the above command does produce output, use the following update sequence:

  1. Open the run file for each non-standard SMTP and QMQP service identified above in a text editor. The path to the run file is the filesystem path that was output by the find command, plus /run. For example, /service/hvmail-qmail-smtpd4/run.

  2. Insert the following line after the existing exec 2>&1 line, and save your changes:
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/hvmail/openssl/lib/
    
  3. Perform the rest of the GreenArrow update steps.

  4. Restart all of the SMTP and QMQP services, and verify that they come back up, and stay up for at least 5 seconds:
    svc -t /service/hvmail-qmail-{smtpd,qmqp}*
    sleep 10
    svstat /service/hvmail-qmail-{smtpd,qmqp}* |
    grep -v "/service/hvmail-qmail-smtpd3:"
    

Most GreenArrow installations do not have a QMQP service, so if you receive an error indicating that the /service/hvmail-qmail-qmqp* file does not exist, that’s normal.

4.1.284

March 13, 2020

Before updating to this release, run the greenarrow_config validate command. Correct any errors, as well as any Passenger-related warnings that it reports. The Passenger warnings need to be corrected because as of this release, the only supported Passenger configuration parameters are the greenarrow.conf’s directives with a “passenger_” prefix.

4.200.0

June 1, 2020

The following extra steps are required to upgrade to this version or later:

  1. If you’re running a Red Hat-based Linux distribution, then replace:
    yum --enablerepo=greenarrow update greenarrow
    
    in the upgrade instructions with:
    yum --enablerepo=greenarrow \
      --setopt=exclude=greenarrow-engine install greenarrow
    
  2. If you’re running a Debian-based Linux distribution, then replace:

    apt-get --only-upgrade install greenarrow
    
    in the upgrade instructions with:
    apt-get install greenarrow
    

  3. After you run the yum or apt-get package command above, restart all of GreenArrow:
    service greenarrow restart
    

4.212.0

November 12, 2020

Anniversary autoresponders in Studio were using their delay settings inverted of what was intended. So if you have an anniversary autoresponder set to send “2 days before at 6pm”, it would actually send “2 days after at 6pm” (the reverse is also true - so “4 days after at 12pm” would be treated as “4 days before at 12pm”). This behavior has been fixed.

You can review your anniversary autoresponders by following the procedure below.

  1. Disable the Studio worker:

    svc -d /service/hvmail-studio-worker
    

  2. Upgrade GreenArrow.

  3. Review your anniversary autoresponders with the following command:

    /var/hvmail/studio/script/print_anniversary_autoresponders
    

  4. If you’d like to invert the offsets in order to maintain the old behavior: Run the command below to invert all of the offsets or edit the offsets manually in the user interface if there are only some that you want modified.

    /var/hvmail/studio/script/invert_autoresponder_anniversary_shift
    

  5. Enable the Studio worker:

    svc -u /service/hvmail-studio-worker
    

If you are confident you have no anniversary autoresponders that would be adversely affected by this change, you can perform a normal upgrade without the above procedure.

4.217.0

December 22, 2020

This release adds the greenarrow bounce_recovery tool to enable recovery from erroneous bounce results. We added the tool in response to the December 2020 Gmail outages during which Google generated bounces that indicated valid email addresses were invalid.

To restore subscribers that were improperly deactivated because of the Gmail outage, we recommend upgrading to this release or later of GreenArrow, then completing the instructions in the December 2020 Gmail Outages document.

4.235.1

September 20, 2021

This release adds a greenarrow_fix_v4.209.0_duplicate_links tool to fix an error in Studio’s link tracking. This error could cause multiple links (in GreenArrow’s database) to be created for the same URL, resulting in incorrect reporting and segmentation of subscriber clicks.

After running hvmail_migrate migrate, run this command to correct the incorrect link tracking:

greenarrow_fix_v4.209.0_duplicate_links
Running this command will also prevent links from being tracked incorrectly in the future.

4.242.0

November 5, 2021

This release upgrades Apache to a version that requires a specific system call to be available by the kernel (getrandom). If you’re running a very old kernel, this system call might not be available.

To test that you’re not in a bad state, run the following command:

ruby -e ' ( syscall(318,"_"*17,17,0) == 17 or
fail("getrandom not found") ) ; puts("getrandom found") '

It should output simply:

getrandom found

If you see something like this, then you need to upgrade your kernel and reboot prior to upgrading GreenArrow:

Traceback (most recent call last):
    1: from -e:1:in `<main>'
-e:1:in `syscall': Function not implemented (Errno::ENOSYS)

4.253.0

March 1, 2022

This release adds to Studio the ability to segment on whether an open was privacy or non-privacy. In support of this, some extra data needs to be migrated.

Run this command to complete the migration:

greenarrow update populate_privacy_recent_activities

This migration may take several hours to complete, but GreenArrow is fully operational during the migration.

4.293.0

August 24, 2023

This release updated GreenArrow’s systemd service file to be more resilient to non-default system OOMPolicy configurations.

Run this command to effect this change:

systemctl daemon-reload

4.304.0

December 21, 2023

This release adds support for PostgreSQL 16. We recommend Upgrading to PostgreSQL 16 after you’ve upgraded to this release of GreenArrow. Upgrading PostgreSQL is optional for now. PostgreSQL 16 comes with performance enhancements that may speed up your GreenArrow installation.