GreenArrow Updates

This update introduces the greenarrow.conf configuration file. If GreenArrow’s Apache or PostgreSQL installation has a customized configuration, then one or more configuration directives may need to be moved into greenarrow.conf.

For this update, please:

1. Complete all other upgrade steps.

2. Run the following command, and examine the output:

   greenarrow config validate
   

   If any conflicts are found, then a warning describing the conflict and how to resolve it are displayed as part of the output.

Starting with this update, the contents of the /var/hvmail/control/httpd.ssl.custom.conf file get included in alll HTTPS VirtualHost definitions. Previously, it was only included in the default HTTPS VirtualHost’s definition.

Before performing this update, please review the contents of your /var/hvmail/control/httpd.ssl.custom.conf file (which is blank by default) and verify that you’re okay with it being included in all HTTPS VirtualHost definitions.

If the new behavior presents a problem, then instead of defining VirtualHosts in /var/hvmail/control/httpd.ssl.listen, define them in /var/hvmail/control/httpd.custom.conf. See our HTTP Server documentation for more details on those two files.

This release lowers the dynamic defaults for apache_max_clients and passenger_max_requests_in_queue, reducing overall memory utilization to be closer to what it was before the April 2018 release of Engine 4.1.216.

If you’re running Engine 4.1.215 or earlier, then your release predates dynamic defaults, so no extra steps are required.

If you’re running Engine 4.1.216 or later, then we recommend the following:

• Before upgrading, compare your existing configured or default limits (which you can get from the show_dynamic_defaults command) to the new defaults for these limits, which are described below. Most installations can use the new defaults for optimum performance.

• After upgrading, you may want to monitor your utilization of these limits.

The new dynamic defaults are calculated as follows:

All systems start with the following:

apache_max_clients = 150
passenger_max_requests_in_queue = 100

If the system has more than 4GB of RAM and at least 4 CPU cores, those numbers are increased:

modifier = ( memory_in_gb - 4.0 ) * 32
apache_max_clients += modifier
passenger_max_requests_in_queue += 3 * modifier / 4

Before installing this update, please check if the /var/hvmail/control/domainkeys.legacy directory exists.

If the directory does exist, then you’re using the old command line DKIM configuration system that was replaced in this release. You’ll need to convert any DKIM keys in that directory to a Web or API based DKIM configurations before the upgrade or the new command line system after the upgrade.

If the directory does not exist, then no additional steps are required.

1. The X-GreenArrow-Signing-Key-Filename and X-GreenArrow-DomainKeys-Enable headers are no longer supported, starting with this release. If you’re currently using them, then transition away from them before proceeding with this upgrade:

   1. The recommended replacement for X-GreenArrow-Signing-Key-Filename is the X-GreenArrow-DKIM header.
   2. There is no replacement for the X-GreenArrow-DomainKeys-Enable header. DomainKeys is obsolete. DKIM should be used instead.

2. For new installations, SimpleMH now defaults to using URL suffixes of /click and /open for clicks and opens. Previously, both clicks and opens used /click.php

   If you wish to use the new defaults, perform the update as normal, then come back and do the following:

   1. Edit /var/hvmail/control/simplemh-config.
   2. Find the definition of $CLICKTHROUGH_URL.
   3. On that line, replace /click.php with simply /click.
   4. Restart SimpleMH by running svc -t /service/hvmail-simplemh*.

Search for non-standard SMTP and QMQP services by running the following command:

find /service/ -name "hvmail-qmail-smtpd*" -o -name "hvmail-qmail-qmqp*" | 
grep -v "/service/hvmail-qmail-smtpd[23]*$"

If the above command produces no output, then only default services are present, so you can ignore the rest of this version’s update instructions.

If the above command does produce output, use the following update sequence:

1. Open the run file for each non-standard SMTP and QMQP service identified above in a text editor. The path to the run file is the filesystem path that was output by the find command, plus /run. For example, /service/hvmail-qmail-smtpd4/run.

2. Insert the following line after the existing exec 2>&1 line, and save your changes:

   export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/hvmail/openssl/lib/
   

3. Perform the rest of the GreenArrow update steps.

4. Restart all of the SMTP and QMQP services, and verify that they come back up, and stay up for at least 5 seconds:

   svc -t /service/hvmail-qmail-{smtpd,qmqp}*
   sleep 10
   svstat /service/hvmail-qmail-{smtpd,qmqp}* |
   grep -v "/service/hvmail-qmail-smtpd3:"
   

Most GreenArrow installations do not have a QMQP service, so if you receive an error indicating that the /service/hvmail-qmail-qmqp* file does not exist, that’s normal.

Before updating to this release, run the greenarrow config validate command. Correct any errors, as well as any Passenger-related warnings that it reports. The Passenger warnings need to be corrected because as of this release, the only supported Passenger configuration parameters are the greenarrow.conf’s directives with a “passenger_” prefix.

The following extra steps are required to upgrade to this version or later:

1. On RHEL systems, replace:

   yum --enablerepo=greenarrow update greenarrow
   

   in the upgrade instructions with:

   yum --enablerepo=greenarrow \
     --setopt=exclude=greenarrow-engine install greenarrow
   

2. On Debian systems, replace:

   apt-get --only-upgrade install greenarrow
   

   in the upgrade instructions with:

   apt-get install greenarrow
   

3. After you run the yum or apt-get package command above, restart all of GreenArrow:

   service greenarrow restart
   

Anniversary autoresponders in Studio were using their delay settings inverted of what was intended. So if you have an anniversary autoresponder set to send “2 days before at 6pm”, it would actually send “2 days after at 6pm” (the reverse is also true - so “4 days after at 12pm” would be treated as “4 days before at 12pm”). This behavior has been fixed.

You can review your anniversary autoresponders by following the procedure below.

1. Disable the Studio worker:

   svc -d /service/hvmail-studio-worker
   

2. Upgrade GreenArrow.

3. Review your anniversary autoresponders with the following command:

   /var/hvmail/studio/script/print_anniversary_autoresponders
   

4. If you’d like to invert the offsets in order to maintain the old behavior: Run the command below to invert all of the offsets or edit the offsets manually in the user interface if there are only some that you want modified.

   /var/hvmail/studio/script/invert_autoresponder_anniversary_shift
   

5. Enable the Studio worker:

   svc -u /service/hvmail-studio-worker
   

If you are confident you have no anniversary autoresponders that would be adversely affected by this change, you can perform a normal upgrade without the above procedure.

This release adds the greenarrow bounce_recovery tool to enable recovery from erroneous bounce results. We added the tool in response to the December 2020 Gmail outages during which Google generated bounces that indicated valid email addresses were invalid.

To restore subscribers that were improperly deactivated because of the Gmail outage, we recommend upgrading to this release or later of GreenArrow, then completing the instructions in the December 2020 Gmail Outages document.

This release is a Milestone Version.

This release adds a greenarrow_fix_v4.209.0_duplicate_links tool to fix an error in Studio’s link tracking. This error could cause multiple links (in GreenArrow’s database) to be created for the same URL, resulting in incorrect reporting and segmentation of subscriber clicks.

After running hvmail_migrate migrate, run this command to correct the incorrect link tracking:

greenarrow_fix_v4.209.0_duplicate_links

This release upgrades Apache to a version that requires a specific system call to be available by the kernel (getrandom). If you’re running a very old kernel, this system call might not be available.

To test that you’re not in a bad state, run the following command:

ruby -e ' ( syscall(318,"_"*17,17,0) == 17 or
fail("getrandom not found") ) ; puts("getrandom found") '

It should output simply:

getrandom found

If you see something like this, then you need to upgrade your kernel and reboot prior to upgrading GreenArrow:

Traceback (most recent call last):
    1: from -e:1:in `<main>'
-e:1:in `syscall': Function not implemented (Errno::ENOSYS)

This release adds to Studio the ability to segment on whether an open was privacy or non-privacy. In support of this, some extra data needs to be migrated.

Run this command to complete the migration:

greenarrow update populate_privacy_recent_activities

This migration may take several hours to complete, but GreenArrow is fully operational during the migration.

This release updated GreenArrow’s systemd service file to be more resilient to non-default system OOMPolicy configurations.

Run this command to effect this change:

systemctl daemon-reload

This release adds support for PostgreSQL 16. We recommend Upgrading to PostgreSQL 16 after you’ve upgraded to this release of GreenArrow. Upgrading PostgreSQL is optional for now. PostgreSQL 16 comes with performance enhancements that may speed up your GreenArrow installation.

This release adds a dependency on the systemd-timesyncd.service unit to GreenArrow’s SystemD service. We recommend verifying that time synchronization works.

This can be done by installing and troubleshooting the new time synchronization service before upgrading GreenArrow.

Intructions to install and troubleshoot the service are listed in this document.

This release is a Milestone Version.

This release is a Milestone Version.