GreenArrow Email Software Documentation

Event Processor

Introduction

The Event Processor provides a configurable mechanism for delivering events from GreenArrow to an HTTP server, a logfile, or a SQL database server.

GreenArrow has two event processors – one is more modern and configured by greenarrow.conf. The other is our legacy event processor and is configured by its own JSON file.

Here’s what the modern event processor offers:

  • Deliver the same events to multiple HTTP/HTTPS endpoints.
  • Deliver different event types to one or more HTTP/HTTPS endpoints.
  • Deliver events to logfile endpoints.
  • Deliver events to SQL database endpoints.
  • Higher throughput by delivering events in batches instead of one-at-a-time.
  • Configure event generation via greenarrow.conf instead of many different configuration files.
  • Deliver to logfiles on disk with automatic file rotation.

Configuration

To enable event generation and delivery, use the event_delivery_destination configuration directive.

event_delivery_destination primary {
  event_delivery_url "https://example.com/primary-endpoint"
  event_delivery_events all
}

When using the modern event processor, you do not have to configure event generation individually as described in the Event Notification System Configuration. Simply specifying event_delivery_events is sufficient to generate the events.

Status command

GreenArrow provides commands for inspecting the status of the events queue.

This report will tell you how many events are in the queue (and how many of those events are being retried due to errors).

$ greenarrow events status
+------------------+-----------------------------+-----------------------+
| Destination Name | Event Count (First Attempt) | Event Count (Retries) |
+------------------+-----------------------------+-----------------------+
|       __legacy__ |                          10 |                   n/a |
|          primary |                           0 |                     2 |
+------------------+-----------------------------+-----------------------+

There’s also a JSON output option available.

$ greenarrow events status --json
[
    {
        "destination_name": "__legacy__",
        "destination_deleted": false,
        "event_count_first_attempt": 0,
        "event_count_retries": 0
    },
    {
        "destination_name": "primary",
        "destination_deleted": false,
        "event_count_first_attempt": 0,
        "event_count_retries": 0
    }
]

This report will tell you how many events-per-hour are in the queue.

$ greenarrow events hourly_queue __legacy__
+------------------------------+-----------------+
| Event Time (Rounded to Hour) | Events In Queue |
+------------------------------+-----------------+
|            Sep 20 2022 15:00 |               3 |
|            Sep 20 2022 16:00 |               1 |
|            Sep 21 2022 06:00 |               1 |
|            Sep 21 2022 09:00 |               1 |
|            Sep 21 2022 14:00 |               3 |
|            Sep 21 2022 15:00 |               1 |
+------------------------------+-----------------+

There’s also a JSON output option available.

$ greenarrow events hourly_queue primary --json
{
    "available_destinations": [
        {
            "name": "__legacy__",
            "deleted": false
        },
        {
            "name": "primary",
            "deleted": false
        }
    ],
    "destination_name": "primary",
    "hourly_queue": [
        {
            "hour": "2022-10-12T10:00-04:00",
            "num_events": 4
        }
    ]
}

Renaming an event delivery destination

Renaming an event delivery destination is as simple as changing its name in its event_delivery_destination configuration directive. GreenArrow will detect the rename based on the event_delivery_url not changing and will continue to deliver from the same queue of events.

If both event_delivery_destination and event_delivery_url change in the same operation, a new queue will be established and the old queue will be lost.

Delivery

Unlike the legacy event processor, this configuration will deliver events in batches (JSON arrays).

The receiving endpoint must return an HTTP Success response (200) to acknowledge reception of the event.

How events are generated vs how they are delivered

The event_delivery_events configuration directive determines which events are generated (i.e. this directive applies at the time the event occurs). The event is added to a delivery queue.

Our other event delivery configuration directives (i.e. event_delivery_url) apply at the time the event is pulled from the queue and delivered to the remote server.

Removing event delivery destinations

Eliminating an event delivery destination

If you want to remove an event delivery destination, and you do not care about receiving any events that are already in the queue for delivery, GreenArrow can accomplish this.

Here’s what you need to do:

  1. Remove the event_delivery_destination directive from the greenarrow.conf.

  2. Call greenarrow_config reload to apply that configuration change.

Be aware that events in the queue will remain for up to an hour after deleting the event destination. This means that if you delete an event delivery destination, then re-add an event delivery destination with the same name or event_delivery_url, you may resume receiving events that were generated from before you applied the configuration change. This is intentional to protect users from accidentally purging the events queue.

Winding-down an event delivery destination

If you want to remove an event delivery destination, but want to finish receiving any events that are already in the queue for delivery, GreenArrow can accomplish this.

Here’s what you need to do:

  1. Set the event_delivery_events directive to none. This will cause no further new events to be generated for this destination.

  2. Call greenarrow_config reload to apply that configuration change.

  3. Monitor greenarrow events status and wait for all events to the destination you are removing to be delivered.

  4. Remove the event_delivery_destination directive from the greenarrow.conf.

  5. Call greenarrow_config reload to apply that configuration change.

Event Processor Logs

The event processor logs are kept in /var/hvmail/log/event-processor and /var/hvmail/log/event-processor2. Use these commands to diagnose why an event is not being delivered.

For a streaming view of the log as it happens:

tail -F /var/hvmail/log/event-processor*/current | tai64nlocal

To see a particular time range of events:

logdir_select_time --start "2015-11-24 19:00" --end "2015-11-25 00:00" --dir /var/hvmail/log/event-processor | tai64nlocal

Log Diagnostics

Every three seconds, for each destination in the current configuration, the Event Processor will print the following:

2023-01-30 18:28:53.252187500 destination(logfile) summary: database_id=2 shutdown=false config=1
2023-01-30 18:28:53.252188500 destination(logfile) reader: status=(in=0/32 out=0/32) running=false num_exclusion=0 queue_exclusion=0 zero=true age=696ms
2023-01-30 18:28:53.252189500 destination(logfile) deliverer: status=(in=0/128 out=0/128) queue_pre_delivery=0/(4000-8000) batches_underway=0/100 events_underway=0
2023-01-30 18:28:53.252241500 destination(logfile) updater: status=(in=0/32 out=0/32) queue_pre_update=0 num_running=0/2 catchup=0.0%
2023-01-30 18:28:53.252242500 destination(logfile) 0 events (100.000% success); open_or_move=0ms write=0ms fsync=0ms

Much of the above is implementation diagnostics useful for GreenArrow Technical Support to help diagnose performance issues.

One particular line of note is this (for a logfile destination). It shows the success percentage, how long each batch took to open or move, and how long write and fsync system calls are taking.

2023-01-30 18:28:53.252242500 destination(logfile) 0 events (100.000% success); open_or_move=0ms write=0ms fsync=0ms

The HTTP version of this log line looks like this. It shows the number of events, the number of batches submitted to the remote server, the success percentage, and how long each batch took to submit.

2023-01-30 18:31:42.202888500 destination(http) 0 events in 0 batches (100.000% success); 0ms per batch

Starting and Stopping the Event Processors

To check the running state of the two event processor services:

hvmail_init status | grep hvmail-event-processor

To start the event processor:

svc -u /service/hvmail-event-processor*

To stop the event processor:

svc -d /service/hvmail-event-processor*

 Events Table Maintenance
Legacy Types of Events 

Copyright © 2012–2023 GreenArrow Email