GreenArrow Email Software Documentation

Delivery Event Processed Logfile Format

This document defines the “processed logfile format” which contains log entries of each delivery attempt event.

You get access to this by running the hvmail_report command with the --processed-logfile option.

Logfile format

The processed logfile format contains a single line for each delivery attempt event.

Each line contains a tab-separated set of columns. Each column’s data is guaranteed to not to contain any tabs. There is no escape character or line continuation character.

Columns of the processed logfile format

The columns are as follows:

(1) timestamp

Time of delivery attempt in seconds since the Unix epoch.

(2) channel

Value Meaning
local delivery to locally handled email address (i.e. local mailbox)
remote delivery to remote email address via SMTP

(3) status

Result of this delivery attempt.

Value Meaning
success successful delivery
failure delivery failed due to explicit rejection
failure_toolong delivery failed due to being in the queue longer than the allowed time (due to deferrals or connmaxout errors)
deferral delivery deferred for later retry
connmaxout delivery attempt prevented by throttle (this attempt would have put us over the throttle limits)

(4) is_retry

Value Meaning
1 this is a retry delivery attempt (i.e. this message was already tried before but the attempt resulted in a deferral or a connmaxout)
0 this is the first delivery attempt for this message

There are rare occurrences of where this value is 1 for a delivery attempt that is not a retry, due to the internal bouncer queue being full and overflowing messages to a different queue. THis will only happen for internally created messages (which always have a SendID ending in -ic). This should only happen if your system is under extreme pressure to process a large number of bounce messages or forwarded messages, which are also bouncing or forwarding.

(5) msguid

Internal unique ID of message. These are not repeated unless the system clock goes backwards.

Even through this looks like a floating point number, it is a string. To compare values you must use a string comparison. For example the message ID 12.34 is NOT the same as 12.340 since these are different strings.

(6) recipient

Recipient email address of message.

(7) sender

Return-Path of email message (also called envelope sender, MFROM address, or bounce address.)

(8) MTAid

The identifier for the VirtualMTA / Mail Route of this message. This is set at the time of message injection.

This field determines what IP address the message is delivered from.

MTAid may be altered at the time of injection by set_virtualmta on header_pattern. If the VirtualMTA is later changed on retry due to the evaluate_on_retry option, this field will still contain the VirtualMTA as determined at the time of injection.

(9) SendID

Internal send identifier of message. This determines which group this message belongs to.

(10) ListID

Mailing list identifier of this message.

(11) injected time

Time in seconds past the Unix epoch that this message was created.

(12) message

Success, failure, or deferral message for this delivery attempt.

Newlines in the message are replaced with the / character, null and ascii record separator characters (hex 1e) are replaced with an underscore (_), and tabs are replaced with spaces.

(13) outmtaid

If this delivery attempt was through SMTP (channel=remote), this is the primary key of the IP Address or Relay Server that was used to deliver this message.

You can see the primary keys by looking at the number at the end of the URL for the View page of each VirtualMTA in the GreenArrow Engine web interface, or by looking at the id columns of the mr_ipaddr or mr_relay tables in PostgreSQL.

(14) SendSliceID

Not yet documented.

(15) throttleid

This field is blank if the delivery attempt took place using an SMTP Relay Server or the default throttling rule.

Otherwise this field contains the ID of the Throttling Rule that was used when making the delivery attempt.

(16) clicktrackingid

This field contains the value of the X-GreenArrow-Click-Tracking-ID header, if present. Otherwise it is left blank.

(17) mx_hostname

The hostname to which this delivery attempt occurred (whether or not it came from an actual MX record).

(18) mx_ip

The IP address to which this delivery attempt occurred (whether or not it came from an actual MX record).

(19) from_address

The first email address in the “From” header as it was injected into GreenArrow.

(20) headers

A JSON object containing the headers that are configured to be logged by the log_header configuration directive.

Folded headers will simply contain the folding newlines/whitespace.

(21) message_size

An integer indicating the message size (in bytes) loaded before the delivery attempt was made. If the message was not loaded (for example if the SMTP session failed prior to reaching the DATA segment) then this value will be blank.

(22) smtp_timing

A CSV list of SMTP timings. See the log_smtp_timings directive for more information.

(23) bounce_code

For non-successful delivery attempts, this is the bounce code for this response. This is the same bounce_code that will be included in the delivery_attempt event.

(24) source_ip

The SMTP or HTTP client IP address that injected the message into GreenArrow.

(25) mailclass

The SimpleMH Mail Class used for this message.

(26) instanceid

The InstanceID for this message from either the X-GreenArrow-InstanceID or X-Job headers.

When the Mail Class has its Statistic Report Grouping set to per_instanceid_legacy, the instanceid will be generated automatically (if not explicitly set) in the YYMMDD format.

Example Data

Here is an example delivery attempt event:

Column number Column name Example Data
1 timestamp 1317299024.20073
2 channel remote
3 status success
4 is_retry 0
5 msguid 1317299024.4669464
6 recipient [email protected]
7 sender [email protected]
8 MTAid smtp1-1
9 SendID 1234
10 ListID 5678
11 injected_time 1317299024
12 message 64.21.76.32 accepted message./Remote host said: 250 ok 1317299024 qp 4973
13 outmtaid 2
14 SendSliceID 298602
15 throttleid 35
16 clicktrackingid 1234
17 mxhostname mx1.example.com
18 mxip 1.2.3.4
19 from_address [email protected]
20 headers {"Subject":["Welcome to our list!"]}
21 message_size 20123
22 smtp_timing 978,0,0,141,0,0,0,720,205,0,0,116,121,179,139,0,23,132,155,1,0,0
23 bounce_code 20
24 source_ip 127.0.0.1
25 mailclass default
26 instanceid 1234

Other Considerations

Messages with multiple recipients

If a message is injected to multiple recipients, there will be one msguid for that message. (The msguid is unique to the message.) Since each delivery attempt for each recipient will be logged, you can have multiple log entries for each msguid. In this case, you would want to use the recipient field to differentiate between these different results.

Note: we may add a field to this logfile format that will track the number of the recipient for the message, so it would be easier to differentiate between results for different recipients. or we may change the msguid field to be specific to the recipient and not the

For example:

# date | /var/hvmail/bin/mailsubj test1 [email protected] [email protected]

# hvmail_report -q ram –last ‘1 min’ –processed-logfile

1317240923.22286 remote success 0 1317240923.67920340 [email protected] [email protected] 1317240923 64.21.76.32 accepted message./Remote host said: 250 ok 1317240923 qp 25956/ 2 35 1234

1317240924.03923 remote success 0 1317240923.67920340 [email protected] [email protected] 1317240923 64.21.76.32 accepted message./Remote host said: 250 ok 1317240924 qp 25962/ 2 35 1235

You can see how each line has the same msguid.

Human-readable output

The --processed-logfile option is useful for scripting, but isn’t easily read by humans. The --human option can be added in combination with --processed-logfile to make it a bit more human-friendly.

Here is an example of both covering the same data:

  • Sample Email:

    date | /var/hvmail/bin/mailsubj test1 [email protected]
    

  • --processed-logfile:

    # hvmail_report -q ram --last '1 min' --processed-logfile
    1428427817.05678	remote	success	0	1428427816.76031064	[email protected]	[email protected]				1428427816	207.99.125.72 accepted message./Remote host said: 250 ok 1428427817 qp 15219/	2	mx.example.com	1.2.3.4
    

  • --processed-logfile --human

    # hvmail_report -q ram --last '1 min' --processed-logfile --human
    timestamp=<1428427817.05678> channel=<remote> status=<success> is_retry=<0> msguid=<1428427816.76031064> recipient=<[email protected]> sender=<[email protected]> MTAid=<> SendID=<> ListID=<> injected_time=<1428427816> message=<207.99.125.72 accepted message./Remote host said: 250 ok 1428427817 qp 15219/> outmtaid=<2> SendSliceID=<> throttleid=<> clicktrackingid=<> mx_hostname=<mx.example.com> mx_ip=<1.2.3.4>
    


Copyright © 2012–2024 GreenArrow Email