<img src="https://d5nxst8fruw4z.cloudfront.net/atrk.gif?account=lYCzn1QolK10N8" style="display:none" height="1" width="1" alt="">

HTTP Submission API

Overview

GreenArrow Engine provides an HTTP message submission API.

The information is provided to the API as a POST or PUT request of JSON content. (The header of content-type: application/json must be provided.)

The API provides a response of JSON content with a content type of application/json.

The uploaded information can be compressed using gzip or deflate compression. The header of Content-Encoding: gzip or Content-Encoding: deflate must be provided when doing this.

There is a 10MB size limit per submission. This limit is applied before a compressed payload is uncompressed, so if, for example, a 25MB payload compresses down to 8MB, you can submit it, but if it compresses down to 11MB, you cannot.

Authentication is performed by providing a username/password for an Email User in the JSON document.

The API is provided at /api/v1/send.json on your GreenArrow server’s HTTP service.

Example

Example JSON document:

{
        "username":"test@example.com",
        "password":"test",
        "message":{
                "html":"html content goes <b>here</b>",
                "text":"text content goes here",
                "subject":"this is the subject",
                "to":[
                        {
                                "email":"recipient@example.com",
                                "name":"John Doe"
                        }
                ],
                "from_email":"from@example.com",
                "from_name":"Your Company"
        }
}

Example JSON reply for success:

{"success":1,"message_id":"mid-990f39eb71e52cc5050b7b36ada937d4@example.com"}

Example JSON reply for failure:

{"success":0,"error":"no data in POST or PUT payload"}

or:

{"success":0,"error":"incorrect username\/password"}

Submitting this request with curl:

cat <<'EOT' > post_body.txt
{
        "username":"test@example.com",
        "password":"test",
        "message":{
                "html":"html content goes <b>here</b>",
                "text":"text content goes here",
                "subject":"this is the subject",
                "to":[
                        {
                                "email":"recipient@example.com",
                                "name":"John Doe"
                        }
                ],
                "from_email":"from@example.com",
                "from_name":"Your Company",
                "mailclass":"trans",
                "headers":{
                        "X-foo":"bar"
                }
        }
}
EOT

curl -X POST -H "Content-Type: application/json" --data-binary @post_body.txt http://127.0.0.1/api/v1/send.json

The API can also be accessed through SSL:

(Remove the --insecure option if you have a valid SSL certificate.)

curl -X POST -H "Content-Type: application/json" --data-binary @post_body.txt --insecure https://127.0.0.1/api/v1/send.json

Compression

The content can be gzip or deflate compressed. This allows payloads with lots of messages to be reduced in size.

This requires the Content-Encoding: gzip or Content-Encoding: deflate header on the HTTP request.

Example:

cat <<'EOT' | gzip -c -9 > post_body.txt.gz
{
        "username":"test@example.com",
        "password":"test",
        "message":{
                "html":"html content goes <b>here</b>",
                "text":"text content goes here",
                "subject":"this is the subject",
                "to":[
                        {
                                "email":"recipient@example.com",
                                "name":"John Doe"
                        }
                ],
                "from_email":"from@example.com",
                "from_name":"Your Company",
                "mailclass":"trans",
                "headers":{
                        "X-foo":"bar"
                }
        }
}
EOT

curl -X POST -H "Content-Type: application/json" -H "Content-Encoding: gzip" --data-binary @post_body.txt.gz http://127.0.0.1/api/v1/send.json

Request Document

  • username - email address of an email user (NOT the username of GreenArrow web-interface login)
  • password - password of above user
  • message - Object containing the following fields:
id

string

/

optional

Extra identifier for the message. This value is returned in the response in order to assist in mapping specific messages to responses. It is not otherwise used. It is not the same as the Message-ID header.

html

string

HTML version of content for email message. Either the html or text key is required. Both may be specified.

text

string

Text version of content for email message Either the html or text key is required. Both may be specified.

subject

string

/

required

Subject line of the email.

to

hash

/

required



List of recipients of email. Each entry in the list is an object consisting of:

email

string

/

required

The recipient’s email address.

name

string

/

optional

The recipient’s name. This will appear in the To: header.

from_email

string

/

required

Email address to use in From header.

from_name

string

/

optional

Name to use in From header.

return_path

string

/

optional

Return-Path to use. This is also known as the bounce address and envelope sender. Note that this gets overwritten if SimpleMH is configured to set the Return-Path for you.

mailclass

string

/

default: The default Mail Class

Name of the SimpleMH Mail Class to use. This key has the same effect as the X-GreenArrow-MailClass header.

message_id_domain

string

/

optional

Fully qualified domain name to use in the domain portion of the Message-ID header generated by GreenArrow. This is typically the hostname of the system that generated the message. The domain listed in /var/hvmail/control/me is used by default. If the message that you inject already has a Message-ID header set, then it is left intact.

headers

hash

/

optional

Additional headers to add to the email. The key is the header name and the value is the raw text of the value of the header. (For example, to set the Message ID of the email, specify {'Message-ID':'<foo@bar.com>'}. Notice the brackets around the value since this is the required format of the Message-ID header as defined in RFC 5322.)

Request for Multiple Messages

Up to 500 messages may be submitted per API request.

(Note: Not all messages in a batch are guaranteed to be queued. A response will be provided within 31 seconds with as many messages as could be queued in that time. See the section “Message Acceptance Throttling and Back-Pressure” below.)

To submit multiple messages in a single request, provide a document like this:

{
        "username":"test@example.com",
        "password":"test",
        "messages":[
                {
                        "html":"html content goes <b>here</b>",
                        "text":"text content goes here",
                        "subject":"this is the subject",
                        "to":[
                                {
                                        "email":"recipient-1@example.com",
                                        "name":"John Doe"
                                }
                        ],
                        "from_email":"from@example.com",
                        "from_name":"Your Company"
                },
                {
                        "html":"html content goes <b>here</b>",
                        "text":"text content goes here",
                        "subject":"this is the subject",
                        "to":[
                                {
                                        "email":"recipient-2@example.com",
                                        "name":"Bob Smith"
                                }
                        ],
                        "from_email":"from@example.com",
                        "from_name":"Your Company"
                }
        ]
}

The JSON document most be an object with the following keys:

Key Description
username Same as single message document
password Same as single message document
messages A list containing one or more objects which contain the same information as the message key in a single message document.

The output can be in two formats.

(1) An error that applies globally to the entire request:

{"success":0,"error":"incorrect username\/password"}

(2) Or the overall request succeeded and a specific success value is provided for each message:

{
        "success":1,
        "messages":[
                {
                        "success":1,
                        "message_id":"mid-90df674d4c98c1aa647eaca724086ef0@example.com",
                        "attempted":1,
                        "id":"1"
                },
                {
                        "success":1,
                        "message_id":"mid-e20664ffd71475b270b586889872e4f7@example.com",
                        "attempted":1,
                        "id":"2"
                }
        ]
}

If there is an internal error, then the subsequent messages in the request will not be attempted. For example:

{
        "success":1,
        "messages":[
                {
                        "success":1,
                        "message_id":"mid-90df674d4c98c1aa647eaca724086ef0@example.com",
                        "attempted":1,
                        "id":"1"
                },
                {
                        "success":0,
                        "error":"internal error: unable to make socket connection (Connection refused)",
                        "attempted":1,
                        "id":"2"
                },
                {
                        "success":0,
                        "error":"not attempting due to previous internal errors",
                        "attempted":0,
                        "id":"3"
                }
        ]
}

In this request of three messages, the second message encountered an internal error so the third message was not even attempted. The attempted key provides data on if the message was even attempted.

It is very important to handle error conditions and retry messages appropriately.

If the attempted value is 0 for a message, this lets you know that you should retry it.

Here are the two error messages that cause messages to not be attempted:

  • not attempting due to previous internal errors – a previous message encountered an internal error, so the rest of the messages in the batch are not attempted and return this error message. Please retry these messages in a different batch.

  • not attempting because previous messages have taken too long – this happens when the system is not able to pre-process messages as fast as you are trying to send them to this API (see “Throttling” below). Only the messages that can be accepted within about 30 seconds will be accepted, and the rest of the messages in the batch will not be attempted and return this error. This is put in place so that an HTTP timeout does not prevent you from getting any answer to the HTTP request.

HTTP keep-alive

For maximum speed it is recommended to use HTTP keep-alive in your HTTP API, so that you re-use the same TCP connection for each new request.

This can have a significant performance impact because TCP slow-start is bypassed for subsequent requests.

Message Acceptance Throttling and Back-Pressure

Message batches may not be larger than 500 messages.

However, not all messages in a batch are guaranteed to be queued.

The HTTP API will queue as many messages as possible within 31 seconds and then provide a status response indicating which messages were queued. (This 31 seconds does not include network delay for transferring data.) The purpose of replying within 31 seconds is to prevent HTTP timeouts – which would prevent the caller from knowing whether messages were successfully queued or not.

This is how GreenArrow throttles how fast messages are accepted through the HTTP API so that they can not be accepted faster than messages are pre-processed (click/open tracking & DKIM signing) and added to the MTA’s ram-queue. This is a form of “back-pressure,” which is the MTA slowing down the message injection to the speed it can handle.

(Another common form of “back-pressure” is a fraction-of-a-second delay on accepting a message in a SMTP conversation, which has the effect of slowing down the overall speed of messages injection.)

Messages that were unable to be queued due to this limit will be returned with the not attempting because previous messages have taken too long error message (documented above). It is imperative that you re-submit messages returned with this (and the other) error messages documented above.

This error message is the MTA communicating that the batch size is too large.

Recommended batch size

We recommend that your batches be small enough to be accepted within 10 seconds.

There is little benefit to batches that take longer than 10 seconds to be accepted. (At this size the per-batch overhead is a very small portion of the equation, and larger batches do not gain much if anything.)

There is no benefit (and only a cost) to attempting batches larger than can be handled within 31 seconds.

How many messages can be processed within 10 seconds depends on many things, for example: how many other senders are concurrently using the HTTP API, how many messages are currently being injected through other injection methods, and the overall capacity of the system.

We recommend either (a) using a conservative static batch size or (b) dynamically adjusting the batch size based on injection speed.

To dynamically adjust the batch size:

  • for the last batch, divide how many messages were accepted by how long the batch took, to calculate the messages per second speed.
  • multiply this messages per second speed by 10 to get the size of the next batch
  • if larger than 500, reduce to 500

How to observe back-pressure

If message acceptance is currently being throttled due to back-pressure, the size of the Simplemh pre-processing queue in the hvmail_status status output will be above 90%.

Here is an example:

Simplemh pre-processing queue: 98% used (1960/2000)