Library for sending email with Amazon's SES and hworker.
See README.
About
This is a library that facilitates sending email via AWS SES using the background processor hworker. In particular, it handles rate limiting (for sending rate currently, not daily quotas), as SES does not queue messages.
Rate limiting
Aside from sending emails in the background, the main thing that hworker-ses provides is rate limiting. The rate limiting is only on a per-process (most likely, per server) basis, as it is controlled via in-memory storage. This means that you should take into account how many workers you are likely to have when setting the messages-per-second rate on the workers.
For example, if your rate limit is 30 messages per second, and you expect to have at most 5 workers (you can organize them however you like, but a simple strategy is just to start a worker with each application process), then if you set the rate limit to be 6 messages per second, things will be fine.
Note that if you do run over, it won't mean that your messages actually don't get delivered. That error code will trigger a retry on the message, so eventually it will go out. But it's wasteful (as you can know in advance if a given message will actually be able to be sent), and if you really mess it up (ie, set an absurdly high limit), you potentially will have your API calls rate limited, which isn't good.
Usage
The program in the example directory is probably most of what you need to get started. In particular, you create an hworker (you need to give it a name for the queue - this should be shared by any servers that should be accessing the same queue and sending the same messages, though if you are sharing the redis server with other applications, this should be distinct). The third argument is the per-second rate limit for this server, and the last argument is the address that the messages are sent from.
Using what you get back, you can spawn workers and a monitor. Then you can queue as many messages as you want, and they will be sent out by the workers.
You need to start at least one worker thread and at least one monitor thread. You can have as many of these as you want, though generally speaking, having more than one monitor thread per server is probably overkill (for the system to work, you want at least one monitor running somewhere - so if a whole server crashes, you want there to be a monitor elsewhere). Having many workers makes things run faster. In particular, if you have a high sending limit, you will probably want several workers.
As specified in the hworker documentation, the semantics of this queue is at-least-once, so it's possible that messages can get sent multiple times in error conditions (like if an entire server crashes right after the message is sent, but before the fact that it was sent is acknowledged). But, provided that redis is still available and is reliable, messages that have been queued are guaranteed to get sent eventually (even in the case of servers crashing, etc). The only caveat to that is that for the message to get sent at least one worker and one monitor thread must be running (thoses don't need to be running all the time, but as long as they aren't running, messages might be delayed. Once they're started again, those delayed messages will go out).
Primary Libraries Used
- hworker
- amazonka-ses
Contributors
- Daniel Patterson [email protected].