Using DynamoDB Streams to Handle Future Scheduled/Delayed/Cron Jobs
An alternative to cron jobs, “background jobs”, or CloudWatch Events, for tasks that need to be done in the future, on a flexible schedule.
Many software systems need a mechanism to schedule jobs, and “cron” (or CloudWatch Events/Scheduled Jobs in the AWS cloud) or a “delayed job” system are common solutions. However, for a certain class of jobs, there’s an alternative that is quite nice, or certainly better than cron. The alternative is using DynamoDB records with a TTL (time to live) and DynamoDB Streams, processed by a Lambda function. The lambda is triggered when the record’s TTL expires, at which point you can process the record however you need. What kind of jobs is this applicable to? Any job that has a potentially changing, dynamic, or “x time from now” type of schedule, instead of a fixed cron style schedule (e.g. run every Tuesday at noon). It’s also great for individualized, or sparse situations, where triggering based on the individual record is much more efficient and cost effective than running a process across “all” records.
A couple example use cases:
- Sending reminder messages, where the time to send is X days before the event, or X days after something was due (and hasn’t been received).
- Changing device state/status when X amount of time has gone by without a reading.
There is one important caveat around precision of the TTL, which may not allow use — see Important Caveat…
