CloudCron

A simple distributed, cost effective, backwards-compatible, cloud friendly cron for the masses

Introduction

If you google around looking for a "cloud friendly cron" that runs on AWS: you're out of luck. You're probablly just trying to solve one thing: I have a crontab that I want to run on AWS

Solutions you find around tend to be in the following areas:

• You need to build and maintain a complex, distributed, minimum of three nodes cluster

• Make it a Lambda function (since Lambdas can be scheduled)

• Put the crons code in your app server and call it via HTTP

Lots of times, users just put their cron on one "cron instance"

The CloudCron solution

• Backwards-compatible

Give it a cron file, and it will run the same command you already have, on the same server if you want. No need for reprogramming your jobs

• Cost-Effective

With minimal infrastructure (just a worker node)

• Scalable/Parallelizable

Have more crons than a machine can handle? You can scale your worker nodes at need

• Red/Black deployment friendly

You can deploy a new worker node with new code or patches without downtime

• No time limits for the cron execution

Lambda based solutions have a maximum running time of 5 minutes

• CD Pipeline friendly

You get the opportunity to version control your crontab, and deploy it as a part of your CD

How does it work?

CloudCron is basically split into two halves: a "cron compiler", that transforms a cronfile into CloudWatch events

Your crontab file gets transformed into a bunch of CloudWatch Events that get pushed to CloudFormation so they can be managed as a whole

These CloudWatch events inject a message for each ocurrance of a cron event to an SQS queue, which is getting polled by a worker process. This

worker process runs in the same machine (or machines) where your old cron could execute it's jobs.

Installation

You can install CloudCron with any perl package manager. We recommend to use carton so you don't install dependencies in your system (carton is usually installable via your SOs package manager):

mkdir mycron
cd mycron
echo 'requires "CloudCron";' >> cpanfile
echo 'requires "CloudCron::Worker";' >> cpanfile
carton install
carton exec $SHELL -l

CloudCron is split in two parts:

• the management interface package: CloudCron that enables you to create the necessary infrastructure
• the worker package: CloudCron::Worker the process that runs on your cloud nodes

Get me started

cloudcron init --name MyCronQueue --region eu-west-1

This command deploys the SQS queue and informs you how to start a worker. We can have as many queues as we want (for different groups of crons, for example)

You can start a worker the output of the last command, but first create a log configuration file:

printf "log4perl.appender.Screen = Log::Log4perl::Appender::Screen\nlog4perl.appender.Screen.layout = Log::Log4perl::Layout::PatternLayout\nlog4perl.appender.Screen.layout.ConversionPattern = [%%d][CloudCron] %%p %%m%%n\n# Catch all errors\nlog4perl.logger = INFO, Screen\n" > log.conf

And now launch the worker in background, pointing to the log config you just created

cloudcron-worker --queue_url https://sqs.eu-west-1.amazonaws.com/012345678901/MyCronQueue-CloudCronQueue-LPNF3N07WF68 --region eu-west-1 --log_conf log.conf &

The worker is now idle, waiting for it's first jobs, so we need to create them:

cloudcron deploy --name MyCronFile --destination_queue MyCronQueue --region eu-west-1 path_to_crontab_file

The name of the destination queue is the name we gave to cloudcron init

Once the deploy command finishes, we wait for the worker to start receiving the messages, and executing your jobs!

If you modify your crontab file, just redeploy. cloudcron will detect that you already deployed this cron, and will update the events:

cloudcron deploy --name MyCronFile --destination_queue MyCronQueue --region eu-west-1 path_to_crontab_file

Once you're ready, you can delete queues, and crons with the cloudcron remove command

Nitty gritty details

• Each event launched by CloudWatch events is delivered to the crons' queue. The worker picks up the message only once, even if the job fails to execute (like in cron) it is not redelivered to the queue for retry. It's supposed that it will be executed a next time

• Each cron queue has a dead letter queue. Here you could find messages due to severe malfunction: messages that were delivered to a worker, but never got acknowledged (deleted) in time (notice that workers immediately delete a message after recieving it, even before attempting to execute the job, so not deleting a message should not happen).

• The worker executes with limited privileges. You control the user which the worker runs with. The worker shouldn't be run as root (unless you have good reasons to)

Known Limitations

• The system depends on SQS, so it's possible for

  • events to be processed out of order
  • events to be processed more than one time by two concurrent workers.

  SQS recently got FIFO and exactly once delivery. It's on the TODO list to take advantage of that capability to provide a more cron-like behaviour

• There is no reentrance contol (like cron, you should take care of that in your job) (or you can contribute a patch to avoid reentrance)

• It's recommended to centralize the cloudcron logs with something like CloudWatch Logs if you're executing on autoscaling groups, for example.

• If you don't have workers polling a queue, events still get delivered to it and accumulate, so it's possible that you accumulate invocations of lots of events if you don't have a worker running.

  SQS recently got a deduplication facility which we can use to try to avoid message accumulation

• If the system that hosts the worker fails before an event has fully executed, the event will not be retried

TOPOLOGIES

With cloudcron you can generate a lot of differente topologies to suit your needs:

Single crontab

crontab ---> Queue <--- Worker node

• create a queue with cloudcron init
• deploy your crontab file with cloudcron deploy
• start a cloudcron-worker on the machine that has to do the work

Lots of jobs (multinode)

If one machine is not enough to handle the load of all your crons, you can just add more cloudcron-workers polling the same queue

                                      |---- Worker node 1
crontab (lots of jobs)  ---> Queue <--|---- Worker node 2
                                      |---- Worker node 3

• create a queue with cloudcron init
• deploy your crontab file with cloudcron deploy
• start one cloudcron-worker for each machine that has to do the work

Lots of jobs (autoscaling)

If the load on your worker nodes varies enough, you might want to autoscale your worker node fleet applying autoscaling to the worker node pool autoscaling group

                                     |-A--   
crontab (lots of jobs) ---> Queue <--|-S-- Worker node N
                      |              |-G--
                      |                |
                      +--- CloudWatch--+

Caution should be taken: when autoscaling shuts down a worker instance, it will kill the processes actually executing. There is no facility in cloudcron to let your jobs finish (contributions welcome :))

Manage lots of jobs

You can deploy independant crontab files to the same queue, as long as the worker polling the queue is able to execute the commands in your crontab.

crontab for ETLs -----+
                      |
crontab for cleanup --+---> Queue <--- Worker node
                      |
crontab for X --------+

• create a queue with cloudcron init
• deploy each crontab file with cloudcron deploy to the same queue
• start a cloudcron-worker on the machine that has to do the work

Crons running with different users

You can deploy an independant worker for each user. The two workers can run on the same instance

crontab for user1 ----> Queue1 <---- Worker running with user1

crontab for user2 ----> Queue2 <---- Worker running with user2

• create two queues with cloudcron init
• deploy each crontab to it's queue with cloudcron deploy
• start a worker under user1 pointing to the queue that it's cron points to
• start the other worker under user2 pointing to the queue that it's cron points to

Flexibility (custom topologies)

You can combine these scearios as you want to adapt them to your needs. Mix, match, and report your topologies back so we can document them!

Deploying cloudcron-worker

In the examples directory there is a sample of how to run a cloudcron-worker from an upstart job

Bugs and Source

The source code is located here: https://github.com/capside/CloudCron

Bugs and Issues can be reported here: https://github.com/capside/CloudCron/issues

Authors

Pau Cervera, Eduard Badillo, Jose Luis Martinez

License and Copyright

Copyright (c) 2017 by CAPSiDE SL

This code is distributed under the Apache 2 License. The full text of the license can be found in the LICENSE file included with this module.