Container-based Serverless Scheduled Jobs on Google Compute Engine
Most of my compute workloads today are on Cloud Run, a serverless for containers platform. But some workloads like scheduled jobs don’t fit the service-oriented model of Cloud Run. There are many places I can run those workloads but I’d like to keep the serverless “pay for what you use” model and still use containers as my packaging format. I could use Kubernetes for these and use Cloud Run for Anthos to run everything in one place but I wanted something more bare-bones. I created a way to hook up Cloud Scheduler so that it starts scheduled jobs from containers on Google Compute Engine. Here is a video walkthrough for how to set it up and use it:
How it works
Cloud Scheduler can make an HTTP call as the trigger action. To create an instance based on that HTTP call there needs to be a service that knows how to handle the request and do the create. I’ve created a little HTTP service that does exactly that: webhook-runner. The service handles a POST
request to /
with a JSON body like:
{
"project": "foo",
"zone": "us-central1-a",
"machineType": "n1-standard-1",
"containerImage": "hello-world"
}
That post will run the hello-world
container in Google Cloud project foo
, in the us-central1-a
zone, on a n1-standard-1
machine. If a stopped instance for that container already exists, it will just be started, otherwise a new GCE instance will be created. Once the container process has stopped, the instance is shutdown. The webhook-runner
service can be run on Cloud Run so you only pay for what you use there too (which should be nothing since typical usage would be below the free tier). Here is what the architecture looks like:
You definitely do NOT want your webhook-runner
service to be accessible by anyone so luckily it’s pretty easy to only allow it to be invoked by a service account. Cloud Scheduler can use that service account when it makes the HTTP call. Also, the service should have limited access to what it can do, i.e. just create / start GCE instances. So another service account can be used when the webhook-runner
service runs (and calls gcloud
). You can setup these manually following a script or alternatively, just click this button and the webhook-runner
service will be deployed and service accounts setup:
Hopefully this is useful and if you need anything, file an issue on the webhook-runner project.