Skip to main content

Kestra CloudRun Module — Configuration Guide

Kestra is an open-source data orchestration and scheduling platform (Apache 2.0 licence). It allows you to build, schedule, and monitor ETL/ELT pipelines, batch jobs, and workflow automation through a YAML-based flow definition and a rich plugin ecosystem. This module deploys Kestra on Google Cloud Run in standalone mode (server, worker, and scheduler in a single container) with a PostgreSQL 15 backend and GCS artifact storage.

Kestra CloudRun is a wrapper module built on top of App CloudRun. It delegates all GCP infrastructure provisioning to App CloudRun (Cloud Run service, Cloud SQL, networking, Secret Manager, GCS, CI/CD) and uses a Kestra Common sub-module to supply Kestra-specific application configuration, secret generation, and storage bucket definitions.

Note: Variables marked as platform-managed are set and maintained by the platform. You do not normally need to change them.


How This Guide Is Structured

This guide documents variables that are unique to Kestra_CloudRun or that have Kestra-specific defaults differing from the App_CloudRun base module. For variables with identical semantics, refer to the App_CloudRun Configuration Guide.

Key differences from App CloudRun defaults:

FeatureApp CloudRun defaultKestra CloudRun default
container_port80808080
cpu_limit"1000m""2000m"
memory_limit"512Mi""4Gi"
min_instance_count01
max_instance_count11
container_image_source"prebuilt""custom"
execution_environment"gen1""gen2"
enable_cloudsql_volumefalsetrue
enable_redisfalsefalse (fixed)
Health probe pathvaries/health

Platform-Managed Behaviours

The following behaviours are set automatically and cannot be overridden via user variables.

BehaviourDetail
Admin password auto-generatedA 24-character random KESTRA_BASICAUTH_PASSWORD is generated by Kestra Common and stored in Secret Manager as <resource_prefix>-admin-password. Injected via module_explicit_secret_values (bypasses Secret Manager read-after-write delays).
GCS bucket always provisionedKestra Common provisions a -kestra-storage GCS bucket. Its name is injected automatically as KESTRA_STORAGE_GCS_BUCKET.
PostgreSQL queue and repositoryKESTRA_QUEUE_TYPE=postgres and KESTRA_REPOSITORY_TYPE=postgres are always injected. Kestra uses PostgreSQL as both its execution queue and flow repository in standalone mode.
Redis is disabledenable_redis = false is hardcoded in main.tf. Kestra standalone mode uses PostgreSQL for queuing.
socat Unix-socket bridgeThe custom Dockerfile installs socat and replaces the entrypoint with entrypoint.sh. On Cloud Run, the Cloud SQL Auth Proxy creates a Unix socket; Java JDBC cannot connect via Unix sockets natively, so entrypoint.sh bridges the socket to TCP 127.0.0.1:5432 using socat. On GKE, the Auth Proxy sidecar already exposes TCP, so the bridge is skipped.
Service URL pre-computedThe predicted Cloud Run URL (https://<resource_prefix>-<project_number>.<region>.run.app) is passed to Kestra_Common and injected as a Kestra configuration variable before deployment.
Fixed environment variablesMICRONAUT_SERVER_PORT=8080, KESTRA_QUEUE_TYPE=postgres, KESTRA_REPOSITORY_TYPE=postgres, KESTRA_STORAGE_TYPE=gcs, KESTRA_BASICAUTH_ENABLED=true, KESTRA_BASICAUTH_USERNAME=admin, DATASOURCES_POSTGRES_DRIVERCLASSNAME, ENDPOINTS_ALL_PORT, and two Flyway baseline variables are always injected.
Scripts directoryResolved as abspath("${module.kestra_app.path}/scripts") — points to Kestra Common's bundled scripts directory.
execution_environment normalisedThe platform UI may send "EXECUTION_ENVIRONMENT_GEN2" — this is normalised to "gen2" before passing to App CloudRun.
backup_format normalisedLowercased before passing to App CloudRun (UI may send uppercase "SQL").

§1 · Module Metadata (Group 0)

VariableDefaultDescription
module_description(long description)Platform UI description. Do not modify unless customising.
module_documentation"https://docs.radmodules.dev/docs/modules/Kestra_CloudRun"External documentation URL.
module_dependency["Services GCP"]Platform modules that must be deployed first.
module_services["Cloud Run", "Cloud Run Jobs", "Cloud Build", "Artifact Registry", "Cloud Storage", "Cloud SQL (PostgreSQL 15)", "VPC Network", "Serverless VPC Access", "Secret Manager", "Cloud IAM", "Cloud Logging", "Cloud Monitoring", "Health Checks", "Webhooks", "Filestore (NFS)"]GCP services consumed.
credit_cost50Platform credits consumed on deployment.
require_credit_purchasesfalseEnforces credit balance check before deploy.
enable_purgetruePermits full resource deletion on destroy.
public_accesstrueControls platform catalogue visibility.
deployment_id""Auto-generated suffix. Set explicitly to pin resource names across Terraform runs.
resource_creator_identity"rad-module-creator@tec-rad-ui-2b65.iam.gserviceaccount.com"Service account used by Terraform.

§2 · Project & Identity (Group 1)

VariableDefaultDescription
project_id(required)GCP project ID. Must be 6–30 chars, lowercase letters/numbers/hyphens.
tenant_deployment_id"demo"Short suffix appended to resource names. 1–20 lowercase alphanumeric characters and hyphens.
support_users[]Emails granted IAM access and added to monitoring alert channels.
resource_labels{}Labels applied to all module-managed resources.

§3 · Application Identity (Group 2)

VariableDefaultDescription
application_name"kestra"Base name for the Cloud Run service, Artifact Registry repo, and Secret Manager secrets. Do not change after initial deployment — changing it forces resource recreation.
application_version"latest"Version tag applied to the container image. Increment to trigger a new build and rollout (e.g. "0.17.0").
display_name"Kestra Data Orchestration"Human-readable name in the platform UI and Cloud Run console.
description"Kestra Data Orchestration - ETL/ELT pipeline and workflow orchestration on Cloud Run"Brief description of the deployment.
deploy_applicationtrueSet false to provision supporting infrastructure without deploying the container.

§4 · Runtime & Scaling (Group 3)

VariableDefaultOptions / FormatDescription
cpu_limit"2000m"CPU quantityCPU limit per Cloud Run container instance. Kestra (Java JVM) benefits from at least 2 vCPUs for concurrent pipeline execution.
memory_limit"4Gi"Memory quantityMemory limit per container. Kestra requires a minimum of 2Gi; 4Gi is recommended for typical workloads.
container_port8080IntegerPort Kestra listens on. Must match MICRONAUT_SERVER_PORT.
min_instance_count1Integer ≥ 0Kestra (Java JVM) has a slow cold start — keep at 1 for production.
max_instance_count1Integer ≥ 1Kestra standalone mode is single-instance. Set to 1 to avoid conflicting state.
timeout_seconds300Integer 0–3600Maximum request timeout for the Cloud Run service.
execution_environment"gen2"gen1, gen2"gen2" is required for GCS Fuse volume mounts. Also accepted as "EXECUTION_ENVIRONMENT_GEN2" (normalised automatically).
ingress_settings"all"all, internal, internal-and-cloud-load-balancingControls which traffic sources can reach the Cloud Run service.
vpc_egress_setting"PRIVATE_RANGES_ONLY"ALL_TRAFFIC, PRIVATE_RANGES_ONLYRoutes VPC traffic. Use "ALL_TRAFFIC" for strict egress control.
container_protocol"http1"http1, h2cHTTP protocol version for the Cloud Run backend.
enable_image_mirroringtrueboolMirrors the container image into Artifact Registry before deployment.
enable_cloudsql_volumetrueboolInjects Cloud SQL Auth Proxy sidecar. Required for the Unix socket bridge.
cloudsql_volume_mount_path"/cloudsql"Path stringFilesystem path for the Cloud SQL Auth Proxy Unix socket.
service_annotations{}map(string)Custom annotations applied to the Cloud Run service.
service_labels{}map(string)Custom labels applied to the Cloud Run service.
traffic_split[]list(object)Traffic allocation across Cloud Run revisions. All entries must sum to 100. Empty sends all traffic to the latest revision.
max_revisions_to_retain7Integer 0–100Maximum Cloud Run revisions to retain after each deployment.
max_images_to_retain7Integer 0–100Maximum container images to retain in Artifact Registry.
delete_untagged_imagestrueboolAutomatically deletes untagged images from Artifact Registry.
image_retention_days30Integer 0–3650Days after which container images are eligible for deletion.

§5 · Access & Networking

Identity-Aware Proxy (Group 19)

VariableDefaultDescription
enable_iapfalseEnables IAP authentication for the Kestra UI. Provides Google identity authentication in addition to Kestra's built-in basic auth.
iap_authorized_users[]User allowlist. Format: "user:email@example.com".
iap_authorized_groups[]Group allowlist. Format: "group:name@example.com".

Cloud Armor & CDN (Group 9)

VariableDefaultDescription
enable_cloud_armorfalseEnables Cloud Armor WAF + Global HTTPS Load Balancer.
admin_ip_ranges[]CIDR ranges for administrative access bypass.
application_domains[]Custom domain names. Only used when enable_cloud_armor = true.
enable_cdnfalseEnables Cloud CDN. Only active when enable_cloud_armor = true.

VPC Service Controls (Group 21)

VariableDefaultDescription
enable_vpc_scfalseEnforces VPC Service Controls perimeters.
vpc_cidr_ranges[]VPC subnet CIDR ranges. Auto-discovered from VPC when empty.
vpc_sc_dry_runtrueWhen true, violations are logged but not blocked.
organization_id""GCP Organization ID. Auto-discovered when empty.
enable_audit_loggingfalseEnables detailed Cloud Audit Logs.

§6 · Environment Variables & Secrets (Group 4)

VariableDefaultDescription
environment_variables{}Additional static env vars for Kestra. Use for plugin configurations and custom Micronaut properties. Do not set platform-managed vars.
secret_environment_variables{}Map of env var name to Secret Manager secret name. Example: { KESTRA_ENCRYPTION_SECRET = "kestra-enc-key" }.
secret_propagation_delay30Seconds to wait after secret creation. Valid range: 0–300.
secret_rotation_period"2592000s"Rotation notification period. Format: "<N>s".
enable_auto_password_rotationfalseDeploys automated database password rotation.
rotation_propagation_delay_sec90Seconds to wait after rotation before Cloud Run restarts.

Do not set in environment_variables: MICRONAUT_SERVER_PORT, KESTRA_QUEUE_TYPE, KESTRA_REPOSITORY_TYPE, KESTRA_STORAGE_TYPE, KESTRA_STORAGE_GCS_BUCKET, KESTRA_BASICAUTH_ENABLED, KESTRA_BASICAUTH_USERNAME, DATASOURCES_POSTGRES_URL, DATASOURCES_POSTGRES_USERNAME, DATASOURCES_POSTGRES_PASSWORD, or KESTRA_BASICAUTH_PASSWORD. These are injected automatically by Kestra Common and entrypoint.sh.


§7 · Database Backend (Group 11)

Kestra requires PostgreSQL for both its execution queue and flow repository.

VariableDefaultDescription
db_name"kestra"PostgreSQL database name. Do not change after initial deployment.
db_user"kestra"PostgreSQL application user.
database_password_length32Auto-generated password length. Valid range: 16–64.

§8 · Storage (Groups 8, 10)

NFS (Group 8)

VariableDefaultDescription
enable_nfsfalseProvisions a Cloud Filestore NFS instance and mounts it into the Cloud Run service.
nfs_mount_path"/mnt/nfs"Container path for the NFS volume.
nfs_instance_name""Existing NFS GCE VM name. Auto-discovered when empty.
nfs_instance_base_name"app-nfs"Base name for the inline NFS GCE VM.

Cloud Storage & GCS Fuse (Group 10)

Kestra Common always provisions a -kestra-storage bucket. Additional buckets can be added via storage_buckets.

VariableDefaultDescription
create_cloud_storagetrueControls whether the module provisions the buckets in storage_buckets.
storage_buckets[]Additional GCS buckets to provision.
gcs_volumes[]GCS buckets mounted as filesystem volumes via GCS Fuse. Requires execution_environment = "gen2".
manage_storage_kms_iamfalseCreates CMEK KMS keys and enables CMEK encryption on storage buckets.
enable_artifact_registry_cmekfalseEnables CMEK encryption on container images in Artifact Registry.

§9 · Backup & Maintenance (Group 16)

VariableDefaultDescription
backup_schedule"0 2 * * *"Cron schedule (UTC) for automated database backups.
backup_retention_days7Days to retain backup files in GCS.
enable_backup_importfalseTriggers a one-time database import job during deployment.
backup_source"gcs"Source: "gcs" or "gdrive".
backup_uri""Full URI for the backup. For GCS: gs://bucket/path/backup.sql. For Drive: file ID.
backup_format"sql"Format of the backup file. Normalised to lowercase automatically.

§10 · CI/CD & GitHub Integration (Group 11)

VariableDefaultDescription
enable_cicd_triggerfalseCreates a Cloud Build trigger for automatic builds.
github_repository_url""Full HTTPS URL of the GitHub repository.
github_token""GitHub PAT. Sensitive.
github_app_installation_id""Cloud Build GitHub App installation ID.
cicd_trigger_config{ branch_pattern = "^main$" }Branch filter, included/ignored paths, substitutions.
enable_cloud_deployfalseSwitches to a Cloud Deploy pipeline. Requires enable_cicd_trigger = true.
cloud_deploy_stages[dev, staging, prod(approval)]Ordered promotion stages.
enable_binary_authorizationfalseEnforces Binary Authorization attestation policy.

§11 · Custom SQL (Group 17)

VariableDefaultDescription
enable_custom_sql_scriptsfalseRuns custom SQL scripts from GCS against the application database.
custom_sql_scripts_bucket""GCS bucket name containing SQL scripts.
custom_sql_scripts_path""Path prefix within the GCS bucket. Scripts executed in alphabetical order.
custom_sql_scripts_use_rootfalseExecute scripts as the root database user.

§12 · Jobs & Scheduled Tasks (Group 12)

The default db-init job is supplied automatically by Kestra Common (using postgres:15-alpine) when initialization_jobs is empty.

VariableDefaultDescription
initialization_jobs[]Cloud Run Jobs executed once during deployment. Common uses: schema verification, credential seeding, flow imports. When non-empty, replaces the default db-init job.
cron_jobs[]Recurring Cloud Run Jobs triggered by Cloud Scheduler.

§13 · Observability & Health (Group 13)

Kestra's health endpoint is /health. Kestra (Java JVM) has a slow startup — the default startup probe allows up to ~14 minutes (initial_delay=30 + period=20 × failure_threshold=40).

Kestra CloudRun exposes four probe variables across two parallel paths:

VariableRouted toConfigures
startup_probeKestra Commonconfig.startup_probeApplication container startup probe
liveness_probeKestra Commonconfig.liveness_probeApplication container liveness probe
startup_probe_configApp CloudRun directlyApp CloudRun infrastructure startup probe
health_check_configApp CloudRun directlyApp CloudRun infrastructure liveness probe
VariableDefaultDescription
startup_probe{ enabled=true, type="HTTP", path="/health", initial_delay_seconds=30, timeout_seconds=5, period_seconds=20, failure_threshold=40 }Application startup probe.
liveness_probe{ enabled=true, type="HTTP", path="/health", initial_delay_seconds=180, timeout_seconds=5, period_seconds=30, failure_threshold=5 }Application liveness probe.
startup_probe_config{ enabled=true, type="TCP", initial_delay_seconds=0, timeout_seconds=300, period_seconds=300, failure_threshold=1 }App CloudRun startup probe (TCP, no path).
health_check_config{ enabled=true, type="HTTP", path="/health" }App CloudRun liveness probe.
uptime_check_config{ enabled=true, path="/health" }Cloud Monitoring uptime check from multiple global locations.
alert_policies[]Custom metric alert policies. Each requires name, metric_type, comparison, threshold_value, duration_seconds.

§14 · Outputs

Outputs are proxied from App CloudRun:

OutputDescriptionSensitive
service_nameCloud Run service name
service_urlPublic URL of the Cloud Run service
service_locationGCP region of the Cloud Run service
stage_servicesStage-specific Cloud Run service details
database_instance_nameCloud SQL instance name
database_nameApplication database name
database_userApplication database user
database_password_secretSecret Manager secret name for database password
database_hostDatabase hostyes
database_portDatabase port
storage_bucketsCreated GCS buckets
network_nameVPC network name
network_existsWhether the VPC network exists
regionsAvailable regions in the VPC
nfs_server_ipNFS server internal IPyes
nfs_mount_pathNFS mount path in containers
nfs_share_pathNFS share path on server
container_imageContainer image used for the service
container_registryArtifact Registry repository name
monitoring_enabledWhether monitoring is configured
monitoring_notification_channelsMonitoring notification channel names
deployment_idUnique deployment identifier
tenant_idTenant identifier
resource_prefixResource naming prefix
project_idGCP project ID
project_numberGCP project number
initialization_jobsCreated initialization job names
nfs_setup_jobNFS setup job name
deployment_summarySummary of the deployment
cicd_enabledWhether CI/CD pipeline is enabled
github_repository_urlGitHub repository URL for CI/CD
github_repository_ownerGitHub repository owner/organization
github_repository_nameGitHub repository name
artifact_registry_repositoryArtifact Registry repository
cloudbuild_trigger_nameCloud Build trigger name
cloudbuild_trigger_idCloud Build trigger ID
cicd_configurationComplete CI/CD configuration

Configuration Examples

Basic Deployment

project_id           = "my-project-123"
tenant_deployment_id = "demo"

Production Deployment

project_id           = "my-project-123"
tenant_deployment_id = "prod"

application_name = "kestra"
display_name = "Kestra Orchestration"
application_version = "0.17.0"

# Sizing (Kestra needs ≥ 2 vCPU, ≥ 2Gi RAM)
cpu_limit = "4000m"
memory_limit = "8Gi"

# Keep warm — slow JVM startup
min_instance_count = 1
max_instance_count = 1

# Database
db_name = "kestra"
db_user = "kestra"
database_password_length = 32

# Security
enable_iap = true
iap_authorized_users = ["user:alice@example.com"]

# Backup
backup_schedule = "0 2 * * *"
backup_retention_days = 14

# Observability
uptime_check_config = {
enabled = true
path = "/health"
check_interval = "60s"
timeout = "10s"
}

# CI/CD
enable_cicd_trigger = true
github_repository_url = "https://github.com/my-org/kestra-flows"
github_token = "ghp_***"

Deployment with GCS Fuse Volume

project_id           = "my-project-123"
tenant_deployment_id = "flows"

execution_environment = "gen2"

gcs_volumes = [
{
name = "kestra-flows"
mount_path = "/app/flows"
readonly = false
mount_options = ["implicit-dirs", "stat-cache-ttl=60s"]
}
]

Configuration Pitfalls & Sensible Defaults

Risk levels: Critical (data loss, full outage, security breach) — High (service unavailable or significant degradation) — Medium (degraded function or increased cost) — Low (minor impact).

VariableSensible DefaultRiskConsequence of Incorrect Value
KESTRA_BASICAUTH_PASSWORD (auto-generated secret)Auto-generated and stored in Secret ManagerCriticalThe admin password is the sole access control gate for the Kestra UI. There is no secondary authentication — losing this password requires a direct database update to reset it.
KESTRA_BASICAUTH_USERNAME (injected as "admin")"admin"HighThis is a well-known default. Override via environment_variables = { KESTRA_BASICAUTH_USERNAME = "your-admin-name" } before first deploy to reduce credential-stuffing risk.
KESTRA_BASICAUTH_ENABLED (injected as "true")"true"CriticalDisabling basic auth via environment_variables exposes the Kestra UI and all API endpoints without authentication. Only disable if an upstream authentication proxy (IAP, Cloud Armor) is in place.
application_name"kestra"CriticalImmutable after first deploy. Changing it renames all GCP resources and causes full recreation with data loss.
db_name"kestra"CriticalImmutable after first deploy. Changing it causes Kestra to connect to a new empty database, losing all flow definitions, execution history, triggers, and namespaces.
KESTRA_QUEUE_TYPE (injected as "postgres")"postgres"HighKestra uses PostgreSQL as both its queue and repository backend in this deployment. Changing via environment_variables to an unsupported backend type causes startup failures — no other queue backend is provisioned.
KESTRA_REPOSITORY_TYPE (injected as "postgres")"postgres"HighSame as KESTRA_QUEUE_TYPE — only the PostgreSQL repository backend is provisioned. Changing this causes Kestra to fail to find flow definitions and execution records.
KESTRA_STORAGE_TYPE (injected as "gcs")"gcs"HighKestra stores flow inputs, outputs, and internal storage in GCS. Changing this to "local" causes all storage operations to write to ephemeral container filesystem, losing all execution artifacts on restart.
KESTRA_STORAGE_GCS_BUCKET (auto-set from resource prefix)"<prefix>-kestra-storage"HighThe GCS bucket is created by the module. Overriding with a non-existent bucket name via environment_variables causes Kestra to fail all flow executions that produce storage outputs.
memory_limit"4Gi"HighKestra loads all flow definitions and active execution contexts into the JVM heap. Values below 2Gi cause frequent JVM GC pauses and OutOfMemoryErrors under concurrent execution load. The default 4 Gi is the recommended minimum for production.
cpu_limit"2000m"MediumKestra's task runners are CPU-intensive. Values below 1000m cause severe execution throttling, especially for flows with parallel task groups.
min_instance_count1HighKestra must remain running to process scheduled triggers and poll for new executions. Scale-to-zero (0) means scheduled flows miss their trigger window during cold starts. Always keep at least 1 instance warm for scheduler-dependent workloads.
max_instance_count1 (Kestra default — single-instance only)HighKestra in standalone mode uses PostgreSQL-based queue locking. Running multiple instances without proper Kestra Enterprise configuration causes task double-assignment. Keep max_instance_count = 1 for the Community Edition.
FLYWAY_DATASOURCES_POSTGRES_BASELINE_ON_MIGRATE (injected as "true")"true"HighRequired to allow Flyway to baseline against an already-initialized PostgreSQL database (Cloud SQL installs extensions in the public schema by default). Removing this override via environment_variables causes all 52 Kestra migrations to fail on first run with "non-empty schema" errors.
enable_nfstrueMediumKestra stores flow scripts and execution artifacts via GCS storage. NFS is used for binary data mode. Without NFS, local file operations in flow tasks fail.
ingress_settings"all"MediumSetting to "internal" blocks all external webhook triggers and flow API calls from outside the VPC. Use "internal-and-cloud-load-balancing" with Cloud Armor for controlled public access.
enable_iapfalseHighEnabling IAP without valid iap_authorized_users or groups blocks all access to the Kestra UI. The basic auth login page is unreachable when IAP returns 403.
backup_schedule"0 2 * * *" (daily at 02:00)MediumKestra's entire state (flows, triggers, namespaces, execution history) lives in PostgreSQL. Ensure the backup schedule aligns with your RPO.
enable_cloud_armorfalseMediumThe Kestra API and UI are publicly accessible by default. Enable Cloud Armor with admin_ip_ranges to restrict access in production.
execution_environment"gen2"HighNFS mounts require gen2. Changing to gen1 causes NFS mount failures and container startup errors.
secret_propagation_delay30 secondsLowToo short a delay may cause the Kestra JVM to start before the KESTRA_BASICAUTH_PASSWORD secret has propagated from Secret Manager, causing the initial login to use a stale password.
enable_vpc_scfalseMediumRequires organization_id. If empty, VPC-SC is silently skipped.
vpc_sc_dry_runtrueLowLeaving dry-run enabled means VPC-SC rules are logged but not enforced.
ENDPOINTS_ALL_PORT (injected as "8080")"8080"HighExposes the Micronaut management endpoints (including /health) on the main server port for Cloud Run startup and liveness probes. Overriding this port via environment_variables breaks health checks, causing continuous container restarts.

Destroying Resources

Known Deletion Issue: Serverless IPv4 Address Release

When destroying a Cloud Run deployment, you may encounter an error similar to:

Error: Error waiting for Subnetwork to be deleted: The following serverless IPv4 address(es) on subnet ... are still in use.

Cause: GCP holds serverless IPv4 addresses on the VPC subnet asynchronously after a Cloud Run service is deleted. These addresses are released by GCP approximately 20–30 minutes after the Cloud Run service is removed. Terraform/OpenTofu cannot complete the subnet or VPC deletion until they are fully released.

Resolution: Wait 20–30 minutes after the initial destroy attempt, then re-run the destroy command:

tofu destroy

The second run will succeed once GCP has released the reserved addresses.