Chibisafe on GKE Autopilot — Lab Guide

Overview

Estimated time: 45–90 minutes

Chibisafe is a self-hosted file and image uploader with drag-and-drop uploads, albums, and a public API. This module deploys the chibisafe-server backend only (port 8000) as a single custom-built workload on GKE Autopilot, by default a StatefulSet backed by a 20Gi block PersistentVolumeClaim mounted at /data — a better fit than GCS Fuse for a single-writer SQLite application. This lab takes you through the full operational lifecycle of the Chibisafe on GKE Autopilot module: deploy it, access and verify it, run it day-to-day, observe it, diagnose common problems, and tear it down.

The lab focuses on operating the GKE module and the Google Cloud platform, not on Chibisafe product features. For the complete list of provisioned services and every configuration input (organised by group), see the Configuration Guide — this lab deliberately does not duplicate that detail so it stays accurate over time.

Objectives

By the end of this lab you will be able to:

Deploy the module from the RAD platform and locate the resources it provisions.
Connect to the GKE cluster and access the running workload.
Understand why the health probes target /api/health instead of /, and which probe variables actually control the deployed pod.
Perform day-2 operations — inspect the StatefulSet, manage the optional admin secret, and understand the storage/scaling constraints.
Observe the workload with Cloud Logging and Cloud Monitoring.
Diagnose and resolve the most common deployment and runtime issues.
Tear the deployment down cleanly.

Prerequisites

Services_GCP (provides the VPC, GKE Autopilot cluster, Artifact Registry, and shared service accounts this module depends on). You do not need to deploy this yourself first — the platform automatically detects whether it already exists in the target project and provisions it before this module if not (see Task 1).
A Google Cloud project with billing enabled.
gcloud CLI and kubectl installed; gcloud auth login and gcloud auth application-default login completed.
Project Owner (or equivalent) IAM on the project.
RAD platform access with permission to deploy modules into the project.

Set these shell variables once; every task below reuses them:

export PROJECT="<your-gcp-project-id>"
export REGION="us-central1"           # the region you deploy into

Task 1 — Deploy the module [Automated]

In the RAD platform, open Chibisafe (GKE), set project_id, and review the inputs. Configure only what you need — the Configuration Guide documents every input by group, with defaults. Review the estimated cost (if credits are enabled) and click Deploy, which opens the deployment status page with real-time logs.
The platform builds and pushes the custom chibisafe-server image (pinned to v6.5.5 unless you set a specific version), and deploys it into the GKE Autopilot cluster. Because stateful_pvc_enabled = true by default, the workload type auto-resolves to a StatefulSet with a 20Gi standard-rwo block PVC mounted at /data. A storage Cloud Storage bucket is always provisioned too, but is left unmounted while the PVC default is active (it only mounts at /data if you disable the PVC). If enable_api_key = true, an admin-password secret is also created in Secret Manager and delivered as a native Kubernetes Secret. enable_custom_domain = true by default, so a Kubernetes Gateway is provisioned even before you set a domain. No Cloud SQL instance is created — database_type is fixed to NONE. First deploys typically take 10–20 minutes (the custom image build dominates; there is no Cloud SQL instance to provision).

Connect to the cluster and discover the namespace with name-agnostic filters:

CLUSTER=$(gcloud container clusters list --project="$PROJECT" --format="value(name)" --limit=1)
gcloud container clusters get-credentials "$CLUSTER" --region="$REGION" --project="$PROJECT"

NS=$(kubectl get ns -o name | grep chibisafe | head -1 | cut -d/ -f2)
echo "Cluster: $CLUSTER   Namespace: $NS"
kubectl get all -n "$NS"

Task 2 — Access & verify [Manual]

Confirm the workload is running. Note this is a StatefulSet, not a Deployment:
```
kubectl get pods,svc,statefulset,pvc -n "$NS"
```
Health probe path — fixed, but know the layout. Chibisafe_GKE's startup_probe/liveness_probe variables (Group 10) default to /api/health, matching the CloudRun variant, because the chibisafe-server backend serves all routes under an /api prefix and has no root route (GET / returns 404). A fresh deploy on a current module version should reach Ready without any override. Two things to know:
- This was a latent bug — an earlier version of Chibisafe_GKE left startup_probe/liveness_probe at the inherited / default and pods sat in a perpetual not-Ready / crash-restart loop. It has since been fixed at the variable-default level; you should not need a workaround.
- health_check_config/startup_probe_config (also in Group 10) still default to / and always will — they are declared for foundation mirroring only and are never forwarded to App_GKE. Overriding them has no effect on the deployed probe; the variables that actually matter are startup_probe/liveness_probe.
If you still see the symptom below on a fresh deploy, treat it as a real regression rather than the historical known issue — check the pod's actual startup_probe/liveness_probe values with kubectl describe:
```
kubectl describe pod -n "$NS" <pod-name>
# Events will show: Readiness probe failed / Liveness probe failed:
# HTTP probe failed with statuscode: 404 (or similar non-200)
# -> compare against the pod spec's actual probe path:
kubectl get pod -n "$NS" <pod-name> -o jsonpath='{.spec.containers[0].livenessProbe.httpGet.path}'
```
Once the pod is Ready, find the workload's external address. With the default enable_custom_domain = true (Gateway) you need application_domains populated for the managed certificate to attach; alternatively switch service_type to LoadBalancer for a direct external IP:
```
kubectl get svc,gateway,httproute -n "$NS"
EXTERNAL_IP=$(kubectl get svc -n "$NS" \
  -o jsonpath='{.items[?(@.spec.type=="LoadBalancer")].status.loadBalancer.ingress[0].ip}')
echo "External IP: $EXTERNAL_IP"
```

Verify the corrected health endpoint directly (from within the cluster or once externally reachable):

kubectl exec -n "$NS" <pod-name> -- wget -qO- http://localhost:8000/api/health
# or, once externally reachable:
curl -s "http://${EXTERNAL_IP}/api/health"   # expect HTTP 200, {"status":"yes"}

This module deploys the backend API only — Chibisafe's separate SvelteKit front-end and Caddy reverse proxy are not part of this module. Administer the instance through the backend's REST API, or point a separately hosted Chibisafe front-end at the workload's address. By default (enable_api_key = false) the backend seeds its first-run administrator account with Chibisafe's well-known upstream default credential; redeploy with enable_api_key = true to have the module generate and inject a random ADMIN_PASSWORD instead.

Task 3 — Operate & keep it running (Day-2) [Manual]

Inspect the workload — StatefulSet, pods, and PVC:

kubectl get statefulset,pods,pvc -n "$NS"
kubectl describe statefulset -n "$NS"

Do not scale beyond one replica. min_instance_count = max_instance_count = 1 by default and this is a hard requirement: Chibisafe is a single-writer SQLite application, and even though each StatefulSet replica gets its own PVC, scaling risks split writers and inconsistent state.
Update the application version by changing the application_version input in the RAD platform and applying it via Update; the image is rebuilt with the pinned CHIBISAFE_VERSION build arg and the StatefulSet pod is replaced.

Manage the optional admin secret and inspect stored state — SQLite, uploads, and logs live directly on the pod's PVC (there is no database client to connect with):

kubectl get secrets -n "$NS"
gcloud secrets list --project="$PROJECT" --filter="name~chibisafe"
kubectl exec -n "$NS" <pod-name> -- ls -la /data/database /data/uploads /data/logs
kubectl exec -n "$NS" <pod-name> -- df -h /data

Note the unmounted storage bucket. A storage Cloud Storage bucket is always provisioned alongside the PVC, but stays unmounted while stateful_pvc_enabled = true (the default) — this is expected, not an error:
```
gcloud storage buckets list --project="$PROJECT" --filter="name~chibisafe"
```

Task 4 — Observe: Logging & Monitoring [Manual]

Logs — from kubectl or the Logs Explorer (note the StatefulSet naming, unlike a Deployment):
```
kubectl logs -n "$NS" statefulset/"$(kubectl get statefulset -n "$NS" -o jsonpath='{.items[0].metadata.name}')" --tail=50
```
Logs Explorer filter: resource.type="k8s_container" AND resource.labels.namespace_name="<namespace>".
Monitoring — open the GKE / Kubernetes dashboards and review pod CPU and memory utilisation, restart counts (watch for the health-probe issue above manifesting as repeated restarts), and PVC disk usage. Keep an eye on the regional SSD_TOTAL_GB quota if you are running other stateful modules alongside Chibisafe — standard-rwo is SSD-backed by default. An uptime check is available but disabled by default (uptime_check_config.enabled = false).

Task 5 — Troubleshoot & debug [Manual]

Durable techniques for the failure modes you are most likely to hit. These are platform-level diagnostics and do not change with Chibisafe releases.

Pod stuck not-Ready / crash-restart loop: startup_probe/liveness_probe default to /api/health (see Task 2), so this should not occur on a fresh deploy. If kubectl describe pod shows probe failures against path /, something has overridden startup_probe/liveness_probe back to the root route — check environment_variables/probe overrides in your deployment config; do not bother overriding health_check_config/ startup_probe_config, they are inert and never reach the pod spec.
```
kubectl describe pod -n "$NS" <pod>          # Events section shows probe failures
kubectl logs -n "$NS" <pod> --previous       # confirm the process actually started
kubectl get pod -n "$NS" <pod> -o jsonpath='{.spec.containers[0].livenessProbe.httpGet.path}'
```
PVC stuck Pending / Quota 'SSD_TOTAL_GB' exceeded: the default standard-rwo StorageClass is SSD-backed and draws a tight regional quota. Override stateful_pvc_storage_class = "standard" (HDD) — Chibisafe's SQLite/media write pattern does not need SSD IOPS. Reclaiming quota requires deleting the PVC/namespace; scaling to zero does not release it.
Gateway / managed certificate never attaches: confirm application_domains is populated — enable_custom_domain = true is on by default but the Gateway has no hostname to bind a certificate to until you set one.
Data appears to reset after a redeploy: confirm stateful_pvc_enabled is still true and stateful_pvc_mount_path is still /data — the entrypoint's relocation symlinks are hard-coded to that path.
Image pull errors: confirm the image exists in Artifact Registry and the node service account can pull it.

See the Configuration Guide's Configuration Pitfalls section for setting-specific gotchas (including the health-probe variable layout above, the SSD quota trade-off, and the inert enable_redis / container_port variables).

Task 6 — Tear down [Automated]

On the Deployments page, open the deployment and click the Trash icon (Delete). Delete runs terraform destroy and is irreversible (the deployment record is retained for history). If a deployment is stuck and the RAD platform can no longer manage it (for example after manual changes that conflict with the Terraform state), use Purge instead — it removes the deployment from RAD's records without destroying the cloud resources (it makes RAD forget the project). This removes everything the module created — the StatefulSet workload and namespace, its PVC, the Cloud Storage bucket, and the optional admin-password secret. There is no Cloud SQL database to remove — none was ever created. Resources owned by Services_GCP (the VPC, GKE cluster, Artifact Registry) are managed separately and are not removed here.

Summary

Task	Type	Outcome
1 — Deploy	Automated	Module builds the custom image and deploys a StatefulSet with a 20Gi block PVC at `/data`, an unmounted GCS bucket, and an optional admin secret — no Cloud SQL
2 — Access & verify	Manual	Connect to the cluster; confirm the `/api/health` startup/liveness probes (fixed default) let the pod reach Ready
3 — Operate	Manual	Inspect the StatefulSet/PVC, update version, manage the admin secret, inspect SQLite/uploads/logs on the PVC; never scale past 1 replica
4 — Observe	Manual	Query Cloud Logging; review pod/PVC metrics and optional uptime check
5 — Troubleshoot	Manual	Diagnose probe-path regressions, PVC/SSD quota, Gateway/certificate, and image-pull issues
6 — Tear down	Automated	Delete (Trash) removes the StatefulSet, PVC, bucket, and optional secret

By Dr Shiyghan Emmanuel Navti · Google Cloud certified: ACE · PCA · PCD · PDE · PSE · CDL

Overview​

Objectives​

Prerequisites​

Task 1 — Deploy the module [Automated]​

Task 2 — Access & verify [Manual]​

Task 3 — Operate & keep it running (Day-2) [Manual]​

Task 4 — Observe: Logging & Monitoring [Manual]​

Task 5 — Troubleshoot & debug [Manual]​

Task 6 — Tear down [Automated]​

Summary​