DataOps

Orchestrate ETL pipelines and control your cloud resource inventory across AWS, Azure, and GCP, from the dashboard.

DataOps brings ETL pipeline orchestration and broader cloud resource control into KubeWatch, so you can trigger a Glue job or shut down an idle database without leaving the dashboard. ETL pipeline orchestration covers AWS Glue and Step Functions, Apache Airflow, Azure Data Factory, and GCP Dataflow and Cloud Composer. Broader cloud resource control now spans AWS, Azure, and GCP as well. There's no plan-tier gate: it's available on every plan. Any org member can view it, but triggering a run, connecting a tool, and acting on a resource all require the admin role.

What it does

ETL pipeline orchestration: trigger, cancel, and monitor runs across AWS Glue and Step Functions, Apache Airflow, Azure Data Factory, and GCP Dataflow and Cloud Composer from the DataOps page, with run history (status, duration, failures) kept up to date automatically.
Broader cloud resource control: beyond the compute instances already shown on Infrastructure Estimates, KubeWatch now inventories databases, storage, and serverless functions across all three connected cloud accounts, and lets an admin start, stop, resize, or delete them from Infrastructure > Resources:
- AWS: RDS databases, S3 buckets, and Lambda functions.
- Azure: Azure Database for PostgreSQL (Flexible Server), Storage Accounts, and Function Apps.
- GCP: Cloud SQL instances, Cloud Storage buckets, and Cloud Functions.

Connecting a data source

Each ETL tool has its own connection, set up from the DataOps page:

AWS Glue / AWS Step Functions: reuses the AWS account already connected on Cloud Costs. No new credential to set up: if an AWS connection already exists for Cloud Costs, the DataOps page and Infrastructure > Resources pick it up automatically.
Apache Airflow: connect a self-hosted Airflow instance directly, with its host, port, username, and password. DataOps talks to Airflow's REST API over HTTP Basic Auth.
Azure Data Factory: reuses the Azure connection already set up on Cloud Costs, plus the Data Factory's factory name and resource group.
GCP Dataflow: reuses the GCP connection already set up on Cloud Costs, plus a region and a template GCS path, since Dataflow jobs launch from a template rather than arbitrary code.
GCP Cloud Composer: reuses the GCP connection already set up on Cloud Costs, plus the environment's Airflow web server URL. Find this by running gcloud composer environments describe <env> --location <region> and reading config.airflowUri, or from the GCP Console. Most Composer environments put that web server behind Identity-Aware Proxy (IAP), which needs an extra, optional IAP OAuth client ID field on the connection. Without it, triggering, polling, and cancelling runs against an IAP-fronted environment will fail with an auth error. Find the client ID under the IAP-secured resource's OAuth client in the GCP Console, or via gcloud iap oauth-brands list.

If you haven't connected an AWS, Azure, or GCP account yet, do that first from Cloud Costs; see Connecting a provider for the credential each one needs.

Triggering and monitoring an ETL run

Open the DataOps page in the sidebar to see every job, run, or execution visible across your connected tools: AWS Glue jobs and Step Functions state machines, Airflow DAGs, Azure Data Factory pipelines, and GCP Dataflow jobs and Composer DAGs. From there you can:

Trigger a job run or a state machine execution on demand.
Cancel a run that's currently in progress.
Review run history: status, duration, and failure details for every past run.

A background poller refreshes run status roughly every 20 seconds, so a run that finishes outside the dashboard (started from the AWS console, an Airflow DAG schedule, Azure Data Factory, GCP Dataflow, or Composer) still shows up here without a manual refresh. A run stuck in progress for a long time is flagged so you know it's taking longer than expected, but polling continues underneath: long-running batch jobs (Glue, Dataflow) routinely run for many hours and this doesn't cut them off.

Cancel is a real stop for AWS Glue, Step Functions, Azure Data Factory, and GCP Dataflow. For Apache Airflow and GCP Cloud Composer, neither of which exposes a real "stop a running DAG" API, Cancel only marks the run as failed in KubeWatch. The underlying DAG keeps executing in Airflow/Composer itself. The dashboard flags this on the Cancel button for those two tools.

Viewing and controlling cloud resources

The Infrastructure > Resources page lists resources across every connected AWS, Azure, and GCP account, alongside the compute instances already covered by Infrastructure Estimates. From this page an admin can:

Start or stop a resource that supports it (e.g. an RDS instance, an Azure Postgres Flexible Server, a Cloud SQL instance, or a Lambda/Function App).
Resize a resource (e.g. changing an RDS instance class, an Azure Postgres sku, or a Cloud SQL tier).
Delete a resource.

Not every resource type supports every action: a bare storage bucket (S3, a Storage Account, or GCS) only supports delete, and Cloud Functions is delete-only too, since GCP's Cloud Functions API has no start/stop primitive at all. Azure Function Apps have a real start/stop; Lambda has no such primitive either, so "stop" instead sets its reserved concurrency to zero (blocking invocations) and "start" removes that throttle. Neither Lambda nor Function Apps support resize.

Any destructive action (stop, resize, or delete) asks for a simple confirmation first. There's no multi-step or multi-person approval workflow here, unlike Infrastructure Automation's plan-then-approve-then-apply flow: confirm once, and the action runs.

Access control

Both the DataOps page and Infrastructure > Resources are visible to any authenticated org member, viewers included: anyone can browse connections, ETL run history, and the resource inventory. Acting on any of it (connecting or deleting a tool connection, triggering or cancelling a run, syncing resources, or starting/stopping/resizing/deleting one) requires the admin role; a non-admin's attempt is rejected by the API. This is a role check, not a plan check: DataOps has no plan-tier restriction and is available on Free, Pro, and Enterprise alike.

Most of these admin-only actions aren't actually hidden from a `viewer` in the UI today: Add connection, Trigger, Cancel, Start, Stop, Resize, and Delete all render regardless of role, and a non-admin only finds out an action is admin-only when the API rejects it. The one exception is the Resources page's "Sync now" button, which is hidden from non-admins.

What's next

ETL pipeline orchestration now covers all 4 tool families: AWS Glue/Step Functions, Apache Airflow, Azure Data Factory, and GCP Dataflow/Cloud Composer. Broader cloud resource control now covers all three clouds too (RDS/S3/Lambda, Azure Postgres Flexible Server/Storage Accounts/Function Apps, and Cloud SQL/GCS/Cloud Functions); more resource types within each cloud are planned for upcoming phases.