GCP audit logs incident response

GCP audit logs incident response

TL;DR: GCP Cloud Audit Logs split into four streams: Admin Activity (always on, free), Data Access (opt-in, per-service, expensive), System Event (auto), Policy Denied (auto). IR focuses on Admin Activity for control-plane changes and Data Access for object reads — when enabled. Trace via principal email / service account, correlate with VPC Flow Logs and Cloud DNS for egress. Companion to cloud-ir-aws-cloudtrail and gcp-workload-identity-federation-abuse.

What each log captures

Admin Activity Logs

  • Always enabled, free, 400-day retention default.
  • Covers create/update/delete on resources, IAM changes, project lifecycle.
  • This is the first-stop log for IR.

Data Access Logs

  • Opt-in per service.
  • Three sub-types: ADMIN_READ (read config), DATA_READ (read data), DATA_WRITE (write data).
  • Disabled by default for cost reasons; Storage, BigQuery, Pub/Sub etc.
  • Tenants without this enabled are blind to data exfil.

System Event Logs

  • Generated by GCP automatically (e.g., live migration, instance preemption).
  • Useful for context.

Policy Denied Logs

  • When a request is denied by IAM / VPC SC.
  • Indicator: attacker enumerating before they have access.

Other relevant signals

  • VPC Flow Logs — network egress.
  • Cloud DNS query logs — exfil via DNS or pivoting to attacker C2.
  • Cloud Asset Inventory — point-in-time resource state for before/after comparison.
  • Security Command Center — managed findings.

Identity model recap

GCP IAM principals:

  • User — user:email@domain.
  • Service account — serviceAccount:name@project.iam.gserviceaccount.com.
  • Group — group:.
  • Workload Identity Federation — principal://iam.googleapis.com/projects/<n>/.../subjects/... (gcp-workload-identity-federation-abuse).

authenticationInfo.principalEmail and authenticationInfo.serviceAccountKeyName (for SA-key-authenticated calls) are key identity fields.

Investigation flow

Step 1 — Anchor

Alert points at:

  • Principal (user / SA / WIF subject).
  • Resource (project / bucket / instance).
  • Method (API call).
  • Time.

Step 2 — Principal tracing

For a suspicious principal:

  • protoPayload.authenticationInfo.principalEmail — caller.
  • protoPayload.authenticationInfo.serviceAccountDelegationInfo — if SA was impersonated.
  • For WIF: protoPayload.authenticationInfo.principalSubject — federated subject (e.g., GitHub repo).

Recurse through impersonation chains:

  • GenerateAccessToken from one SA to another.
  • ImpersonateServiceAccount events.

Step 3 — Resource tracing

For each impacted resource:

  • protoPayload.resourceName — what was touched.
  • protoPayload.methodName — what operation.
  • protoPayload.request / protoPayload.response — details.

Step 4 — Persistence hunt

GCP-specific persistence:

  • Service account key creation — google.iam.admin.v1.CreateServiceAccountKey. SA keys never expire by default.
  • IAM grant — SetIamPolicy on a resource granting attacker broad rights.
  • Custom role with broad permissions and inconspicuous name.
  • Workload Identity Pool / Provider added trusting attacker (see gcp-workload-identity-federation-abuse).
  • OS Login SSH key added on Compute instance.
  • VM instance metadata — startup script added.
  • Cloud Function / Cloud Run service deployed under privileged SA.
  • Cloud Build trigger with broad IAM.
  • Org policy / VPC SC perimeter modification.

Step 5 — Exfil hunt

  • storage.buckets.getIamPolicy / storage.objects.list mass-listing.
  • bigquery.jobs.create with large export queries.
  • compute.instances.export snapshot operations.
  • secretmanager.secrets.versions.access mass access.
  • VPC Flow Logs egress to non-Google IPs.

Common attacker patterns

  • Service-account key download + use from attacker IP.
  • WIF misconfig exploit (see gcp-workload-identity-federation-abuse) yielding SA tokens.
  • gcloud auth print-access-token style enumeration.
  • GKE control-plane access via misconfigured RBAC (see cloud-ir-k8s-audit-logs).
  • OAuth consent grant to malicious external app for Workspace user.

Tooling

  • gcloud logging read — interactive.
  • Cloud Logging in console — Log Explorer with structured filters.
  • BigQuery sink — query CloudAudit logs as a normal table (recommended for IR).
  • Datadog Security Labs GCP playbooks.
  • Goat — recon emulation.
  • Stratus Red Team for GCP (has some scenarios).

Pitfalls

  • Data Access opt-in — most tenants don’t have it; data reads are invisible.
  • Cross-project audit — by default each project has its own logs; centralise via Org-level log sink.
  • Service account impersonation chains — multi-hop chains can be hard to trace; map each SA’s roles/iam.serviceAccountTokenCreator grants in advance.
  • Workspace events — Workspace audit (Admin SDK, Drive, etc.) is in a separate Admin Console log.

Workflow to study in a lab

  1. Create a GCP project with org-level log sink to BigQuery.
  2. Enable Data Access logs on Storage and BigQuery.
  3. Run attacker-emulation scripts (Stratus or custom).
  4. Practice SQL queries on the BigQuery sink to surface each scenario.
  5. Author Cloud Logging alerting policies for the most signal-rich queries.

References