Skip to main content

Operations Guide

This guide covers day-to-day operations for a deployed Genie Workbench instance: Lakebase management, MLflow configuration, monitoring, and GSO job management.

Lakebase​

Schema and Tables​

The app creates the genie schema and all tables on first startup (the SP owns everything it creates). Data is stored in the databricks_postgres database:

TablePurpose
scan_resultsIQ scan history: score, maturity, checks, findings, timestamps
starred_spacesUser-starred agents for quick access
seen_spacesTracks which agents the user has visited
optimization_runsLegacy optimization accuracy records (used by scanner checks 11–12)
agent_sessionsCreate agent session persistence (message history, step state)

Credential Refresh​

Lakebase credentials are auto-generated via the Databricks SDK (postgres.generate_database_credential for autoscaling, database.generate_database_credential for provisioned). These OAuth tokens expire after ~1 hour, so the app recreates the asyncpg connection pool every 50 minutes to stay ahead of expiration.

Graceful Degradation​

If LAKEBASE_HOST is not configured (no Lakebase attached), the app falls back to in-memory dictionaries. The app remains fully functional but:

  • Scan results are lost on restart
  • Starred Genie Agents are lost on restart
  • Agent sessions are lost on restart
  • The Admin Dashboard shows no historical data

Troubleshooting Lakebase​

SymptomCauseFix
"Failed to list spaces"Lakebase not attachedRe-run deploy.sh --update to auto-attach the postgres resource
Connection errors after ~1 hourToken refresh failedCheck app logs for credential generation errors
Tables not createdSP lacks CONNECT or CREATE ON DATABASERe-run deploy.sh --update to re-create the SP role and grants

MLflow​

Experiment Tracking​

LLM calls in the create agent and optimization pipeline are traced via MLflow. Tracing is optional — controlled by the MLFLOW_EXPERIMENT_ID environment variable in app.yaml.

At startup, the app validates that the experiment ID exists in the workspace. If it doesn't, tracing is silently disabled (the variable is cleared).

Prompt Registry​

Auto-Optimize requires MLflow Prompt Registry for versioned judge prompts. If Prompt Registry is not enabled on the workspace, the optimization preflight task will fail with FEATURE_DISABLED.

Configuration​

# In app.yaml
- name: MLFLOW_TRACKING_URI
  value: "databricks"
- name: MLFLOW_REGISTRY_URI
  value: "databricks-uc"
- name: MLFLOW_EXPERIMENT_ID
  value: "<your-experiment-id>"

The experiment ID is workspace-specific. The installer can create one during setup, or you can create one manually and update app.yaml.

Monitoring​

App Logs​

databricks apps logs <app-name> --profile <profile>

App Status​

databricks apps get <app-name> --profile <profile>

Verify Workspace Files​

databricks workspace list /Workspace/Users/<email>/<app-name>/backend --profile <profile>

Key Log Patterns​

Log PatternMeaning
OBO: using user token for /api/...Request authenticated via user's OBO token
OBO: no x-forwarded-access-token, using SPNo user token — using SP (expected for health checks)
OBO token lacks genie scope, retrying with service principalGenie API scope fallback triggered
Lakebase pool createdDatabase connection established
Lakebase pool re-created (credential refresh)Scheduled 50-minute token refresh
Failed to persist scan resultLakebase write failed (check connectivity)

GSO Job Management​

Job Creation​

The optimization job is created automatically during deploy.sh via databricks bundle deploy -t app. It uses Terraform state scoped to the deployer.

Job Reuse​

If the job already exists (from a previous deploy), it is reused. To force recreation:

  1. Delete the job in the Databricks UI
  2. Re-run ./scripts/deploy.sh --update

ensure_job_run_as Self-Healing​

At app startup, _ensure_gso_job_run_as() checks that the optimization job's run_as matches the current app SP. If they don't match (e.g., the app was redeployed with a different SP), the job is automatically updated. This avoids manual reconfiguration when the app identity changes.

Bundle Management​

The GSO job is managed by Databricks Asset Bundles (DABs):

# Deploy/update the job (done automatically by deploy.sh)
databricks bundle deploy -t app --profile <profile>

Important: Do NOT run databricks bundle deploy -t dev for production deployments — it creates [dev username] prefixed orphan jobs with separate Terraform state.

The app target uses mode: development for per-deployer Terraform state with presets.name_prefix: "" for clean job names.

Post-Deploy: Genie Agent Access​

After deploying, the app's SP needs access to Genie Agents for API fallback and optimization:

  1. The installer grants SP access to your existing Genie Agents
  2. For agents created after install, share them with the SP (CAN_MANAGE)
  3. Grant SP SELECT on referenced schemas:
GRANT SELECT ON SCHEMA <catalog>.<schema> TO `<service-principal-name>`;

See Authentication & Permissions for the full permission model.