Skip to content

depictio ↔ Spectio harmonisation — code audit and proposal

Source: depictio checkout at scratchpad/dep (paths below are relative to dep/depictio/ unless they start with packages/). Read on 2026-10-25.

1. What depictio does today

1.1 Identity and authentication

  • User (models/models/users.py): UserBase = email, is_admin, is_anonymous, is_temporary, expiration_time; UserBaseUI adds is_active, is_verified, last_login, registration_date, display_name, title, login, sso_provider, must_change_password; User adds password. Stored with Beanie (UserBeanie, MongoDB).
  • Group (models/models/users.py, class Group): name, users_ids, admin_ids, pi_id (principal investigator), pending_pi_email, sso_managed. Admins and the PI are always members (validator _admins_and_pi_are_members).
  • Sysadmin: a single boolean is_admin on the user; sysadmins bypass every permission (api/v1/permissions.py docstring).

  • Auth modes (api/v1/configs/settings_models.py, AuthConfig, env prefix DEPICTIO_AUTH_):

    • single_user_mode: "Grants admin privileges to the anonymous user for full functionality without authentication"; the anonymous user email defaults to anonymous@depict.io (anonymous_user_email).
    • public_mode (alias unauthenticated_mode): visitors get temporary users (temporary_user_expiry_hours), optional public_access_code.
    • demo_mode: public mode plus guided tour.
    • registration_disabled, password_login_disabled.
    • The UI asks GET /auth/me/optional-style info returning auth_mode ∈ {single_user, unauthenticated, standard} (api/v1/endpoints/user_endpoints/routes.py ~l.633–703).

  • Login methods: email + password (POST /auth/login), Google OAuth (/auth/google, google_oauth_* settings), SAML 2.0 SSO tested with Keycloak (/auth/saml, saml_* settings, api/v1/endpoints/auth_endpoints/saml_routes.py), magic links (MagicLinkTicket, POST /auth/me/magic_link, /auth/magic/exchange). No generic OIDC provider.

  • SSO group sync: saml_attribute_groups (e.g. memberOf) mirrors sso_managed groups on each login (api/v1/endpoints/auth_endpoints/sso_sync.py).
  • Tokens (models/models/users.py, TokenData / TokenBase): JWT (sub = user ObjectId, exp, signed RS256 by default with keys in keys_dir, AuthConfig.keys_algorithm), token_lifetime ∈ {short-lived, long-lived, permanent}, access + refresh tokens, optional name. Users create named long-lived CLI tokens with POST /auth/me/tokens (user_endpoints/routes.py l.1260).
  • Service-to-service: internal_api_key (auto-generated) and a separate provisioning_api_key used by pipelines to create-or-get a passwordless user and obtain a long-lived run token (POST /auth/provision_user, routes.py l.327).
  • CLI config (models/models/cli.py, CLIConfig): YAML with api_base_url, user (email, is_admin, id, token …), s3_storage (MinIO/S3 credentials for direct Delta writes), instance_label, remote_storage_options. Default path ~/.depictio/CLI.yaml; env overrides DEPICTIO_CLI_TOKEN, DEPICTIO_CLI_API_BASE_URL, DEPICTIO_CLI_CONFIG_PATH (documented in cli/configs/nextflow/depictio.config). Generated by POST /auth/generate_agent_config, which includes the S3 secret (restricted to admins in single-user/public modes, routes.py l.1422).

1.2 Tenancy and permissions

  • Project (models/models/projects.py): name, data_management_platform_project_url (link to an external data-management platform), workflows, data_collections, joins, links, yaml_config_path, permissions, is_public, project_type ∈ {basic, advanced}, realtime, template_origin, registration_time, hash.
  • Permission (models/models/users.py, class Permission), embedded in each permission-carrying document (projects, dashboards, workflow runs…): owners, editors, viewers (viewers may contain the wildcard "*"), plus group_owners, group_editors, group_viewers. One direct role per user and one role per group.
  • Enforcement (api/v1/permissions.py): three levels viewer < editor < owner; effective role = highest of direct role and any group role; is_public / "*" give a viewer floor; anonymous users see only public documents; group membership resolved live from the groups collection; helpers build_access_query, has_permission, resolve_effective_role used by every endpoint.

1.3 Workflows, runs and data

  • Workflow (models/models/workflows.py): name, engine (name, version), version, catalog (name, url), workflow_tag, repository_url, data_collections, runs (dict of WorkflowRun), config (WorkflowConfig: workflow_parameters, engine_name, pipeline_version, nextflow_version, tools_executed), data_location (structure, locations, runs_regex).
  • WorkflowRun (same file): workflow_id, run_tag, files_id, workflow_config_id, run_location, creation_time, last_modification_time, registration_time, run_hash, scan_results, permissions. It is a results directory registered for dashboards, not an execution.
  • WorkflowRunInfo (models/models/run_info.py): engine-agnostic provenance read from a results directory (engine, pipeline_name, pipeline_version, engine_version, run_name, params, tools_executed, paths to software versions, params JSON, execution report/trace, DAG). Readers are pluggable connectors (Nextflow reader in models/models/nextflow.py, NextflowRunInfoReader); used by depictio ingest <results dir> to auto-select a template.
  • Jobs (models/models/jobs.py): JobState = pending | running | success | failed | cancelled, JobKind = deltatable.upsert | project.ingest, with idempotency_key, project_id, progress — background ingestion jobs (Celery).
  • Data collections (models/models/data_collections.py and data_collections_types/: table, image, multiqc, jbrowse, geojson, indexed_file, phylogeny…), scanned with ScanRecursive (regex + wildcards), ScanSingle, ScanURL, ScanS3Prefix, ScanManifest.
  • Templates and catalog: per-pipeline, per-version project templates for ~35 nf-core pipelines (projects/nf-core/<pipeline>/<version>/template.yaml, e.g. projects/nf-core/rnaseq/3.26.0/template.yaml with template_id: nf-core/rnaseq/3.26.0, variables DATA_ROOT, SAMPLESHEET_FILE, skip flags, bundled dashboards). A tool catalog maps nf-core outputs → recipes → dashboard components (catalog/README.md, one folder per module).
  • Ingestion CLI (cli/cli/commands/run.py, ingest): depictio ingest DATA_DIR with --template, --project-config-path, --manifest, --project, --attach-run (add a run to an existing project), --update-config, --pipeline-id nf-core/ampliseq/2.16.0, --triggered-by ("a pipeline's completion trigger passes its engine"), --user + --provisioning-key (run as a provisioned user, emit a login link), --provenance-file, --result-json (project/dashboard ids and links for scripts), --compose (build from the catalog when no template fits).
  • Existing Nextflow completion trigger: cli/configs/nextflow/depictio.config installs a workflow.onComplete handler that runs depictio-cli ingest on params.outdir (params depictio_enabled, depictio_template, depictio_project, depictio_attach, depictio_user, …). No Snakemake equivalent was found.
  • Storage: MongoDB via Beanie (MongoDBConfig), S3/MinIO + Delta tables for data (S3DepictioCLIConfig, models/models/deltatables.py), Redis cache and Celery workers (settings_models.py ~l.794–827).
  • IDs: MongoDB ObjectId (PyObjectId in models/models/base.py).
  • API: prefix /depictio/api/v{major} (api/main.py l.264, version.py get_api_version), routers /projects, /workflows, /runs, /datacollections, /files, /deltatables, /dashboards, /groups, /auth, /events, /jobs, … (api/v1/endpoints/routers.py). Routes are partly verb-style (/runs/list/{workflow_id}, /runs/get/{run_id}, /runs/upsert_batch).

1.4 Events and external hooks

  • WebSocket /events/ws with the JWT passed as a query parameter (api/v1/endpoints/events_endpoints/routes.py); EventMessage carries event_type, source_type, project_id, dashboard_id, data_collection_id, payload (models/models/realtime.py); projects opt in with RealtimeConfig.
  • No outgoing webhooks were found (the only "webhook" match is a secret-redaction regex in api/v1/endpoints/projects_endpoints/run_folders.py).
  • External automation relies on the provisioning key, long-lived tokens and the CLI.

1.5 Front end

  • Shared React core packages/depictio-react-core (Mantine theme in brandTheme.ts, chrome variants).
  • Instance branding (models/models/branding.py, BrandTheme): app_name, logo modes, primary / secondary / tertiary, success / warning / danger, tint_mode, light and dark surfaces, plot colorway, fonts, radius. brandTheme.ts maps primary → Mantine blue, secondary → teal, tertiary → palette brandTertiary.
  • Section colours (viewer/src/chrome/AppSidebar.tsx): Dashboards = orange (role tertiary), Projects = teal (secondary), Administration = blue (primary), About = gray.
  • Admin (viewer/src/admin/AdminApp.tsx l.74–103): nav groups People (Users, Groups), Content (Projects, Dashboards), Activity (Tasks, Ingestion, Watchers, Logs, Health), Instance (Branding, Backups, Maintenance, Settings).
  • Profile (viewer/src/profile/ProfileApp.tsx l.642–653): Email, Name, Title, Role (Administrator/User), Database ID, registration date, last login, avatar, CLI tokens.
  • Project permissions (viewer/src/projects/detail/PermissionsApp.tsx l.842–1042): sections Visibility (public toggle), Members (role per email), Groups (role per group).
  • Groups (viewer/src/groups/GroupsApp.tsx).

2. Harmonisation proposal

(a) Identity

  • One identity provider for both. depictio speaks SAML 2.0 (validated against Keycloak) and Google OAuth, not OIDC. Spectio should support the same SAML configuration keys (or put Keycloak in front as a broker that offers SAML to depictio and OIDC/SAML to Spectio). Same IdP, same memberOf attribute → the same SSO-managed groups appear in both apps with the same names.
  • Co-deployed phase (before SSO): depictio is the identity source. Spectio accepts depictio JWTs (verify the RS256 signature with depictio's public key, or call GET /depictio/api/v1/auth/me), creates or updates a principal keyed by depictio:user/<ObjectId> with the same email, and mirrors groups (users_ids, admin_ids, pi_id) on demand. One login, one token for a user's browser.
  • Service-to-service: Spectio holds a depictio service identity (provisioning key, or a long-lived token of a depictio service user) and uses --user / POST /auth/provision_user to ingest as the run's owner, so dashboards land in that person's projects.
  • Single-user: depictio single-user mode = anonymous admin (anonymous@depict.io); Spectio single-user = local principal. When both run in single-user mode, Spectio maps its local user to depictio's anonymous admin; no tokens are needed on localhost.

(b) Entity mapping

Spectio depictio Notes
Principal (kind user) User match on email; store depictio:user/<id> as an identity link
Principal (kind service/agent) none (provisioning key / long-lived token of a service user) depictio has no service-account model
Instance admin User.is_admin same meaning (bypass)
Group, member / admin Group.users_ids / admin_ids add pi to Spectio group roles to match pi_id; keep sso_managed
Project Project 1:1 link through external_refs (depictio:project/<id>); Spectio project visibility instance ↔ is_public
Grant role viewer / owner viewers / owners direct mapping
Grant roles launcher, reviewer, maintainer editors depictio editor ↔ Spectio maintainer by default; launcher and reviewer exist only in Spectio
Group grant group_owners / group_editors / group_viewers same semantics (highest role wins)
Workflow + version Workflow (engine, version, catalog, repository_url) depictio template id nf-core/<name>/<version> = Spectio workflow version key
Run (execution) WorkflowRun (results directory) + ingestion Job one Spectio run → one depictio WorkflowRun after ingestion; run_tag = Spectio run ID
Artifact (report, output) DataCollection + files depictio decides the data collections from its template
Run provenance / RO-Crate WorkflowRunInfo + --provenance-file pass Spectio's run summary as the provenance file
Event EventMessage (WebSocket) different transports; see (c)

(c) Run → dashboard hand-off

depictio already supports pipeline-triggered ingestion (depictio.config + depictio ingest --pipeline-id --triggered-by --attach-run --user --result-json). Spectio should drive it rather than invent a new path:

  1. In spectio.yml (or the workflow's Spectio settings), an optional block: 

    depictio:
  on: completed            # or: gate:<id>, stage:<n>
  template: auto           # auto = pick by pipeline id (nf-core/rnaseq/3.27.0); or an explicit template id
  project: "{project.name}"
  attach_run: true         # one depictio project per Spectio project, one run per Spectio run

  2. When the run completes, the runner (which already has file access next to the results) runs depictio-cli ingest <outdir> --pipeline-id <workflow>/<version> --triggered-by spectio --project <name> --attach-run --user <owner email> --provenance-file <spectio run summary> --result-json <file>, with credentials from the server (provisioning key or service token).

  3. The runner returns the result JSON; the server stores depictio:project/<id> and dashboard links on the run, shows "Open in depictio" in Run detail, and emits spectio.run.dashboard_ready.
  4. Spectio disables depictio's own Nextflow onComplete trigger for runs it launches (--depictio_enabled false) to avoid a double ingestion. For Snakemake, Spectio provides the trigger that depictio lacks.
  5. Later, a server-side ingestion call (depictio Job kind project.ingest) can replace the CLI call when depictio can read the storage directly.

(d) IDs and naming

  • Keep each side's native IDs (ObjectId in depictio, prefixed ULIDs in Spectio) and link them with external_refs both ways (depictio:project/…, spectio:run/…).
  • Use the same pipeline identifier string everywhere: <org>/<name>/<version> (depictio template ids and --pipeline-id already use it).
  • Role vocabulary: expose depictio-compatible role names in Spectio's sharing API (owner, editor, viewer) and treat launcher, reviewer as Spectio capabilities layered on editor; or map as in (b). Decide once.
  • API paths for co-hosting on one domain: depictio is /depictio/api/v1; mount Spectio at /spectio/api/v1 (and the UI at /spectio/) so both live behind one reverse proxy and share cookies/SSO.
  • CLI: both CLIs keep their own config (~/.depictio/CLI.yaml, ~/.spectio/config.yaml), but spectio login can reuse an existing depictio token when the server is configured to trust depictio.

(e) UI consistency

  • Same Mantine theme and same branding model: Spectio reads the instance BrandTheme (primary / secondary / tertiary, surfaces, fonts, radius) when co-deployed, so an institute brands both apps once.
  • Align section colours: depictio uses Projects = teal (secondary), Dashboards = orange (tertiary), Administration = blue (primary). Spectio's own menus (Runs, Review, Launch, Workflows, Environments) should use colours distinct from depictio's, except Projects/Settings, which should match. Today Spectio's Workflows orange (#C9771A) is close to depictio's Dashboards orange: to revisit.
  • Mirror depictio's admin structure in Spectio Settings: People (Users, Groups, Service accounts), Content (Projects, Workflows, Environments), Activity (Runs, Events, Logs, Health), Instance (Branding, AI & agents, Webhooks, Settings).
  • Same Profile fields (email, name, title, role, ID, dates, avatar, tokens) and the same project-permissions screen layout (Visibility, Members, Groups).
  • An app switcher in the top bar (depictio ↔ Spectio) and deep links both ways (run → dashboard, dashboard → producing run).

(f) Gaps and risks

  • Databases differ (MongoDB vs SQLite/Postgres): no shared tables; integration only through APIs, tokens and identity links. Keep it that way.
  • No OIDC in depictio: Spectio's planned OIDC must be complemented by SAML, or both put behind Keycloak.
  • Role mismatch (3 levels vs 5 roles) needs an explicit mapping, otherwise a depictio editor may get launch rights unintentionally.
  • Single-user semantics differ (anonymous admin vs named local user): only matters when both run together; map explicitly.
  • Secrets: depictio's agent config contains the S3 root secret (generate_agent_config); Spectio runners must not receive it unless needed for ingestion, and should use a scoped token instead where possible.
  • depictio API style is partly verb-based with deprecated routes; Spectio should wrap the few calls it needs (/auth/me, /auth/provision_user, /projects, ingestion) in one adapter plugin (output sink "depictio").
  • Event transports differ (depictio WebSocket only, Spectio SSE + webhooks): bridge only what is useful (Spectio run.dashboard_ready → depictio project refresh event).