A practical reference for working in SILKEN.

What the application is for

SILKEN is a visual astronomy workspace for inspecting the same sky position across image surveys, catalogues, time-domain cutouts, and event footprints. It is aimed at astronomers who need to assemble and compare public or CANFAR-accessible data before moving into more specialised reduction or analysis tools.

It is not a replacement for a reduction pipeline or a data archive. It keeps the exploratory context around those systems together: sky position and field of view, visible imagery, catalogue queries and overlays, frame playback, plot state, job results, and links to the underlying products.

Connect code, create tools, and run reviewed revisions

GitHub and CANFAR Harbor are optional connections in Settings → Connections. GitHub uses a dedicated SILKEN OAuth App with authorization code, PKCE, and signed single-use state. The broad repo scope lets SILKEN inventory repositories that include private entries, but SILKEN uses it only for repository metadata and never requests workflow. Harbor accepts and validates an existing CLI secret; it never creates or rotates one.

A short first workflow

A catalogue result does not recenter or change the zoom. Reposition the viewer explicitly when you want a different comparison field.

SILKEN user manual

Use this section as a compact map of the interface. Controls that are icon-only have a tooltip: hover the icon, or focus it with the keyboard, to see its name before using it.

Start, restore, and preserve a workspace

Navigate the sky and manage imagery

Query, visualize, and inspect catalogues

Use time-domain imagery

Work with files, code, tools, and jobs

Settings, handoffs, and common recovery actions

• Use a descriptive work-session name before building a large comparison; saved sessions are easiest to distinguish by name and last-save time.
• Keep generated products in personal or project ARC paths. Treat /scratch and the container filesystem as temporary.
• If an image obscures catalogue markers, lower its opacity, change blend mode, move it in the Stack panel, or temporarily hide it.
• For smooth playback, wait for the initial frame window before pressing Play. SILKEN bounds the decoded browser window and caches normalized FITS bytes in the contributed CANFAR runtime; larger session RAM can increase that cache, while GPU allocation is reserved for server-side workloads that explicitly support it.

Import FITS images, cubes, mosaics, and existing HiPS

Use Import image data in Image Layers to choose one FITS file, an existing HiPS root, or a directory tree. Inspection runs inside the contributed CANFAR session, so mounted ARC and user storage are read directly. Only roots that are actually mounted in that runtime are offered for image import; a remotely browsable path is not mistaken for a locally importable source. The scan opens headers lazily and does not read every pixel just to identify the products.

Supported names are case-insensitive .fits, .fit, .fts, .fits.gz, .fit.gz, and .fz. Every image HDU in a multi-extension file is considered. SILKEN records celestial frame, footprint, projection, pixel scale, axes, filter or frequency, epoch, unit, beam, exposure, product role, and the evidence used for inferred classifications. Filename clues can identify masks, weights, PSFs, noise maps, or moments, but filenames are never used to guess a sky position. Nested directories are crawled recursively, symbolic-link cycles and generated .silken products are skipped, and unreadable branches or scan limits are listed explicitly in the review.

What the review means

FITS collections have no universal required directory layout. Archives commonly arrange files by target, observation, band, epoch, pointing, tile, or pipeline product; the recursive scanner does not depend on one of those conventions. It uses headers to group compatible data and shows the proposed groups and footprints before import without moving the current Aladin centre or zoom.

Combine compatible is the default. The review can additionally split the saved header manifest by band, observable, product role, unit, epoch, Stokes, HDU, cube-axis type, or resolution, and by a chosen relative-folder depth or leaf folder. These controls can be combined. Regrouping is immediate because it never rereads FITS headers. A Back icon abandons the inspection and restores the previous path, selected item, filter, sort direction, and scroll position.

Image-layer badges describe the evidence that exists: Survey image, Single epoch, Multi-epoch image, Epoch sequence, or a time, spectral, polarization, distance, or generic data cube. A broad survey HiPS such as ZTF DR7 is a survey/coadded image; archive cutouts imported as a frame sequence are the playable time-domain product. SILKEN does not infer cadence from a dataset name.

Inspection manifests, fingerprints, exclusions, provenance, prepared assets, and generated HiPS products are cached below /arc/home/<cadc-user>/.silken/layers/ by default. Re-importing unchanged inputs and settings reuses the existing product. Large builds appear in Jobs with staging, indexing, tiling, validation, and registration progress. Saved sessions restore the registered layer; a changed or missing source remains in the sidebar with a repair or re-import action instead of disappearing.

How comprehensive catalog discovery works

The setup wizard searches a durable semantic index rather than downloading every table and column into the browser. The public part is refreshed nightly from configured HATS and Youcat sources, curated CADC/Rubin/ multimessenger descriptors, and TAP services published through the IVOA Registry. If RegTAP lacks a schema, SILKEN records the gap and attempts a bounded direct VOSI or TAP_SCHEMA read. Administrative registry schemas are indexed for coverage accounting but hidden by default.

Science area is a broad discipline, science use case describes the intended analysis, and Object / event type names the physical source or event. Object/event tags are multi-label. Science-area matching also follows use-case membership, so DESI and other redshift tables remain discoverable under Cosmology & Large-scale Structure even when their broad display group is galaxies.

Coverage labels

Source finding can include scientific tables without sky coordinates. Cross-match mode filters to catalogs with a usable RA/Dec pair. The default selection means all catalogs matching the current filters plus any explicit inclusions and minus exclusions, so a registry-scale plan does not require thousands of browser checkboxes. SILKEN shows the estimated catalogs, providers, metadata gaps, and unbounded-row warning and requires confirmation before execution.

Query slices and execution

A source-finding plan creates one provider-query slice per selected catalog table. A cross-match plan creates one slice for every sky-region row and catalog-table combination; results are grouped by region before any pair matching, so positions and tolerances never leak between imported rows. The wizard reports this slice count before the user confirms the run.

For private Google Sheets, download an XLSX copy and use the XLSX importer. SILKEN does not ask the browser for a separate Google credential.

Import examples: valid cross-match CSV · invalid CSV that first fails at C4.

Launch SILKEN from the CANFAR Science Portal

Sign in to the CANFAR Science Portal and use Launch New Session. SILKEN is a contributed web application, so choose contributed, not notebook. The image already contains the required /skaha/startup.sh and serves the application on port 5000.

Standard tab

If candiapl / silken is not offered in the Standard image selectors, use Advanced. The Standard list depends on the registry inventory visible to the portal.

Advanced tab

CANFAR’s contributed-application contract and troubleshooting guidance are maintained in the official contributed applications guide.

A gateway in front of replaceable compute

The public endpoint and the application process have different lifetimes. The gateway is inexpensive and continuously reachable. The application process is created for a user, receives mounted storage and identity, and can be destroyed when it is idle.

The request lifecycle

The session type describes the interface

“Interactive session” does not imply a notebook. The selected session type is a routing and startup contract. A custom web-application session can run any HTTP application as long as its image satisfies that contract.

name=application-user-unique-id
image=registry.example.org/team/application:immutable-tag
type=custom-web-application
command=/platform/startup.sh
environment=APP_HOST=0.0.0.0
environment=APP_PORT=<platform-web-port>
# The process stays in the foreground and answers a health endpoint.

Publish registry discovery metadata separately

Container configuration labels and registry artifact labels serve different consumers. An image can declare its runtime interface in its Docker configuration while a platform discovers eligible images through a separate Harbor artifact label. Making a project public only permits pulls; it does not attach a discovery label, and a project-scoped label is not a substitute when the platform expects an existing global label.

Create a project robot

A project robot is a non-human Harbor identity whose permissions stop at one project. It lets deployment automation maintain discovery metadata without inheriting a maintainer's account privileges. In Harbor, open Projects → <project> → Robot Accounts → New Robot, choose a descriptive name such as <application>-github-labeler, and set an expiration that is monitored and rotated.

• Grant Artifact Label: Create (artifact-label:create).
• Grant Artifact: Read/List and Repository: Pull/List so the workflow can resolve and verify the artifact.
• Do not grant repository push/delete, scanner, project-admin, or system permissions.
• Copy the generated username and secret immediately. Harbor prefixes the username, for example robot$<project>+<application>-github-labeler, and shows the generated secret only once.
• Save the values as repository secrets HARBOR_LABEL_USERNAME and HARBOR_LABEL_PASSWORD.
• Run the repository's label-only workflow against an existing tag before relying on the credential for a release.

For rotation, create or refresh the robot credential, update both repository secrets, pass the label-only workflow, and only then retire the previous credential. Never place either generated value in source, workflow YAML, logs, or documentation.

gh secret set HARBOR_LABEL_USERNAME --repo OWNER/REPOSITORY
gh secret set HARBOR_LABEL_PASSWORD --repo OWNER/REPOSITORY

import os
from urllib.parse import quote

import requests

api = os.environ["HARBOR_API"].rstrip("/")
project = os.environ["HARBOR_PROJECT"]
repository = os.environ["HARBOR_REPOSITORY"]
reference = os.environ["HARBOR_REFERENCE"]  # immutable release or commit tag
label_name = os.environ["HARBOR_LABEL"]
auth = (os.environ["HARBOR_LABEL_USERNAME"], os.environ["HARBOR_LABEL_PASSWORD"])

# Resolve the existing global label. Never hard-code its registry-owned ID.
found = requests.get(
    f"{api}/labels",
    auth=auth,
    params={"scope": "g", "name": label_name, "page_size": 100},
    timeout=30,
)
found.raise_for_status()
label = next(item for item in found.json() if item["scope"] == "g" and item["name"] == label_name)

root = f"{api}/projects/{quote(project, safe='')}/repositories/{quote(repository, safe='')}/artifacts"
artifact = requests.get(f"{root}/{quote(reference, safe='')}?with_label=true", auth=auth, timeout=30)
artifact.raise_for_status()
digest = artifact.json()["digest"]

attached = requests.post(
    f"{root}/{quote(digest, safe='')}/labels",
    auth=auth,
    json={"id": int(label["id"])},
    timeout=30,
)
if attached.status_code not in (200, 201, 409):  # 409 means already attached
    attached.raise_for_status()

for verify_reference in (digest, reference, "latest"):
    checked = requests.get(
        f"{root}/{quote(verify_reference, safe='')}?with_label=true",
        auth=auth,
        timeout=30,
    )
    checked.raise_for_status()
    payload = checked.json()
    assert payload["digest"] == digest
    assert any(item["scope"] == "g" and item["name"] == label_name for item in payload["labels"])

{"1":"carta","2":"desktop","3":"desktop-app","4":"notebook","5":"headless","8":"contributed","11":"firefly"}

External references

• Harbor OIDC authentication and CLI-secret scope
• Harbor project robot accounts

Persist state outside the session

A container filesystem, temporary directory, or in-memory session store is not a durable workspace. Store user-visible state under a mounted home or project directory and make the writes resilient to interruption.

payload = json.dumps(state).encode("utf-8")
temporary = state_path + ".tmp"

write_bytes(temporary, payload)
rename(temporary, state_path)  # atomic within one filesystem

A replacement test is the strongest proof: save recognizable work, terminate the compute session, create another one, and restore from the same mounted path.

How SILKEN applies the pattern

SILKEN is an astronomy workspace for progressive image layers, catalogs, crossmatching, cutouts, and multimessenger data. Its public shell and scientific runtime are deliberately separate.

Two supported entry paths

Use another account deliberately opens the CADC account chooser. Sign out of SILKEN clears the local pending identity; it does not claim to terminate every CADC identity-provider session in other tools.

Container and launch contract

The contributed image advertises its type, installs an executable startup script, and serves HTTP on CANFAR’s contributed-application port.

LABEL ca.nrc.cadc.skaha.type="contributed"

COPY scripts/silken-backend-startup /skaha/startup.sh
RUN chmod +x /skaha/startup.sh

ENV HIPS_SERVER_HOST=0.0.0.0
ENV HIPS_SERVER_PORT=5000
EXPOSE 5000
CMD ["/skaha/startup.sh"]

name=silken-<user>-<unique-suffix>
image=images.canfar.net/candiapl/silken:latest
cores=2
ram=8
type=contributed

The BFF lists existing type=contributed sessions with the user-specific SILKEN prefix. It re-enters one only after checking health, version, and required capabilities; otherwise it replaces it. A per-user launch lock prevents duplicate backends. Skaha uses the image command and injects the authenticated user, session ID, and mounted home; contributed launches do not accept caller-supplied command or environment fields.

Private proxy boundary

Browser requests remain on the public origin. The BFF keeps Skaha’s connect URL server-side and adds X-SILKEN-Session-Secret and the authenticated username when forwarding a request. The private runtime rereads an owner-bound record in /arc/home/<cadc-user>/.silken/backend-secrets/ for every protected request. The BFF writes, rotates, and deletes that record through authenticated ARC operations; it never places the secret in the contributed launch form.

Saved workspaces in .silken

The interactive container is ephemeral. SILKEN stores restorable work in the user’s mounted home directory, so a later or replacement runtime sees the same files.

The workspace state includes the sky view, active layers, query tabs, ADQL, overlay settings, time-domain controls, tab order, HATS state, CADC search/provenance state, and multimessenger state.

Import products from CARTA, Firefly, or Neloura

The viewer toolbar has one compact external-products icon. The source choice tailors discovery and filters: CARTA focuses on radio images, spectral cubes, and regions; Firefly on catalogues, tables, survey images, and Vault products; Neloura on FITS images/cubes and catalogue files. The choice is provenance and browsing context only. Every product enters the same SILKEN installer and becomes an ordinary layer, catalogue, or region.

Files already available in personal or project ARC storage stay referenced in place. Catalogues use VOTable, images and cubes use FITS, and regions use DS9 where conversion is needed. Compute remains running after a return. The source-application shortcut can reopen, start, or stop the official CANFAR application when needed.

Interactive coordination, headless execution

The contributed session handles responsive API work. Large generation, indexing, source-finding, crossmatch, and cutout operations can run in CANFAR headless sessions or through the configured distributed executor.

Because jobs persist their manifest, progress, log, result, retry, and cancellation state under .silken/jobs, the Jobs view does not depend on one Python process staying alive.

The Jobs tab is a compact live queue view with state, backend, progress, elapsed time, and current phase. It refreshes once per second while visible, pauses network polling when the user leaves the tab, and refreshes immediately on return. Infrastructure-owned semantic snapshot refreshes use the deployment workflow and never appear in a user’s Jobs tab.

Deployment and replacement checks

• The contributed image satisfies the provider-neutral registry-publication contract described in Architecture.
• /skaha/startup.sh is executable and leaves the server in the foreground.
• The runtime binds 0.0.0.0:5000 and answers /healthz.
• The browser cannot see CADC tokens, Harbor credentials, private keys, connect URLs, or backend secrets.
• Saved state resolves beneath /arc/home/<user>/.silken, not /tmp or /opt.
• A brand-new contributed session restores a workspace saved by the previous session.
• Heavy jobs use persistent manifests and do not block the interactive API process.
• The single SILKEN importer path and each optional official CARTA, Firefly, and Neloura launch action pass authenticated CANFAR acceptance before the feature flag is enabled by default.

docker image inspect images.canfar.net/candiapl/silken:latest \
  --format '{{ index .Config.Labels "ca.nrc.cadc.skaha.type" }}'
# expected: contributed

curl --fail http://127.0.0.1:5000/healthz

test -d "/arc/home/$USER"
find "/arc/home/$USER/.silken" -maxdepth 4 -type f -print

Implementation map

External references

• CANFAR container and session-type overview
• CANFAR contributed applications
• CANFAR batch and headless sessions