Core Architecture

Overview

suctl core is two stores — modules (config · identity · supervision) and messages (the work) — operated on by logic that owns no truth. Each store has exactly one orchestrator (its sole runtime writer): the broker over messages, the supervisor (Coordinator in code) over modules. Both consult the gate. Disk feeds the modules’ config, possession of an inherited wire feeds identity, and each store has exactly one presentation (inventory, jobs).

Two stores

A store is the category: a thing that holds non-derivable truth, keyed by its subject. suctl has exactly two — one per subject. Everything else is logic that reads or decides over them.

Store	Subject	Facets	Source / lifetime	Key
modules	the module	config (manifest, caps, footprint, requires) · identity (possession of inherited wire) · supervision (lifecycle state, restart history)	config from disk (boot/restart); identity+supervision per-launch	module short name
messages	the work	the recorded exchange: request + response envelopes, `ts_received` per direction, the resolved module + capability (the callee), and the caller (the calling module)	core itself; in-memory, reset on restart	request `id`

A module is one entity. config/identity/supervision are facets of the same row, written at different times — not separate truths. They differ only in refresh trigger (config = restart-stable; identity/supervision = per-launch).
Reality is the source of truth. External domain reality (system, processes, nginx) is read live, never cached — it is never a store. The two stores hold only what reality cannot re-derive: identity, supervision, and messages are core-generated; config is loaded from disk and core-owned until restart.
Disk is read only at boot / restart. After that, core runs from the modules store — edit a manifest and core ignores it until restart. Activation intent is persisted to disk on the operation and re-applied at boot, never polled.
Stores are built at boot, before any orchestrator exists. Construction is not orchestration: module.Scan reads disk into the modules store and messages.New creates the empty messages store. These boot-time builders are the one sanctioned writer that precedes the orchestrators; once core is running, each store has exactly one runtime writer — its orchestrator.

Vocabulary

store — the category: holds non-derivable truth. suctl has two: modules and messages.
modules store — every module’s truth, keyed by short name, in three facets: config, identity, supervision (see Identity below).
messages store — the recorded work. A record — one exchange — lives here, keyed by request id.
envelope — one JSON line on the wire: a request or a response. The unit the protocol defines (protocol.Request / protocol.Response); it carries no history and is discarded once read.
exchange — a request envelope paired with its response, sharing one request id. The unit the messages store records (as a record, + ts_received per direction). Some exchanges are actionable (a queued invoke); others are reports (job_update) or queries (job_status). “Actionable” is a state an exchange carries, not the type’s name.
record — one row in a store, keyed by its subject: module.Record (short name) · messages.Record (request id = one exchange). A messages.Record holds both envelopes, ts_received per direction, the resolved module + capability the exchange was routed to (the callee), and the caller (the calling module; empty = the system/face originator) — a full, self-describing message. Both ends of every exchange are known: caller (who asked) and module + capability (what ran). A module route forwards the same request id outward, so the crossing is a field of this one record, not a sibling row.
job — the set of records sharing a job_token: a derived query, never a stored row. State is read (terminal exchange in store ⇒ done/failed; else running once the queue manager has stamped its Started fact, else queued), never folded.
wire — the transport medium. An envelope is “on the wire” while moving.
orchestrator — core’s manager logic: the single authority over a domain, owning no truth of its own. A store mandates an orchestrator (its sole runtime writer); an orchestrator does not mandate a store — when its domain has a store (or stores) it is their sole writer, but the domain may hold no truth at all (a pure composer of reads). The two store-backed orchestrators are realized once per store, never as a single object:
- messages orchestrator (the broker) — the hot-path sequencer: identify → authorize → schedule → record → route. Owns the queue manager that promotes queued jobs; the queue itself is derived from the store, so the broker keeps only the wakeup primitive that un-parks promoted callers. Faces reach it on its inbound face listener (in-process today); no module dials it — every module rides its own socketpair.
- modules orchestrator (the supervisor, Coordinator in code) — owns module lifecycle (activate / deactivate / health / BIST) and writes the per-launch facets through the per-module supervisors it drives. Routes the lifecycle trigger; consults the gate before tearing down a busy module. Module BIST (activation-time protocol conformance) runs over the module’s wire, off the broker; its verdict is reflected as lifecycle state.
- surface orchestrator is the storeless case (the definition’s second half): the face’s single door for loading / refreshing a surface. It owns the surface/composition domain and no store — single authority, sole writer of nothing. It is the in-front sender (initiator) for face-originated work: it mints the originating id and job_token and calls the broker for each read the face asks of it — the survey on load, then one read per distinct column from the face requests after the survey resolves (whole-column fill). The broker originates nothing — it records and routes the complete envelopes it receives, so it stays the sole messages writer and every read is recorded; the orchestrator holds no truth and consults no gate of its own. It is two-way — the broker pushes each surface-originated job_update onto an inbound stream the face drains, so an async survey’s later result correlates back; its face-side mirror is the REPL orchestrator (below). See Presentation.
Both (message orchestrator, modules orchestrator) consult the gate; neither is dialed by a module.
gate — a pure decision logic (the orchestrator’s schedule step): an orchestrator asks, gate answers yes / no(+reason) from modules(footprint)
- messages(running). A pure function — no state.
supervisor — the modules orchestrator’s machinery: the per-module logic that writes the modules store’s per-launch facets. It binds identity at spawn (it holds core’s end of the module’s wire) and maintains supervision (lifecycle state, restart history). Activation/deactivation is its operation: validate requirements (read config) → persist intent to disk → on success spawn (or stop) and monitor/restart. The modules orchestrator routes the trigger. Activation needs no gate (it is not a job); deactivation first consults the gate’s busy verdict and refuses while a running job holds the module.
face — the initiator role: any client that speaks the wire to the orchestrator and renders sdk/surface DTOs. REPL is the only incarnation today.
REPL orchestrator — the face-side mirror of the surface orchestrator: the REPL’s single door for surface traffic in both directions. Outbound it asks the surface orchestrator to load / refresh a surface; inbound it correlates the answer back to the load that asked for it. Like its core-side peer it owns no truth and no cache — only the transient correlation it needs in flight. Because it learns a load’s job_token late (on the accepted reply), it holds a parking area: a buffer keyed by job_token for job_updates that arrive before the load is registered, replayed the instant it is. This makes correlation order-independent — a fast async survey whose result lands before its accepted is parked, not lost. Sync is the degenerate case of async: a synchronous result is just an async one that resolves in the same frame, so both ride the identical correlate-and-apply path.
module — the executor; owns domain reality. Reads-and-actions; holds no truth except rare justified cases (e.g. nginx suspend/maintenance intent).
router / requires — not components. Reads over the modules store’s config facet (cap→socket; declared requires), folded into the orchestrator.

Components by class

Two stores (truth):

modules — config · identity · supervision facets, keyed by module name.
messages — the recorded work.

Logic (owns no truth) — one orchestrator per store, plus storeless logic:

messages orchestrator (broker) — manages messages; reads modules (identify · authorize · route); the only runtime writer to messages.
modules orchestrator (supervisor / Coordinator) — owns module lifecycle (activate/deactivate = spawn/stop + monitor/restart); the only runtime writer to the per-launch facets (identity at spawn, supervision).
surface orchestrator (storeless) — the face’s read-composition authority for the surface domain; the initiator that mints the originating id+job_token and serves each read the face requests — the survey on load, then one broker invoke per distinct column from the face requests after the survey resolves; cells fill the face’s projection progressively. Two-way: it also pushes every surface-originated job_update onto an inbound stream so async results correlate back. Owns no store; writes nothing. Reaches everything through the broker’s one invoke surface (which fans out behind it to modules for actions and, via system.*, to the coordinator for lifecycle); the broker originates nothing and records the complete envelopes it receives.
gate — decision over modules(footprint) + messages(running) ⇒ yes/no; consulted by both store-backed orchestrators, dialed by no module.
builders (boot only) — module.Scan (disk → modules) and messages.New (empty messages); construct the stores before any orchestrator exists.

Endpoints (outside core):

face (initiator; REPL first) · module (executor; domain truth).
REPL orchestrator (face-side, storeless) — the face’s single two-way door for surface traffic and the surface orchestrator’s mirror: loads outbound, correlates inbound. Owns no truth/cache; holds only a transient parking area (job_token → unclaimed job_updates) that makes async correlation order-independent.

Sources feeding the stores:

disk — module config; read at boot / restart only.
spawn (core) — module identity; bound by possession of the inherited wire when core launches the module (no live kernel read).

Presentation — surfaces project stores

A presentation is a surface: a projection of a store, never a new truth. A surface is declared in surface.json — one subject with its survey (list) and focus (detail) contract — and built by capabilities that return the same sdk/surface DTOs (SurveyResponse / FocusResponse) any face renders.

One store may project several co-equal surfaces (multi-subject); each is an independent presentation, keyed by subject. The manifest stays singular — one process, one identity — while the surfaces array fans out the presentations.

Core declares its own surface.json exactly like a module. The control-plane surfaces are defined once, in core’s embedded surface.json (sdk/system) — the single source of truth shared by every face and by core BIST — never hardcoded per face:

module surface → projects the modules store (system.module.*)
job surface → projects the messages store (system.jobs.*)

A surface carries an optional name/desc so a static surface (core, or a module’s secondary surface) supplies its own display label and one-liner where there is no enclosing manifest to inherit them from.

A “job” is not a stored object — it is the messages store filtered by job_token. A “connections” surface over the identity facet would be a surface, not a new store.

The surface orchestrator drives a surface’s read path. Loading or refreshing a surface goes REPL → surface orchestrator → broker: as the initiator the orchestrator mints the originating id+job_token and serves each read the face asks of it — the survey on load, then one read per distinct column from the face requests after the survey resolves — and the cells fill the face’s projection progressively, not as one upfront composition. The broker originates nothing — it records and routes the complete envelopes it receives. The orchestrator is storeless — single authority over the surface/composition domain, sole writer of nothing — so it never caches and every load is a fresh read. The face holds the materialized projection: a derived framebuffer, wiped and rebuilt on every load, never a cache and never a third store. suctl still has exactly two stores; everything the face renders is projection.

The path is two-way, and symmetric. A surface read is a conversation, not a one-shot fetch: the load may resolve synchronously (the answer rides the same exchange) or asynchronously (the broker returns accepted and the module streams job_updates, the terminal one carrying the result). The surface orchestrator therefore has an inbound half — it pushes every surface-originated job_update onto a stream — and the REPL orchestrator is its face-side mirror that drains that stream and correlates each update back to the load that asked for it. Correlation is order-independent: because the face learns a load’s job_token only on the accepted reply, the REPL orchestrator parks updates that arrive before that registration (keyed by job_token) and replays them the instant the load registers — so a fast async survey whose result beats its own accepted is buffered, never dropped. Neither orchestrator holds truth or a cache; the parking area is the only transient state, discarded once drained. Sync is the degenerate case of async — a synchronous result is an asynchronous one that resolves in the same frame — so one correlate-and-apply path serves both, and a surface need not know which mode a given load took.

Identity

Identity is a facet of the modules store, not a separate store. It records who is on the other end of a core↔module connection, and answers that without credentials — possession of the wire is the identity.

Core creates one private, address-less, bidirectional socketpair per module and hands the child end over at spawn; the child inherits it as SUCTL_BROKER_FD. Possession of that wire is the module’s identity — no credential check, no peer-cred, no payload trust. The kernel, not a records table, guarantees who is on the wire. A socketpair has no name or path, so a sibling has nothing to dial and cannot impersonate; every module↔core link is a private wire, named at birth. A single socketpair carries both directions — core→module invokes and module→core calls/updates — so there is no separate command socket and no shared, dialable socket between core and its modules.

Boundary — only core↔module is policed. A module may create or attach to a socket inside an external service (e.g. odoo, fail2ban) for its domain work. That socket is outside suctl’s boundary — core neither knows nor reads it — so it carries zero identity burden. Identity is only ever the core↔module question. Possession proves who holds the core↔module wire and nothing more; it is not an external- or network-auth mechanism, which is a separate concern.

Windows seam. The possession wire ports cleanly: core creates the equivalent private, pre-connected duplex channel (two bonded anonymous pipes via SysProcAttr.AdditionalInheritedHandles) and hands the child one end, exactly as the Unix socketpair. The trust model is OS-agnostic; only the seam differs. All OS-specific transport bytes — pair creation, handle inheritance, and module-side recovery — live in one audited place, the sdk/channel package (Spawn / Attach / CloseRemote / Inherit); core’s broker, the conformance harness, and the module’s brokerclient all drive the wire through it. Process lifecycle (graceful stop) is a separate seam, not transport: all OS-specific stop logic — SIGTERM on Unix, a CTRL_BREAK console event to the child’s process group on Windows, plus the portable signal set the child waits on — lives in one audited place, the sdk/lifecycle package (Configure / Stop / StopSignals), used by core’s supervisor, the conformance harness, and the daemon/module wait loops.

OS-agnostic scope

Core and SDK are OS-agnostic; domain modules may be OS-bound. The broker, orchestrator, supervisor, and the entire SDK carry no platform branches in business logic. Every platform-specific decision is confined to a single, named, audited seam — split by build tag (*_unix.go / *_windows.go) or isolated in one package — so the same core runs on Unix and Windows without runtime.GOOS scattered through the hot path. Domain modules (mod-os, mod-nginx, mod-certbot) are intentionally OS-bound; the platform gate in internal/module/scan.go makes that explicit — a module whose manifest platform list omits the host GOOS is indexed StateUnavailable with a reason, never silently hidden, so the inventory honestly reflects every installed module regardless of host OS.

Seam inventory. Each row is the only place its OS-specific code lives:

Seam	Location	Unix	Windows
Transport channel	`sdk/channel`	socketpair	bonded inheritable pipes
Process lifecycle (stop)	`sdk/lifecycle`	`SIGTERM`	`CTRL_BREAK` to the child’s group
Path layout	`sdk/paths` (`paths_{unix,windows}.go`)	FHS (`/etc`, `/var`, `/run`, `/usr`)	under `%ProgramData%\suctl`
Install paths	`sdk/paths` (`BinDir` / `SuctlBin` / `PurgeDirs`)	`/usr/local/bin`, separate FHS roots	`%ProgramData%\suctl\bin`, one root
Log destination	`internal/logging` via `paths.LogDir`	`/var/log/suctl`	`%ProgramData%\suctl\log`
Filesystem ownership	`internal/bootstrap/ownership_{unix,windows}.go`	`chown` to owner	no-op (ACL inheritance)
Privilege check	`internal/privilege`	`uid == 0`	elevated token
Runtime probe	`internal/probe` (`probe_linux.go` / `probe_other.go`)	systemd D-Bus + journald	no-op stub

The flow (hot path)

face → (wire) → orchestrator → [identify · authorize · schedule · record] → (wire) → module → (wire: result / job_update) → orchestrator → record → (wire) → face.

The messages orchestrator (broker) is the only sequencer of the hot path; it reads the modules store and writes the messages store. One bidirectional socketpair per module — the inherited wire, possession = identity — carries both directions: core→module invokes and module→core calls/updates. Faces reach the orchestrator over its inbound face listener (in-process today).

Admission (the queue mechanism)

The broker is the sole runtime writer of the messages store: every exchange (invoke / job_status) is recorded by it and only by it (open on request, completed on response). Diagnostics are not exchanges and never enter the ledger: handshake and health ride the module wire directly, and BIST is a reachability probe, not an exchange — module BIST runs over the module’s wire (off the broker), while core BIST dispatches through a non-recording broker probe path, the in-process analogue of the handshake crossing (D71). The queue is not a structure — queued and running are derived from the store: a job is running once its Started fact is stamped (and not yet terminal), queued before that. There is no background poller — the broker’s queue manager (advance()) re-reconciles only at the two moments the gate’s inputs can change:

Message entry — a new actionable exchange (a queued invoke) is recorded. Its footprint may now fit, or it stays queued.
Job terminal state — a done/failed terminal exchange lands, freeing a running job’s footprint. Queued jobs may now fit.

On either trigger advance() walks the queued set oldest-first and asks the gate (pure function over modules(footprint) + messages(running)) yes/no; every yes is promoted by stamping its Started fact — which makes it part of messages(running) for the next question — then a single cond.Broadcast() un-parks the promoted callers so each routes its handler on its own goroutine, outside the lock. A synchronous caller whose job never promotes within the admission timeout is recorded failed (timeout), never hung. No other event admits work, because no other event changes the gate’s inputs. The queue carries no durable state of its own — the recorded exchanges plus the Started facts are the whole truth, and the queue is derived from them on demand.

The stores package

internal/stores owns both stores and their boot-time builder. It is the one place that constructs the modules store (module.Scan over disk) and the empty messages store (messages.New) and hands them to the orchestrators. The builder is the single sanctioned writer that precedes the orchestrators (§Two stores); once core is running, each store has exactly one runtime writer — its orchestrator. The package exposes the two stores as a unit so startup wires them in one step and no caller reaches around it to build a store directly.