Multi-Server Management¶

Stentor's operator UI supports connecting to multiple server instances simultaneously from a single browser session. This is essential for operators managing separate engagements, maintaining staging and production environments, or coordinating across geographic regions.

This page documents the connection management architecture, data aggregation system, and operational patterns for multi-server workflows.

Architecture Overview¶

Each server connection is fully independent: separate authentication tokens, separate WebSocket connections, separate data stores. The UI aggregates data from all connected servers into unified views while preserving server origin metadata.

graph TB
    subgraph Browser ["Operator Browser"]
        UI["Stentor UI"]
        Store["Connection Store<br/><small>Zustand + localStorage</small>"]
    end

    subgraph ServerA ["Server A (Production)"]
        API_A["REST API"]
        WS_A["CockpitHub WS"]
    end

    subgraph ServerB ["Server B (Staging)"]
        API_B["REST API"]
        WS_B["CockpitHub WS"]
    end

    subgraph ServerC ["Server C (Region 2)"]
        API_C["REST API"]
        WS_C["CockpitHub WS"]
    end

    UI --> Store
    Store -.->|"Token A"| API_A
    Store -.->|"Token B"| API_B
    Store -.->|"Token C"| API_C
    UI <-->|"Events"| WS_A
    UI <-->|"Events"| WS_B
    UI <-->|"Events"| WS_C

Connection Management¶

Adding a Server¶

Use the Server Connection Dialog to add a new server. Provide a descriptive name and the server's API URL, then authenticate with your credentials for that server.

Steps:

1. Click the + button in the server switch bar (visible when 2+ servers exist) or open the connection dialog from the settings menu
2. Enter a server name (e.g., "Prod-East", "Staging-Lab", "Client-Acme")
3. Enter the server URL (e.g., https://stentor-prod.example.com/api)
4. The UI validates the URL by probing GET /v1/health with a 10-second timeout
5. Enter your email and password for that server
6. On successful login, the connection is stored and a WebSocket connection is established

Each server is automatically assigned a color badge from an 8-color palette (blue, red, green, amber, violet, pink, cyan, orange). Colors cycle through the palette based on connection count.

Server Profiles¶

Each server connection tracks the following state:

interface ServerConnection {
  id: string            // UUID (auto-generated)
  name: string          // User-assigned label
  url: string           // Base API URL
  status: ServerStatus  // 'connected' | 'disconnected' | 'connecting' | 'error'
  error?: string        // Error message when status is 'error'
  accessToken: string | null  // JWT access token for this server
  user: ServerUser | null     // Authenticated user { id, email }
  wsConnected: boolean  // Whether the CockpitHub WebSocket is active
  lastSeen: number      // Timestamp of last successful interaction
  color: string         // UI badge color (hex, auto-assigned)
}

Status lifecycle:

Connection Lifecycle¶

The full connection flow from adding a server to receiving real-time events:

sequenceDiagram
    participant Op as Operator
    participant UI as Stentor UI
    participant Store as Connection Store
    participant Server as Target Server

    Op->>UI: Add server (name, URL, credentials)
    UI->>Store: addServer(name, url) -> id
    Store->>Store: Assign color from palette
    UI->>Store: setServerStatus(id, 'connecting')

    UI->>Server: GET /v1/health
    Server-->>UI: { status: 'ok', server: 'stentor' }

    UI->>Server: POST /v1/auth/login
    Server-->>UI: { access_token, user }
    UI->>Store: setServerAuth(id, tokens, user)
    Store->>Store: status = 'connected'

    UI->>Server: POST /v1/auth/ws-ticket
    Server-->>UI: { ticket: '...' }
    UI->>Server: WS /v1/cockpit/ws?ticket=...
    Server-->>UI: 101 Switching Protocols
    UI->>Store: setWsStatus(id, true)

    loop Real-time Events
        Server->>UI: SequencedEvent (tagged with serverId)
    end

Persistence: The connection store uses Zustand with localStorage persistence. On page reload:

1. Server profiles are rehydrated from localStorage (key: stentor-connections)
2. Only serializable fields persist: id, name, url, accessToken, color
3. Runtime fields reset to defaults: status='disconnected', wsConnected=false, user=null
4. If the stored accessToken is still valid, the UI reconnects automatically

Switching Active Server¶

The active server determines which server's data is shown in single-server views (e.g., the cockpit shell targets a specific server's beacons).

• The Server Switch Bar appears as a fixed bottom bar when 2+ servers exist
• Click a server button to set it as active (click again to deselect)
• Each button displays:
  • A color dot matching the server's assigned color
  • A WebSocket status indicator (green = connected)
  • The server name (double-click to rename inline)
  • A status border (green = connected, amber = connecting, red = error)
• Right-click a server for context menu options: Rename, Disconnect, Remove, Reconnect

Data Aggregation¶

How It Works¶

The useMultiServerQuery hook is the core aggregation mechanism. It fans out the same query to all connected servers in parallel, then merges the results into a unified list with server origin metadata.

Under the hood, it uses TanStack React Query's useQueries to create one query per connected server. Each query gets a server-prefixed key (e.g., [serverId, 'beacons']) for independent caching and invalidation.

// Example: Fetch beacons from all connected servers
const { data, isLoading, errors } = useMultiServerQuery(
  ['beacons'],
  (apiClient) => apiClient.get('/v1/c2/beacons'),
  { refetchInterval: 5000 }
);

// data is AggregatedItem<Beacon>[]
// Each item carries server origin metadata
data.forEach(item => {
  console.log(`[${item.serverName}] ${item.data.hostname} (${item.data.status})`);
});

AggregatedItem Wrapper¶

Every item returned from a multi-server query is wrapped in an AggregatedItem envelope that identifies which server it came from:

interface AggregatedItem<T> {
  serverId: string     // Server connection UUID
  serverName: string   // User-assigned label (e.g., "Prod-East")
  serverColor: string  // Badge color (hex, e.g., "#3b82f6")
  data: T              // The actual data item from this server
}

This wrapper enables the UI to:

• Render color badges next to items showing their server origin
• Filter views to a single server's data
• Sort by server in aggregated tables
• Route commands to the correct server when acting on an item

Which Views Aggregate¶

Not all pages show multi-server data. Some views are inherently server-specific (e.g., an interactive shell session targets one beacon on one server).

Partial Failure Handling¶

If one server is unreachable or returns an error, data from other servers still displays normally. The multi-server query result provides granular error reporting:

interface MultiServerQueryResult<T> {
  data: AggregatedItem<T>[]     // Items from ALL responding servers
  isLoading: boolean             // True if ANY server query is loading
  isError: boolean               // True if ANY server query has errored
  errors: ServerError[]          // Per-server error details
  serverResults: Map<string, ServerResult<T>>  // Per-server breakdown
}

interface ServerError {
  serverId: string
  serverName: string
  error: Error
}

Automatic retry is handled by React Query's built-in retry mechanism. Failed server queries are retried independently without affecting successful ones.

Server-Specific Operations¶

useServerQuery Hook¶

For views that target a single server, the useServerQuery hook provides server-scoped queries:

// Fetch data from a specific server only
const { data, isLoading } = useServerQuery(
  activeServerId,
  ['beacons', beaconId, 'tasks'],
  (apiClient) => apiClient.get(`/v1/c2/beacons/${beaconId}/tasks`)
);

The hook:

• Prepends serverId to the query key for cache isolation
• Disables the query if serverId is empty/null
• Returns standard React Query result (not aggregated)

API Client Routing¶

Each server connection has a dedicated API client that handles authentication and URL routing. See the REST API Reference for the full endpoint catalog:

// Get a cached API client for a specific server
import { getApiClientForServer } from '@/lib/api-client';

const client = getApiClientForServer(serverId);
const beacons = await client.get('/v1/c2/beacons');

The getApiClientForServer function:

1. Checks a client cache (one ServerApiClient per server ID)
2. If not cached, calls useConnectionStore.getState().connections[serverId] to look up the server's URL and token
3. Creates a ServerApiClient configured with the correct base URL
4. The client automatically attaches the Authorization: Bearer <token> header from the connection store

WebSocket Per-Server¶

Each connected server gets its own independent WebSocket connection via the useServerWebSocket hook:

• URL derivation: The WebSocket URL is built from the server's configured base API URL (not window.location)
• Ticket-based auth: Fetches a fresh ticket from POST /v1/auth/ws-ticket before each connection
• Exponential backoff: Reconnection uses backoff with jitter (initial 1s, max 30s, random jitter up to 500ms)
• Independent lifecycle: Disconnecting one server's WebSocket does not affect others
• Event tagging: All incoming events are tagged with serverId before dispatching to callbacks

interface CockpitCallbacks {
  onBeaconUpdate?: (update: BeaconUpdate & { serverId: string }) => void
  onConsoleLog?: (entry: ConsoleLogEntry & { serverId: string }) => void
  onTaskStarted?: (event: TaskEvent & { serverId: string }) => void
  onTaskCompleted?: (event: TaskEvent & { serverId: string }) => void
  onShellOutput?: (output: ShellOutputEvent & { serverId: string }) => void
  onFileTransferProgress?: (progress: FileTransferProgress & { serverId: string }) => void
  // ... additional event callbacks
}

Every callback receives the event payload augmented with a serverId field, allowing handlers to route events to the correct server context.

Operational Tips¶