Skip to content

Serve SPARQL endpoint when its initial dataset is ready

Decided · 15 August 2026

Context

CI starts sparqld --no-watch over a directory of EARL Turtle reports and then runs several queries against that one dataset. Today, serve_at_with_options loads the complete dataset before it resolves and binds the HTTP addresses, and logs Serving … at … only afterwards. That makes a client infer readiness from a human-readable log line before it can reuse the loaded dataset.

The existing loader builds a fresh store and replaces the shared store only after the complete load finishes (reload_dataset). When watching, the watcher is registered before that initial load so events cannot be missed (DirectoryWatcher::start_with_patterns). The decision must retain those properties while giving CI and long-running clients an endpoint-level startup contract.

flowchart LR CurrentCli["Current CLI start"] --> CurrentLoad["Load complete dataset"] CurrentLoad --> CurrentBind["Bind HTTP listener"] CurrentBind --> CurrentLog["Log Serving"] CurrentLog --> CurrentQueries["Clients query"] DesiredCli["Desired CLI start"] --> DesiredValidate["Validate inputs and addresses"] DesiredValidate --> DesiredBind["Bind HTTP listener"] DesiredBind --> DesiredGate["Root requests wait"] DesiredBind --> DesiredLoad["Load fresh dataset in background"] DesiredLoad --> DesiredPublish["Atomically publish dataset"] DesiredPublish --> DesiredGate DesiredGate --> DesiredQueries["Existing SPARQL dispatch"]

Decision

Startup alternatives compared against the same directory-backed endpoint.
Criterion 2. Bind; wait at ordinary requests 3. Bind; return 503 + readiness route 1. Load before binding 4. Load before binding + ready signal 5. Batch multi-query mode
Parse shared dataset once ✅ One initial load serves every request. ✅ One initial load serves every request. ✅ One loaded store serves later requests. ✅ One loaded store serves later requests. ✅ Could reuse one load within a new batch interface.
Correct results while loading ✅ Dispatch follows atomic first publication. ✅ 503 explicitly declines service while unavailable. ✅ No request can arrive before binding. ✅ No request can arrive before binding. ✅ A process-local batch could query after its load.
CI and deployment ergonomics ✅ The first normal query is the readiness wait. ⚠ Requires a client retry or polling policy after 503. ❌ Clients need an out-of-band availability check. ⚠ Requires defining a separate process-signal contract. ❌ Does not start the existing HTTP service.
Health, readiness, and liveness ⚠ Readiness is expressed by the normal endpoint, not a probe route. ✅ A readiness probe can remove an unready service from traffic. ⚠ A listening port implies only post-load availability. ⚠ A signal needs its own health-semantics contract. ❌ Provides no long-running endpoint for probes.
Timeout, cancellation, and failure ✅ Each client keeps its HTTP deadline; shared loading remains independent. ✅ 503 communicates temporary unavailability. ⚠ Connection failures reveal no initialization outcome. ⚠ Signal consumers need a defined failure and timeout protocol. ⚠ A separate input protocol must define batch cancellation and errors.
Live reload and atomic replacement ✅ Retains pre-load watch registration and atomic replacement. ✅ Can retain the same watcher and replacement model. ✅ Current model already preserves both. ✅ Current model already preserves both. ❌ A one-shot mode cannot supply live reload.
Implementation complexity and API stability ✅ Adds one internal readiness gate while retaining /. ⚠ Adds endpoint and retry contract beyond the existing / API. ✅ Preserves the current startup order. ⚠ Adds a separate readiness-output contract. ⚠ Adds a second query-supply interface.
Outcome ✅ Chosen Not selected ❌ Excluded — preserves log-coupled startup ❌ Excluded — adds out-of-band coordination ❌ Excluded — does not improve server startup

Green column: chosen. Yellow column: satisfies the core constraints but was not selected. Red columns: excluded; stronger-red cells identify the decisive shortcoming. ⚠ denotes a trade-off. The HTTP SPARQL Protocol defines query request forms, while Kubernetes probes distinguish traffic readiness from liveness; sparqld deliberately adds neither /readyz nor /livez, so both continue to be ordinary unknown paths (404) under the root-only handler.

Bind the listener immediately after synchronous validation of the directory, patterns, and resolved addresses. Start the initial load in the background. Until that load reaches a terminal state, every GET or POST request at /, including the landing GET /, waits without a server-imposed deadline. It is then passed to the existing root-only request dispatch.

On success, atomically publish the fresh dataset, release all waiting requests, and run their normal dispatch. Per-file parse errors remain nonfatal: the catalog already records them, so that completed load is ready. A client timeout or disconnect cancels only that request, never the shared load or another waiter. On a fatal initialization failure, pending and subsequent root requests receive 503 Service Unavailable, then sparqld shuts down and exits nonzero without a successful Serving log. Other paths return 404 immediately. After readiness, live reload never re-gates requests and retains staged atomic replacement.

Logs remain informative for people and diagnostics, not a client readiness contract. The existing Serving … at … line remains after readiness for compatibility.

Consequences

  • Clients can issue their first ordinary query immediately after starting sparqld, with a client-side timeout appropriate to their CI or deployment.
  • The CLI flags, library function signatures, SPARQL request forms, responses, and non-root 404 behavior remain stable; only the startup timing changes.
  • Release notes document that log parsing is unnecessary and that an immediate root request waits for the initial dataset rather than inferring readiness from the listener or logs.

Implementation Steps

  • Add an internal, shared initial-load state that wakes all root-request waiters after success or fatal failure.
  • Bind only after validation, begin the initial load in the background, and preserve watcher registration before the first load.
  • Dispatch root requests only after successful first publication; return 503 after fatal initialization and stop the listener before nonzero exit.
  • Document the startup contract in the HTTP API and CLI guidance without introducing a health endpoint.
  • Test listener reachability during load, concurrent GET and POST waiters, atomic complete-dataset results, nonfatal file errors, independent client cancellation, fatal-load 503 and exit, absent probe paths, and post-ready atomic live reload.