Follow-up #6590: /materialize-async still blocks a server executor thread until engine job completes #6609

Context

Came up while reviewing SparkApplication (#6550) against remote materialization (#6590).

Problem

POST /materialize-async returns 202 and runs store.materialize(_force_local=True) in a background executor. That frees the HTTP request coroutine, but the executor thread still runs the engine’s blocking poll.

For SparkApplicationComputeEngine, materialize() creates the ConfigMap + SparkApplication, then calls _wait_for_completion (poll loop up to job_timeout_seconds, default ~1 hour) until the job finishes.

So for a multi-client feature server we still hold a worker thread for the full Spark job. Review feedback on #6550 called this out as a scalability concern (“this should be async”).

What #6590 already solves (correct)

• Clients don’t need local Spark/K8s
• Clients don’t hold a long HTTP connection
• Clients can poll FeatureView state via the registry

What’s still missing for true non-blocking server behavior

On the async path, the server should submit only (create ConfigMap + SparkApplication / kick off the engine) and return 202 — without waiting in-process.

Completion should stay registry-driven (Spark driver pod already calls apply_materialization → AVAILABLE_ONLINE), which the client poller already uses.

Sync /materialize can keep blocking; that’s fine for CLI / single-tenant.

Suggested direction

1. Split submit vs wait on engines that run external jobs (at least spark_application).
2. Have /materialize-async (and /materialize-incremental-async) call submit-only.
3. Design cleanup for ConfigMap/CR when the feature-server process no longer waits (finally: _cleanup today runs after wait).
4. Leave sync POST /materialize behavior unchanged.

Related

• PR: feat: Add remote materialization support via feature server #6590
• PR: feat: Bring Your Own Spark - SparkApplication #6550
• Code: feature_server.py (run_in_executor → materialize(_force_local=True))
• Code: spark_application/compute.py (_wait_for_completion)