fix(web): harden managed config sync between console and clients (#2567)

* fix(web): fence managed config runtime reconciliation Keep runtime reconciliation tied to the currently authorized session so stale connections cannot mutate a replacement session runtime. Accumulate only contiguous dirty IDs and load their latest SQLite state. Require the applied revision to match the earliest Patch base and the persisted revision to match the latest target. Otherwise, reconcile the full desired state. Use separate runtime-state and config-cache epochs. Managed updates can reuse observed configs; direct mutations invalidate them. Update sync documentation to match. * fix(web): interrupt validation retry on state changes Track meaningful validation state changes separately from periodic dirty signals. Applied revision changes wake a failed validation immediately, while heartbeat-driven revalidation retains the retry backoff. Treat Notify as a wake-up hint and recheck the state-change epoch after every wake so stored permits and periodic heartbeats cannot cause retry storms. * fix(web): retry unconfirmed connected webhooks Retry node-connected webhook delivery on retryable errors with a short 100ms/500ms backoff and give up immediately on non-retryable errors. Re-check that the session still owns the connection before every attempt and before recording the delivery, so a replaced session can no longer record a stale connected binding. * fix(web): fence disconnects by session ownership Return whether session removal actually removed the current route owner, and emit disconnected only for that owner. Replaced sessions can no longer invalidate a newer connected route. * fix(web): hot-patch managed hostnames Include hostname changes in the hot-patch path instead of falling back to a full restart. When a full overwrite run is required and the desired config has no hostname, inherit the current runtime hostname so an unmanaged value survives until it is explicitly cleared. Read back the runtime config after an overwrite run and verify it converged instead of assuming the desired state was applied. * fix(web): retry transient runtime reconciliation failures Keep the per-session managed runtime reconciliation worker alive when a single database round fails. Retry from the next heartbeat so persisted managed revisions can still converge after restart-time contention. Reserve terminal worker shutdown for destroyed session or storage state, and cover recovery after a transient revision read failure. * fix(web): accept omitted hostname after runtime apply Release 2.6.4 omits hostname from config readback when it matches the device hostname. Trust a successful hostname mutation only when the returned field is absent, while continuing to verify every other field and rejecting explicit mismatches. * fix(web): ignore unmanaged runtime device names Windows release 2.6.4 generates a random interface name when the managed config leaves dev_name empty. Exclude that runtime-owned value from reconciliation unless the desired config explicitly sets a non-empty device name, preventing endless overwrite restarts. * feat(web): report failed network instances to console Expose stopped Core instances with startup errors in heartbeats. Merge Core failures with direct managed-run RPC failures in easytier-web. Send failed instance IDs during token validation without error text. Prune local run failures when managed configs are deleted. * fix(web): distinguish unknown runtime application state Track whether the current session has observed its applied revision separately from the optional revision value. Report this fact through validate-token so Console can preserve application state across receiver restarts while recognizing deliberate pending mutations. * feat(web): configure heartbeat timing from server Heartbeat responses now provide the interval and RPC timeout. Legacy servers use local defaults and remote values are clamped. Web configuration and session receive timeout follow the policy. * fix(web): reject inactive control sessions Route control RPCs by machine id only to sessions whose RPC manager is still running, so a session that has been stopped or replaced can no longer receive control traffic addressed to the device. * fix(core): filter network info before collection When a collect-network-info request names specific instances, collect those instances only instead of collecting every instance and filtering the result afterwards, so unrequested instances no longer run per-collection work on every request. * feat(web): enable focused runtime diagnostics Enable easytier-web info logs by default while preserving explicit log configuration. Record startup settings, session lifecycle, failed instance changes, webhook queue and request latency, and managed runtime operation timings for production diagnosis. * fix(web): preserve managed revision across reconnects Keep one runtime identifier for each Core WebClient lifetime. Reuse its managed runtime state after transport reconnects. Retain applied revisions and reconcile hints while disconnected. Preserve runtime epochs so stale work cannot mark a revision applied. Reject stale sessions from reclaiming routes after reconnect. Core or Web restarts and legacy clients still use unknown state. Immediately revalidate a restored revision after authentication. Document local management RPC drift as an accepted trade-off. This lets Console converge without waiting for periodic validation. * fix(web): satisfy clippy across managed config sync tests Scope managed runtime guards to blocks in runtime revision tests so no std MutexGuard is held across await points, return the applied revision directly instead of through a let binding, and pass WebhookValidationInput to request_heartbeat_validation instead of expanding it into eight separate arguments. * fix(core): stop reporting failed instances as running in heartbeats A stopped instance with a startup error appeared in both running_network_instances and failed_network_instances, so the server treated it as running and never re-ran its managed config. Exclude failed instance ids when building the running list so the reconciler restarts them. * fix(core): close missed-wakeup race in instance state changes wait_for_change created the Notified future before reading the generation but only registered it when awaited. A change landing in between fired notify_waiters with no registered waiter and delayed the heartbeat by a full interval. Enable the future before reading the generation so every change wakes a waiting heartbeat. * fix(web): address review findings Fence webhook validation and connection transitions against stale state, redact credentials from default-level logs, and stabilize runtime reconciliation: - Record connected bindings only while the session still owns the machine route, and skip disconnect compensation once a replacement owns the route so a stale disconnect cannot revoke it. - Discard webhook validation results when the change epoch moved during the HTTP round, so a stale rejection cannot invalidate the current session. - Drop user_token fields from info and warn logs that became visible with info-level defaults. - Restore a hostname omitted by the 2.6.4 readback into the cached runtime config after a successful mutation, so later rounds stop re-sending the same hostname patch. - Reconcile running web configs when no revision is tracked so legacy unrevisioned updates converge, and wake sessions for unrevisioned full updates instead of waiting for the next heartbeat. * chore(go): regenerate web proto bindings for heartbeat fields Add failed_network_instances, support_heartbeat_policy, and the heartbeat policy response fields to the checked-in Go bindings. Other proto packages are left as-is because their drift predates this change. * fix(web): redact user tokens from positional log arguments Three runtime reconciliation info logs and the user lookup error contexts printed user_token through format arguments, which the earlier field-syntax redaction missed. The reconcile log now fires every round for unrevisioned machines, so remove the token from these messages as well. * fix(web): fence stale validation and runtime reconcile rounds Check webhook validation epochs while holding the session write lock, so stale success and rejection responses cannot change session state. Advance the runtime epoch for unrevisioned full config updates, and exclude failed instances from heartbeat and RPC reconciliation lists so stopped instances are restarted instead of repeatedly hot-patched. Release test read guards before awaiting validation apply calls. Set up the no-pending condition before asserting that an applied revision is a no-op, and verify that its runtime epoch remains unchanged. Validation: all 137 client_manager tests passed. * test(credentials): cover P2P with active VPN portal Model an admin and temporary credential peer connected as a foreign network through a public server with data relay disabled. Verify their direct connection can be replaced after a WireGuard portal client comes online. * test(credentials): stabilize two-admins failover assertions The two-admins non-reusable credential test could fail on slow convergence: after dropping the winning peer it relied on a single route sample passing a bare AND condition, then re-asserted the same expectations through one-shot checks seconds later. A transient route flap in that window (for example a briefly resurrected winner route from stale conn info) turned a passing convergence into a hard assert failure. This matches the 48.9s CI flake of credential_non_reusable_across_two_admins_allows_only_one_peer observed on 2026-08-12. Changes: - wait for bidirectional admin connectivity (AND) with a 20s budget before issuing the credential, instead of a one-directional OR - replace the failover wait_for_condition with wait_stable_failover_visibility_on_admins, which requires three consecutive samples of loser-present and winner-absent on both admins within the same 60s budget and logs every sample - enrich the stable-single-winner timeout message with per-admin visibility flags and elapsed time for triage All existing contracts are preserved; only observation windows and diagnostics change. Validated in the rust container: three passes at normal speed (54.1s / 53.8s / 53.1s) plus one slow-convergence round (172.7s) that would have raced the old one-shot sampling; it now passes with failover samples logged. cargo fmt and clippy -D warnings clean.
2026-10-14 08:26:14 -08:00 · 2026-09-13 01:13:28 +08:00
1 parent e0bdb516b6
commit c96b6c1961
@@ -15,7 +15,7 @@ use dashmap::DashMap;
 use uuid::Uuid;

 use crate::config::toml::TomlConfig;
-use crate::instance::{CoreInstance, CoreInstanceHost};
+use crate::instance::{CoreInstance, CoreInstanceHost, CoreInstanceState};
 use crate::process_runtime::CoreProcessRuntime;
 #[cfg(feature = "web-client")]
 use crate::{
@@ -205,6 +205,39 @@ struct ActiveStopGuard {
    notifier: Arc<tokio::sync::Notify>,
 }

+#[derive(Default)]
+struct InstanceStateChanges {
+    generation: AtomicUsize,
+    notify: tokio::sync::Notify,
+}
+
+impl InstanceStateChanges {
+    fn generation(&self) -> usize {
+        self.generation.load(Ordering::Acquire)
+    }
+
+    fn mark_changed(&self) {
+        self.generation.fetch_add(1, Ordering::AcqRel);
+        self.notify.notify_waiters();
+    }
+
+    async fn wait_for_change(&self, generation: usize) -> usize {
+        loop {
+            let notified = self.notify.notified();
+            tokio::pin!(notified);
+            // Register the waiter before reading the generation: notify_waiters
+            // does not retain permits, so a change landing between the read
+            // and the await would otherwise be missed until the next change.
+            notified.as_mut().enable();
+            let current = self.generation();
+            if current != generation {
+                return current;
+            }
+            notified.await;
+        }
+    }
+}
+
 impl Drop for ActiveStopGuard {
    fn drop(&mut self) {
        let previous = self.active_stops.fetch_sub(1, Ordering::AcqRel);
@@ -224,6 +257,7 @@ pub struct InstanceManager<F: InstanceFactory> {
    mutation_lock: Arc<tokio::sync::Mutex<()>>,
    runtime_handle: Option<tokio::runtime::Handle>,
    active_stops: Arc<AtomicUsize>,
+    instance_state_changes: Arc<InstanceStateChanges>,
 }

 impl<F: InstanceFactory> InstanceManager<F> {
@@ -238,6 +272,7 @@ impl<F: InstanceFactory> InstanceManager<F> {
            mutation_lock: Arc::new(tokio::sync::Mutex::new(())),
            runtime_handle,
            active_stops: Arc::new(AtomicUsize::new(0)),
+            instance_state_changes: Arc::new(InstanceStateChanges::default()),
        }
    }

@@ -311,6 +346,16 @@ impl<F: InstanceFactory> InstanceManager<F> {
            notifier: self.notifier.clone(),
        }
    }
+
+    pub(crate) fn instance_state_generation(&self) -> usize {
+        self.instance_state_changes.generation()
+    }
+
+    pub(crate) async fn wait_for_instance_state_change(&self, generation: usize) -> usize {
+        self.instance_state_changes
+            .wait_for_change(generation)
+            .await
+    }
 }

 impl<F: ProcessRuntimeProvider> InstanceManager<F> {
@@ -339,10 +384,12 @@ where
        let instance_id = instance.instance_id();
        self.config_controls.insert(instance_id, control);
        let notifier = self.notifier.clone();
+        let instance_state_changes = self.instance_state_changes.clone();
        runtime.spawn(async move {
            if let Err(error) = instance.start().await {
                tracing::error!(%error, %instance_id, "instance failed to start");
            }
+            instance_state_changes.mark_changed();
            notifier.notify_one();
        });
        Ok(instance_id)
@@ -373,6 +420,7 @@ where
            drop(active_stop);
            return Ok(self.instance_ids());
        }
+        self.instance_state_changes.mark_changed();

        runtime
            .spawn(async move {
@@ -403,6 +451,19 @@ where
            .collect()
    }

+    pub fn failed_instance_ids(&self) -> Vec<Uuid> {
+        self.list()
+            .into_iter()
+            .filter(|instance| {
+                instance.state() == CoreInstanceState::Stopped
+                    && instance
+                        .latest_error()
+                        .is_some_and(|error| !error.trim().is_empty())
+            })
+            .map(|instance| instance.instance_id())
+            .collect()
+    }
+
    pub fn instance(&self, instance_id: Uuid) -> Option<Arc<CoreInstance<H>>> {
        self.get(instance_id)
    }
@@ -533,6 +594,39 @@ where
    }
 }

+#[cfg(test)]
+mod instance_state_change_tests {
+    use super::InstanceStateChanges;
+    use std::{sync::Arc, time::Duration};
+
+    #[tokio::test]
+    async fn state_change_wait_observes_existing_and_future_changes() {
+        let changes = Arc::new(InstanceStateChanges::default());
+
+        let observed = changes.generation();
+        changes.mark_changed();
+        assert_eq!(
+            tokio::time::timeout(Duration::from_secs(1), changes.wait_for_change(observed))
+                .await
+                .expect("an existing change must not be lost"),
+            1
+        );
+
+        let observed = changes.generation();
+        let waiter_changes = changes.clone();
+        let waiter = tokio::spawn(async move { waiter_changes.wait_for_change(observed).await });
+        tokio::task::yield_now().await;
+        changes.mark_changed();
+        assert_eq!(
+            tokio::time::timeout(Duration::from_secs(1), waiter)
+                .await
+                .expect("a future change must wake the waiter")
+                .unwrap(),
+            2
+        );
+    }
+}
+
 #[cfg(test)]
 mod tests {
    use std::{
@@ -1395,6 +1395,74 @@ virtual_ip = "10.82.0.2/24"
        );
    }

+    #[tokio::test]
+    async fn manager_reports_only_stopped_instances_with_errors() {
+        use crate::instance::manager::{InstanceFactory, InstanceManager};
+
+        struct StateTestFactory;
+
+        impl InstanceFactory for StateTestFactory {
+            type Instance = CoreInstance<TestHost>;
+            type CreateContext = ();
+            type Error = anyhow::Error;
+
+            fn create(
+                &self,
+                config: TomlConfig,
+                (): Self::CreateContext,
+            ) -> Result<Arc<Self::Instance>, Self::Error> {
+                let (packet_sink, _packet_receiver) = tokio::sync::mpsc::channel(16);
+                CoreInstance::from_toml(config, adapters(None, Arc::new(packet_sink)))
+            }
+        }
+
+        fn create_instance(
+            manager: &InstanceManager<StateTestFactory>,
+            name: &str,
+        ) -> Arc<CoreInstance<TestHost>> {
+            let config = TomlConfig::new_from_str(&format!("instance_name = \"{name}\"")).unwrap();
+            manager.create(config, ()).unwrap()
+        }
+
+        let manager = InstanceManager::new(StateTestFactory, None);
+        let running = create_instance(&manager, "running");
+        running
+            .latest_error
+            .write()
+            .replace("old startup error".to_owned());
+        running.set_state(CoreInstanceState::Running);
+
+        let starting = create_instance(&manager, "starting");
+        starting
+            .latest_error
+            .write()
+            .replace("old startup error".to_owned());
+        starting.set_state(CoreInstanceState::Starting);
+
+        let stopped_without_error = create_instance(&manager, "stopped-without-error");
+        stopped_without_error.set_state(CoreInstanceState::Stopped);
+
+        let stopped_with_blank_error = create_instance(&manager, "stopped-with-blank-error");
+        stopped_with_blank_error
+            .latest_error
+            .write()
+            .replace("  \n".to_owned());
+        stopped_with_blank_error.set_state(CoreInstanceState::Stopped);
+
+        let failed = create_instance(&manager, "failed");
+        failed
+            .latest_error
+            .write()
+            .replace("startup failed".to_owned());
+        failed.set_state(CoreInstanceState::Stopped);
+        let failed_id = failed.instance_id();
+
+        assert_eq!(manager.failed_instance_ids(), vec![failed_id]);
+
+        manager.delete_network_instances([failed_id]).await.unwrap();
+        assert!(manager.failed_instance_ids().is_empty());
+    }
+
    #[tokio::test]
    async fn aborting_host_prepare_runs_unified_cleanup() {
        #[derive(Default)]
@@ -1739,6 +1807,142 @@ virtual_ip = "10.82.0.2/24"
        assert!(instances.instances().is_empty());
    }

+    #[cfg(feature = "web-client")]
+    #[tokio::test]
+    async fn process_management_rpc_collects_only_requested_instances() {
+        use std::{collections::VecDeque, sync::Mutex as StdMutex};
+
+        use crate::{
+            config::toml::TomlConfig,
+            instance::manager::InstanceFactory,
+            management::{InstanceManager, ProcessManagementRpc, UnsupportedConfigFileStorage},
+        };
+        use easytier_proto::{
+            api::manage::{CollectNetworkInfoRequest, WebClientService},
+            rpc_types::controller::BaseController,
+        };
+
+        #[derive(Default)]
+        struct RecordingRuntimeHost {
+            collection_count: AtomicUsize,
+        }
+
+        #[async_trait]
+        impl InstanceRuntimeHost for RecordingRuntimeHost {
+            async fn prepare(
+                &self,
+                _packet_plane: Arc<CorePacketPlane>,
+            ) -> anyhow::Result<Option<Arc<dyn DhcpIpv4Host>>> {
+                Ok(None)
+            }
+
+            async fn shutdown(&self) {}
+
+            fn management_events(&self) -> Vec<String> {
+                self.collection_count.fetch_add(1, Ordering::Relaxed);
+                Vec::new()
+            }
+        }
+
+        struct RecordingFactory {
+            process_runtime: Arc<CoreProcessRuntime>,
+            runtime_hosts: StdMutex<VecDeque<Arc<RecordingRuntimeHost>>>,
+        }
+
+        impl InstanceFactory for RecordingFactory {
+            type Instance = CoreInstance<TestHost>;
+            type CreateContext = ();
+            type Error = anyhow::Error;
+
+            fn create(
+                &self,
+                config: TomlConfig,
+                (): Self::CreateContext,
+            ) -> Result<Arc<Self::Instance>, Self::Error> {
+                let runtime_host = self.runtime_hosts.lock().unwrap().pop_front().unwrap();
+                let (packet_sink, _packet_receiver) = tokio::sync::mpsc::channel(16);
+                let mut adapters = adapters_with_process_runtime(
+                    None,
+                    Arc::new(packet_sink),
+                    self.process_runtime.clone(),
+                );
+                adapters.instance_runtime = runtime_host;
+                CoreInstance::from_toml(config, adapters)
+            }
+        }
+
+        let requested_runtime = Arc::new(RecordingRuntimeHost::default());
+        let unrequested_runtime = Arc::new(RecordingRuntimeHost::default());
+        let instances = Arc::new(InstanceManager::new(
+            RecordingFactory {
+                process_runtime: CoreProcessRuntime::new(),
+                runtime_hosts: StdMutex::new(VecDeque::from([
+                    requested_runtime.clone(),
+                    unrequested_runtime.clone(),
+                ])),
+            },
+            Some(tokio::runtime::Handle::current()),
+        ));
+        let requested_id = uuid::Uuid::new_v4();
+        let unrequested_id = uuid::Uuid::new_v4();
+        for instance_id in [requested_id, unrequested_id] {
+            let config = TomlConfig::default();
+            config.set_id(instance_id);
+            config.set_listeners(Vec::new());
+            instances
+                .create(config, ())
+                .unwrap()
+                .set_state(CoreInstanceState::Running);
+        }
+        let rpc = ProcessManagementRpc::<RecordingFactory>::new(
+            instances,
+            Arc::new(()),
+            Arc::new(UnsupportedConfigFileStorage),
+        );
+
+        let response = rpc
+            .collect_network_info(
+                BaseController::default(),
+                CollectNetworkInfoRequest {
+                    inst_ids: vec![
+                        requested_id.into(),
+                        requested_id.into(),
+                        uuid::Uuid::new_v4().into(),
+                    ],
+                },
+            )
+            .await
+            .unwrap();
+        let info = response.info.unwrap().map;
+        assert_eq!(info.len(), 1);
+        assert!(info.contains_key(&requested_id.to_string()));
+        assert_eq!(
+            requested_runtime.collection_count.load(Ordering::Relaxed),
+            1
+        );
+        assert_eq!(
+            unrequested_runtime.collection_count.load(Ordering::Relaxed),
+            0
+        );
+
+        let response = rpc
+            .collect_network_info(
+                BaseController::default(),
+                CollectNetworkInfoRequest::default(),
+            )
+            .await
+            .unwrap();
+        assert_eq!(response.info.unwrap().map.len(), 2);
+        assert_eq!(
+            requested_runtime.collection_count.load(Ordering::Relaxed),
+            2
+        );
+        assert_eq!(
+            unrequested_runtime.collection_count.load(Ordering::Relaxed),
+            1
+        );
+    }
+
    #[cfg(feature = "management")]
    #[tokio::test]
    async fn owned_selection_and_cleanup_share_the_canonical_transaction() {
@@ -28,7 +28,7 @@ use crate::{

 use super::{
    ConfigFileControl, ConfigFilePermission, InstanceManager, config_source_from_rpc,
-    config_source_to_rpc,
+    config_source_to_rpc, network_instance_running_info,
 };

 #[async_trait::async_trait]
@@ -633,17 +633,29 @@ where
        let included = request
            .inst_ids
            .into_iter()
-            .map(|id| uuid::Uuid::from(id).to_string())
+            .map(uuid::Uuid::from)
            .collect::<HashSet<_>>();
-        let map = self
-            .management
-            .instances
-            .collect_network_infos()
-            .await?
-            .into_iter()
-            .map(|(id, info)| (id.to_string(), info))
-            .filter(|(id, _)| included.is_empty() || included.contains(id))
-            .collect();
+        let map = if included.is_empty() {
+            self.management
+                .instances
+                .collect_network_infos()
+                .await?
+                .into_iter()
+                .map(|(id, info)| (id.to_string(), info))
+                .collect()
+        } else {
+            let mut map = std::collections::BTreeMap::new();
+            for instance_id in included {
+                let Some(instance) = self.management.instances.instance(instance_id) else {
+                    continue;
+                };
+                map.insert(
+                    instance_id.to_string(),
+                    network_instance_running_info(instance.as_ref()).await?,
+                );
+            }
+            map
+        };
        Ok(CollectNetworkInfoResponse {
            info: Some(NetworkInstanceRunningInfoMap { map }),
        })
@@ -1,3 +1,4 @@
+use std::collections::HashSet;
 use std::sync::{
    Arc, Weak,
    atomic::{AtomicBool, Ordering},
@@ -7,7 +8,7 @@ use async_trait::async_trait;
 use easytier_proto::{
    rpc_types::controller::BaseController,
    web::{
-        DeviceOsInfo, GetFeatureRequest, GetFeatureResponse, HeartbeatRequest,
+        DeviceOsInfo, GetFeatureRequest, GetFeatureResponse, HeartbeatRequest, HeartbeatResponse,
        WebServerServiceClientFactory,
    },
 };
@@ -33,6 +34,64 @@ const RETRY_INTERVAL: std::time::Duration = std::time::Duration::from_secs(1);
 // Keep retry ownership in this loop when transport or protocol handshakes stall.
 const CONNECT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(20);
 const FEATURE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(3);
+const DEFAULT_HEARTBEAT_INTERVAL_MS: u32 = 3_500;
+const DEFAULT_HEARTBEAT_TIMEOUT_MS: u32 = 15_000;
+const MIN_HEARTBEAT_INTERVAL_MS: u32 = 1_000;
+const MAX_HEARTBEAT_INTERVAL_MS: u32 = 60_000;
+const MIN_HEARTBEAT_TIMEOUT_MS: u32 = 5_000;
+const MAX_HEARTBEAT_TIMEOUT_MS: u32 = 120_000;
+const MIN_HEARTBEAT_TIMEOUT_MARGIN_MS: u32 = 5_000;
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+struct HeartbeatPolicy {
+    interval: std::time::Duration,
+    timeout_ms: i32,
+}
+
+impl Default for HeartbeatPolicy {
+    fn default() -> Self {
+        Self {
+            interval: std::time::Duration::from_millis(DEFAULT_HEARTBEAT_INTERVAL_MS.into()),
+            timeout_ms: DEFAULT_HEARTBEAT_TIMEOUT_MS as i32,
+        }
+    }
+}
+
+impl HeartbeatPolicy {
+    fn from_response(response: &HeartbeatResponse) -> (Self, bool) {
+        let requested_interval = response
+            .heartbeat_interval_ms
+            .unwrap_or(DEFAULT_HEARTBEAT_INTERVAL_MS);
+        let requested_timeout = response
+            .heartbeat_timeout_ms
+            .unwrap_or(DEFAULT_HEARTBEAT_TIMEOUT_MS);
+        let interval_ms =
+            requested_interval.clamp(MIN_HEARTBEAT_INTERVAL_MS, MAX_HEARTBEAT_INTERVAL_MS);
+        let timeout_ms = requested_timeout
+            .clamp(MIN_HEARTBEAT_TIMEOUT_MS, MAX_HEARTBEAT_TIMEOUT_MS)
+            .max(interval_ms.saturating_add(MIN_HEARTBEAT_TIMEOUT_MARGIN_MS));
+        (
+            Self {
+                interval: std::time::Duration::from_millis(interval_ms.into()),
+                timeout_ms: timeout_ms as i32,
+            },
+            interval_ms != requested_interval || timeout_ms != requested_timeout,
+        )
+    }
+
+    fn controller(self) -> BaseController {
+        BaseController {
+            timeout_ms: self.timeout_ms,
+            ..Default::default()
+        }
+    }
+
+    fn remaining_interval(self, elapsed: std::time::Duration) -> Option<std::time::Duration> {
+        self.interval
+            .checked_sub(elapsed)
+            .filter(|delay| !delay.is_zero())
+    }
+}

 async fn connect_config_server(
    connector: &dyn TunnelDialer,
@@ -100,6 +159,16 @@ pub(crate) trait WebClientBackend: Send + Sync + 'static {
    fn register(&self, registry: &ServiceRegistry);

    async fn instance_ids(&self) -> anyhow::Result<Vec<uuid::Uuid>>;
+
+    fn failed_instance_ids(&self) -> Vec<uuid::Uuid>;
+
+    fn instance_state_generation(&self) -> usize {
+        0
+    }
+
+    async fn wait_for_instance_state_change(&self, _generation: usize) -> usize {
+        std::future::pending().await
+    }
 }

 struct NativeWebClientBackend<F>
@@ -141,11 +210,26 @@ where
    async fn instance_ids(&self) -> anyhow::Result<Vec<uuid::Uuid>> {
        Ok(self.instances.instance_ids())
    }
+
+    fn failed_instance_ids(&self) -> Vec<uuid::Uuid> {
+        self.instances.failed_instance_ids()
+    }
+
+    fn instance_state_generation(&self) -> usize {
+        self.instances.instance_state_generation()
+    }
+
+    async fn wait_for_instance_state_change(&self, generation: usize) -> usize {
+        self.instances
+            .wait_for_instance_state_change(generation)
+            .await
+    }
 }

 struct WebClientController {
    config: WebClientConfig,
    backend: Arc<dyn WebClientBackend>,
+    runtime_id: uuid::Uuid,
 }

 /// Portable config-server client. Hosts only supply identity and adapters.
@@ -201,7 +285,11 @@ impl<F> WebClient<F> {
        backend: Arc<dyn WebClientBackend>,
        manager_guard: Option<DaemonGuard>,
    ) -> Self {
-        let controller = Arc::new(WebClientController { config, backend });
+        let controller = Arc::new(WebClientController {
+            config,
+            backend,
+            runtime_id: uuid::Uuid::new_v4(),
+        });
        let connected = Arc::new(AtomicBool::new(false));
        let tasks = AbortOnDropHandle::new(tokio::spawn(web_client_routine(
            controller.clone(),
@@ -312,6 +400,59 @@ struct WebClientSession {
    tasks: Mutex<JoinSet<()>>,
 }

+fn running_instances_for_heartbeat(
+    instance_ids: Vec<uuid::Uuid>,
+    failed_instance_ids: &[uuid::Uuid],
+) -> Vec<uuid::Uuid> {
+    let failed_instance_ids: HashSet<_> = failed_instance_ids.iter().copied().collect();
+    instance_ids
+        .into_iter()
+        .filter(|instance_id| !failed_instance_ids.contains(instance_id))
+        .collect()
+}
+
+fn build_heartbeat_request(
+    config: &WebClientConfig,
+    runtime_id: uuid::Uuid,
+    running_network_instances: Vec<uuid::Uuid>,
+    failed_network_instances: Vec<uuid::Uuid>,
+) -> HeartbeatRequest {
+    HeartbeatRequest {
+        machine_id: Some(config.machine_id.into()),
+        inst_id: Some(runtime_id.into()),
+        user_token: config.token.clone(),
+        easytier_version: config.easytier_version.clone(),
+        hostname: config.hostname.clone(),
+        report_time: chrono::Local::now().to_rfc3339(),
+        device_os: Some(config.device_os.clone()),
+        support_config_source: true,
+        running_network_instances: running_network_instances
+            .into_iter()
+            .map(Into::into)
+            .collect(),
+        failed_network_instances: failed_network_instances
+            .into_iter()
+            .map(Into::into)
+            .collect(),
+        support_heartbeat_policy: true,
+    }
+}
+
+async fn wait_for_next_heartbeat(
+    backend: &dyn WebClientBackend,
+    observed_generation: usize,
+    policy: HeartbeatPolicy,
+    elapsed: std::time::Duration,
+) {
+    let Some(delay) = policy.remaining_interval(elapsed) else {
+        return;
+    };
+    tokio::select! {
+        _ = time::sleep(delay) => {}
+        _ = backend.wait_for_instance_state_change(observed_generation) => {}
+    }
+}
+
 impl WebClientSession {
    fn new(tunnel: Box<dyn Tunnel>, controller: Arc<WebClientController>) -> Self {
        let rpc = BidirectRpcManager::new();
@@ -339,46 +480,60 @@ impl WebClientSession {
        tasks: &mut JoinSet<()>,
    ) {
        let controller = controller.upgrade().expect("web client controller");
-        let machine_id = controller.config.machine_id;
-        let session_id = uuid::Uuid::new_v4();
-        let token = controller.config.token.clone();
-        let hostname = controller.config.hostname.clone();
-        let device_os = controller.config.device_os.clone();
-        let easytier_version = controller.config.easytier_version.clone();
        let controller = Arc::downgrade(&controller);
        let client = rpc
            .rpc_client()
            .scoped_client::<WebServerServiceClientFactory<BaseController>>(1, 1, String::new());
-        let mut tick = time::interval(std::time::Duration::from_secs(1));

        tasks.spawn(async move {
+            let mut heartbeat_policy = HeartbeatPolicy::default();
            loop {
-                tick.tick().await;
+                let heartbeat_started_at = std::time::Instant::now();
                let Some(controller) = controller.upgrade() else {
                    break;
                };
+                let observed_generation = controller.backend.instance_state_generation();
+                let failed_network_instances = controller.backend.failed_instance_ids();
                let running_network_instances = match controller.backend.instance_ids().await {
-                    Ok(instance_ids) => instance_ids.into_iter().map(Into::into).collect(),
+                    Ok(instance_ids) => {
+                        running_instances_for_heartbeat(instance_ids, &failed_network_instances)
+                    }
                    Err(error) => {
                        tracing::error!(%error, "failed to list config-server instances");
                        break;
                    }
                };
-                let request = HeartbeatRequest {
-                    machine_id: Some(machine_id.into()),
-                    inst_id: Some(session_id.into()),
-                    user_token: token.clone(),
-                    easytier_version: easytier_version.clone(),
-                    hostname: hostname.clone(),
-                    report_time: chrono::Local::now().to_rfc3339(),
-                    device_os: Some(device_os.clone()),
-                    support_config_source: true,
+                let request = build_heartbeat_request(
+                    &controller.config,
+                    controller.runtime_id,
                    running_network_instances,
-                };
+                    failed_network_instances,
+                );

-                match client.heartbeat(BaseController::default(), request).await {
+                match client
+                    .heartbeat(heartbeat_policy.controller(), request)
+                    .await
+                {
                    Ok(response) => {
                        tracing::debug!(?response, "config-server heartbeat response");
+                        let (next_policy, adjusted) = HeartbeatPolicy::from_response(&response);
+                        if adjusted {
+                            tracing::warn!(
+                                requested_interval_ms = ?response.heartbeat_interval_ms,
+                                requested_timeout_ms = ?response.heartbeat_timeout_ms,
+                                applied_interval_ms = next_policy.interval.as_millis(),
+                                applied_timeout_ms = next_policy.timeout_ms,
+                                "config-server heartbeat policy was outside safe bounds"
+                            );
+                        }
+                        heartbeat_policy = next_policy;
+                        wait_for_next_heartbeat(
+                            controller.backend.as_ref(),
+                            observed_generation,
+                            heartbeat_policy,
+                            heartbeat_started_at.elapsed(),
+                        )
+                        .await;
                    }
                    Err(error) => {
                        tracing::error!(?error, "config-server heartbeat failed");
@@ -430,6 +585,25 @@ mod tests {
        attempts: AtomicUsize,
    }

+    struct ImmediateStateChangeBackend;
+
+    #[async_trait]
+    impl WebClientBackend for ImmediateStateChangeBackend {
+        fn register(&self, _registry: &ServiceRegistry) {}
+
+        async fn instance_ids(&self) -> anyhow::Result<Vec<uuid::Uuid>> {
+            Ok(Vec::new())
+        }
+
+        fn failed_instance_ids(&self) -> Vec<uuid::Uuid> {
+            Vec::new()
+        }
+
+        async fn wait_for_instance_state_change(&self, generation: usize) -> usize {
+            generation.wrapping_add(1)
+        }
+    }
+
    #[async_trait]
    impl TunnelDialer for StalledThenReadyDialer {
        async fn connect(&self) -> anyhow::Result<Box<dyn Tunnel>> {
@@ -446,6 +620,20 @@ mod tests {
        }
    }

+    #[test]
+    fn heartbeat_hides_failed_instances_from_the_running_list() {
+        let running = uuid::Uuid::new_v4();
+        let failed = uuid::Uuid::new_v4();
+        let stopped_clean = uuid::Uuid::new_v4();
+        let instance_ids = vec![running, failed, stopped_clean];
+        let failed_instance_ids = vec![failed];
+
+        let reported = running_instances_for_heartbeat(instance_ids.clone(), &failed_instance_ids);
+
+        assert_eq!(reported, vec![running, stopped_clean]);
+        assert!(running_instances_for_heartbeat(instance_ids, &[]).len() == 3);
+    }
+
    #[tokio::test]
    async fn stalled_connection_attempt_times_out_and_allows_redial() {
        let connector = StalledThenReadyDialer {
@@ -463,6 +651,26 @@ mod tests {
        assert_eq!(connector.attempts.load(Ordering::Relaxed), 2);
    }

+    #[tokio::test]
+    async fn instance_state_change_interrupts_a_long_heartbeat_interval() {
+        let policy = HeartbeatPolicy {
+            interval: std::time::Duration::from_secs(60),
+            timeout_ms: 65_000,
+        };
+
+        tokio::time::timeout(
+            std::time::Duration::from_secs(1),
+            wait_for_next_heartbeat(
+                &ImmediateStateChangeBackend,
+                0,
+                policy,
+                std::time::Duration::ZERO,
+            ),
+        )
+        .await
+        .expect("instance state change must wake heartbeat before its interval");
+    }
+
    #[test]
    fn endpoint_normalizes_non_websocket_paths() {
        let endpoint =
@@ -502,4 +710,87 @@ mod tests {
    fn endpoint_rejects_an_empty_token() {
        assert!(ConfigServerEndpoint::parse("udp://example.com", |_| true).is_err());
    }
+
+    #[test]
+    fn heartbeat_request_carries_registered_and_failed_instance_ids() {
+        let runtime_id = uuid::Uuid::new_v4();
+        let registered = uuid::Uuid::new_v4();
+        let failed = uuid::Uuid::new_v4();
+        let request = build_heartbeat_request(
+            &WebClientConfig {
+                token: "token".to_owned(),
+                machine_id: uuid::Uuid::new_v4(),
+                hostname: "host".to_owned(),
+                device_os: DeviceOsInfo::default(),
+                easytier_version: "test-version".to_owned(),
+                secure_mode: false,
+            },
+            runtime_id,
+            vec![registered],
+            vec![failed],
+        );
+
+        assert_eq!(request.inst_id.map(uuid::Uuid::from), Some(runtime_id));
+        assert_eq!(
+            request
+                .running_network_instances
+                .into_iter()
+                .map(uuid::Uuid::from)
+                .collect::<Vec<_>>(),
+            vec![registered]
+        );
+        assert_eq!(
+            request
+                .failed_network_instances
+                .into_iter()
+                .map(uuid::Uuid::from)
+                .collect::<Vec<_>>(),
+            vec![failed]
+        );
+        assert!(request.support_heartbeat_policy);
+    }
+
+    #[test]
+    fn heartbeat_policy_uses_safe_defaults_for_legacy_servers() {
+        let (policy, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse::default());
+
+        assert!(!adjusted);
+        assert_eq!(
+            policy.interval,
+            std::time::Duration::from_millis(DEFAULT_HEARTBEAT_INTERVAL_MS.into())
+        );
+        assert_eq!(policy.timeout_ms, DEFAULT_HEARTBEAT_TIMEOUT_MS as i32);
+    }
+
+    #[test]
+    fn heartbeat_policy_clamps_server_values_and_preserves_timeout_margin() {
+        let (minimum, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse {
+            heartbeat_interval_ms: Some(1),
+            heartbeat_timeout_ms: Some(1),
+        });
+        assert!(adjusted);
+        assert_eq!(
+            minimum.interval,
+            std::time::Duration::from_millis(MIN_HEARTBEAT_INTERVAL_MS.into())
+        );
+        assert_eq!(minimum.timeout_ms, 6_000);
+
+        let (maximum, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse {
+            heartbeat_interval_ms: Some(u32::MAX),
+            heartbeat_timeout_ms: Some(u32::MAX),
+        });
+        assert!(adjusted);
+        assert_eq!(
+            maximum.interval,
+            std::time::Duration::from_millis(MAX_HEARTBEAT_INTERVAL_MS.into())
+        );
+        assert_eq!(maximum.timeout_ms, MAX_HEARTBEAT_TIMEOUT_MS as i32);
+
+        let (margin, adjusted) = HeartbeatPolicy::from_response(&HeartbeatResponse {
+            heartbeat_interval_ms: Some(60_000),
+            heartbeat_timeout_ms: Some(5_000),
+        });
+        assert!(adjusted);
+        assert_eq!(margin.timeout_ms, 65_000);
+    }
 }
@@ -265,6 +265,10 @@ impl WebClientBackend for WasiWebClientBackend {
            .await?;
        Ok(response.inst_ids.into_iter().map(Into::into).collect())
    }
+
+    fn failed_instance_ids(&self) -> Vec<uuid::Uuid> {
+        Vec::new()
+    }
 }

 pub(super) struct WasiWebClientRuntime {