x15904151985/EasyTier
1
0 Fork 0
mirror of https://github.com/EasyTier/EasyTier.git synced 2026-10-08 10:56:13 -08:00
Code Issues Packages Projects Releases Wiki Activity

fix(web): harden managed config sync between console and clients (#2567)

Browse Source 
* fix(web): fence managed config runtime reconciliation

Keep runtime reconciliation tied to the currently authorized session so
stale connections cannot mutate a replacement session runtime.

Accumulate only contiguous dirty IDs and load their latest SQLite state.
Require the applied revision to match the earliest Patch base and the
persisted revision to match the latest target. Otherwise, reconcile the
full desired state.

Use separate runtime-state and config-cache epochs. Managed updates can
reuse observed configs; direct mutations invalidate them. Update sync
documentation to match.

* fix(web): interrupt validation retry on state changes

Track meaningful validation state changes separately from periodic dirty signals. Applied revision changes wake a failed validation immediately, while heartbeat-driven revalidation retains the retry backoff.

Treat Notify as a wake-up hint and recheck the state-change epoch after every wake so stored permits and periodic heartbeats cannot cause retry storms.

* fix(web): retry unconfirmed connected webhooks

Retry node-connected webhook delivery on retryable errors with a
short 100ms/500ms backoff and give up immediately on non-retryable
errors. Re-check that the session still owns the connection before
every attempt and before recording the delivery, so a replaced
session can no longer record a stale connected binding.

* fix(web): fence disconnects by session ownership

Return whether session removal actually removed the current route owner, and emit disconnected only for that owner. Replaced sessions can no longer invalidate a newer connected route.

* fix(web): hot-patch managed hostnames

Include hostname changes in the hot-patch path instead of falling
back to a full restart. When a full overwrite run is required and
the desired config has no hostname, inherit the current runtime
hostname so an unmanaged value survives until it is explicitly
cleared.

Read back the runtime config after an overwrite run and verify it
converged instead of assuming the desired state was applied.

* fix(web): retry transient runtime reconciliation failures

Keep the per-session managed runtime reconciliation worker alive when a
single database round fails. Retry from the next heartbeat so persisted
managed revisions can still converge after restart-time contention.

Reserve terminal worker shutdown for destroyed session or storage state,
and cover recovery after a transient revision read failure.

* fix(web): accept omitted hostname after runtime apply

Release 2.6.4 omits hostname from config readback when it matches the device hostname. Trust a successful hostname mutation only when the returned field is absent, while continuing to verify every other field and rejecting explicit mismatches.

* fix(web): ignore unmanaged runtime device names

Windows release 2.6.4 generates a random interface name when the managed config leaves dev_name empty. Exclude that runtime-owned value from reconciliation unless the desired config explicitly sets a non-empty device name, preventing endless overwrite restarts.

* feat(web): report failed network instances to console

Expose stopped Core instances with startup errors in heartbeats.

Merge Core failures with direct managed-run RPC failures in easytier-web.

Send failed instance IDs during token validation without error text.

Prune local run failures when managed configs are deleted.

* fix(web): distinguish unknown runtime application state

Track whether the current session has observed its applied revision
separately from the optional revision value. Report this fact through
validate-token so Console can preserve application state across
receiver restarts while recognizing deliberate pending mutations.

* feat(web): configure heartbeat timing from server

Heartbeat responses now provide the interval and RPC timeout.

Legacy servers use local defaults and remote values are clamped.

Web configuration and session receive timeout follow the policy.

* fix(web): reject inactive control sessions

Route control RPCs by machine id only to sessions whose RPC manager
is still running, so a session that has been stopped or replaced
can no longer receive control traffic addressed to the device.

* fix(core): filter network info before collection

When a collect-network-info request names specific instances,
collect those instances only instead of collecting every instance
and filtering the result afterwards, so unrequested instances no
longer run per-collection work on every request.

* feat(web): enable focused runtime diagnostics

Enable easytier-web info logs by default while preserving explicit log configuration. Record startup settings, session lifecycle, failed instance changes, webhook queue and request latency, and managed runtime operation timings for production diagnosis.

* fix(web): preserve managed revision across reconnects

Keep one runtime identifier for each Core WebClient lifetime.

Reuse its managed runtime state after transport reconnects.

Retain applied revisions and reconcile hints while disconnected.

Preserve runtime epochs so stale work cannot mark a revision applied.

Reject stale sessions from reclaiming routes after reconnect.

Core or Web restarts and legacy clients still use unknown state.

Immediately revalidate a restored revision after authentication.

Document local management RPC drift as an accepted trade-off.

This lets Console converge without waiting for periodic validation.

* fix(web): satisfy clippy across managed config sync tests

Scope managed runtime guards to blocks in runtime revision tests so
no std MutexGuard is held across await points, return the applied
revision directly instead of through a let binding, and pass
WebhookValidationInput to request_heartbeat_validation instead of
expanding it into eight separate arguments.

* fix(core): stop reporting failed instances as running in heartbeats

A stopped instance with a startup error appeared in both
running_network_instances and failed_network_instances, so the
server treated it as running and never re-ran its managed config.
Exclude failed instance ids when building the running list so the
reconciler restarts them.

* fix(core): close missed-wakeup race in instance state changes

wait_for_change created the Notified future before reading the
generation but only registered it when awaited. A change landing in
between fired notify_waiters with no registered waiter and delayed
the heartbeat by a full interval. Enable the future before reading
the generation so every change wakes a waiting heartbeat.

* fix(web): address review findings

Fence webhook validation and connection transitions against stale
state, redact credentials from default-level logs, and stabilize
runtime reconciliation:

- Record connected bindings only while the session still owns the
  machine route, and skip disconnect compensation once a replacement
  owns the route so a stale disconnect cannot revoke it.
- Discard webhook validation results when the change epoch moved
  during the HTTP round, so a stale rejection cannot invalidate the
  current session.
- Drop user_token fields from info and warn logs that became
  visible with info-level defaults.
- Restore a hostname omitted by the 2.6.4 readback into the cached
  runtime config after a successful mutation, so later rounds stop
  re-sending the same hostname patch.
- Reconcile running web configs when no revision is tracked so
  legacy unrevisioned updates converge, and wake sessions for
  unrevisioned full updates instead of waiting for the next
  heartbeat.

* chore(go): regenerate web proto bindings for heartbeat fields

Add failed_network_instances, support_heartbeat_policy, and the
heartbeat policy response fields to the checked-in Go bindings.
Other proto packages are left as-is because their drift predates
this change.

* fix(web): redact user tokens from positional log arguments

Three runtime reconciliation info logs and the user lookup error
contexts printed user_token through format arguments, which the
earlier field-syntax redaction missed. The reconcile log now fires
every round for unrevisioned machines, so remove the token from
these messages as well.

* fix(web): fence stale validation and runtime reconcile rounds

Check webhook validation epochs while holding the session write lock,
so stale success and rejection responses cannot change session state.
Advance the runtime epoch for unrevisioned full config updates, and
exclude failed instances from heartbeat and RPC reconciliation lists
so stopped instances are restarted instead of repeatedly hot-patched.

Release test read guards before awaiting validation apply calls. Set
up the no-pending condition before asserting that an applied revision
is a no-op, and verify that its runtime epoch remains unchanged.

Validation: all 137 client_manager tests passed.

* test(credentials): cover P2P with active VPN portal

Model an admin and temporary credential peer connected as a foreign network through a public server with data relay disabled. Verify their direct connection can be replaced after a WireGuard portal client comes online.

* test(credentials): stabilize two-admins failover assertions

The two-admins non-reusable credential test could fail on slow
convergence: after dropping the winning peer it relied on a single
route sample passing a bare AND condition, then re-asserted the same
expectations through one-shot checks seconds later. A transient route
flap in that window (for example a briefly resurrected winner route
from stale conn info) turned a passing convergence into a hard assert
failure. This matches the 48.9s CI flake of
credential_non_reusable_across_two_admins_allows_only_one_peer
observed on 2026-08-12.

Changes:

- wait for bidirectional admin connectivity (AND) with a 20s budget
  before issuing the credential, instead of a one-directional OR
- replace the failover wait_for_condition with
  wait_stable_failover_visibility_on_admins, which requires three
  consecutive samples of loser-present and winner-absent on both
  admins within the same 60s budget and logs every sample
- enrich the stable-single-winner timeout message with per-admin
  visibility flags and elapsed time for triage

All existing contracts are preserved; only observation windows and
diagnostics change. Validated in the rust container: three passes at
normal speed (54.1s / 53.8s / 53.1s) plus one slow-convergence round
(172.7s) that would have raced the old one-shot sampling; it now
passes with failover samples logged. cargo fmt and clippy -D warnings
clean.
This commit is contained in:
KKRainbow authored and GitHub committed 2026-09-13 01:13:28 +08:00
1 parent e0bdb516b6
commit c96b6c1961
21 files changed
+4332 -561
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

easytier-web/locales/app.yml
+5 -2
View File
@@ -41,8 +41,11 @@ cli:
en: "The path to the GeoIP2 database file, used to lookup the location of the client, default is the embedded file (only country information) , recommend https://github.com/P3TERX/GeoLite.mmdb"
zh-CN: "GeoIP2 数据库文件路径，用于查找客户端的位置，默认为嵌入文件（仅国家信息），推荐 https://github.com/P3TERX/GeoLite.mmdb"
heartbeat_min_response_ms:
en: "Minimum response time for config-server heartbeat RPCs in milliseconds, default is 0"
zh-CN: "配置服务心跳 RPC 的最短响应时间，单位毫秒，默认为 0"
en: "Config-server heartbeat interval in milliseconds, default is 3500"
zh-CN: "配置服务心跳周期，单位毫秒，默认为 3500"
heartbeat_timeout_ms:
en: "Config-server heartbeat RPC timeout in milliseconds, default is 15000"
zh-CN: "配置服务心跳 RPC 超时时间，单位毫秒，默认为 15000"
disable_registration:
en: "Disable user registration"
zh-CN: "禁用用户注册"
easytier-web/src/client_manager/mod.rs
+186 -40
View File
@@ -3,15 +3,20 @@ mod runtime_reconcile;
pub mod session;
pub mod storage;
use std::sync::{
Arc,
atomic::{AtomicU32, AtomicU64, Ordering},
};
use std::time::Duration;
use std::{
collections::HashSet,
sync::{
Arc,
atomic::{AtomicU32, AtomicU64, Ordering},
},
};
use dashmap::DashMap;
use easytier::proto::{
api::manage::WebClientService, rpc_types::controller::BaseController, web::HeartbeatRequest,
api::manage::WebClientService,
rpc_types::controller::BaseController,
web::{HeartbeatRequest, HeartbeatResponse},
};
use easytier_core::{
management::remote_client::{self, RemoteClientManager},
@@ -19,7 +24,7 @@ use easytier_core::{
tunnel::{Tunnel, web_security},
};
use maxminddb::geoip2;
use session::{Location, ManagedConfigRevisionDelta, Session};
use session::{Location, ManagedConfigPersistedChange, Session};
use storage::{Storage, StorageToken};
use crate::FeatureFlags;
@@ -30,6 +35,67 @@ use crate::db::{Db, UserIdInDb, entity::user_running_network_configs};
pub(crate) use managed_config::ManagedConfigError;
const DEFAULT_HEARTBEAT_INTERVAL: Duration = Duration::from_millis(3_500);
const DEFAULT_HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(15);
const MIN_HEARTBEAT_INTERVAL: Duration = Duration::from_secs(1);
const MAX_HEARTBEAT_INTERVAL: Duration = Duration::from_secs(60);
const MIN_HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(5);
const MAX_HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(120);
const HEARTBEAT_TIMEOUT_MARGIN: Duration = Duration::from_secs(5);
#[derive(Debug, Clone, Copy)]
pub(crate) struct HeartbeatPolicy {
interval: Duration,
timeout: Duration,
}
impl Default for HeartbeatPolicy {
fn default() -> Self {
Self {
interval: DEFAULT_HEARTBEAT_INTERVAL,
timeout: DEFAULT_HEARTBEAT_TIMEOUT,
}
}
}
impl HeartbeatPolicy {
pub(crate) fn from_millis(interval_ms: u64, timeout_ms: u64) -> anyhow::Result<Self> {
let interval = if interval_ms == 0 {
DEFAULT_HEARTBEAT_INTERVAL
} else {
Duration::from_millis(interval_ms)
};
let timeout = Duration::from_millis(timeout_ms);
if !(MIN_HEARTBEAT_INTERVAL..=MAX_HEARTBEAT_INTERVAL).contains(&interval) {
anyhow::bail!("heartbeat interval must be between 1000 and 60000 milliseconds");
}
if !(MIN_HEARTBEAT_TIMEOUT..=MAX_HEARTBEAT_TIMEOUT).contains(&timeout) {
anyhow::bail!("heartbeat timeout must be between 5000 and 120000 milliseconds");
}
if timeout < interval.saturating_add(HEARTBEAT_TIMEOUT_MARGIN) {
anyhow::bail!(
"heartbeat timeout must exceed the interval by at least 5000 milliseconds"
);
}
Ok(Self { interval, timeout })
}
fn response(self) -> HeartbeatResponse {
HeartbeatResponse {
heartbeat_interval_ms: Some(self.interval.as_millis() as u32),
heartbeat_timeout_ms: Some(self.timeout.as_millis() as u32),
}
}
fn session_rx_timeout(self) -> Duration {
Duration::from_secs(30).max(self.timeout.saturating_add(HEARTBEAT_TIMEOUT_MARGIN))
}
fn legacy_response_delay(self) -> Duration {
self.interval.min(DEFAULT_HEARTBEAT_INTERVAL)
}
}
#[derive(rust_embed::Embed)]
#[folder = "resources/"]
#[include = "geoip2-cn.mmdb"]
@@ -69,14 +135,14 @@ pub struct ClientManager {
webhook_config: SharedWebhookConfig,
geoip_db: Arc<Option<maxminddb::Reader<Vec<u8>>>>,
heartbeat_min_response_delay: Duration,
heartbeat_policy: HeartbeatPolicy,
}
impl ClientManager {
pub fn new(
db: Db,
geoip_db: Option<String>,
heartbeat_min_response_delay: Duration,
heartbeat_policy: HeartbeatPolicy,
feature_flags: Arc<FeatureFlags>,
webhook_config: SharedWebhookConfig,
) -> Self {
@@ -101,7 +167,7 @@ impl ClientManager {
webhook_config,
geoip_db: Arc::new(load_geoip_db(geoip_db)),
heartbeat_min_response_delay,
heartbeat_policy,
}
}
@@ -117,7 +183,7 @@ impl ClientManager {
let listeners_cnt = self.listeners_cnt.clone();
let next_session_epoch = self.next_session_epoch.clone();
let geoip_db = self.geoip_db.clone();
let heartbeat_min_response_delay = self.heartbeat_min_response_delay;
let heartbeat_policy = self.heartbeat_policy;
let feature_flags = self.feature_flags.clone();
let webhook_config = self.webhook_config.clone();
self.tasks.spawn(async move {
@@ -146,7 +212,7 @@ impl ClientManager {
storage.clone(),
client_url.clone(),
location,
heartbeat_min_response_delay,
heartbeat_policy,
feature_flags.clone(),
webhook_config.clone(),
next_session_epoch.fetch_add(1, Ordering::Relaxed) + 1,
@@ -188,7 +254,7 @@ impl ClientManager {
.get_client_url_by_machine_id(user_id, machine_id)?;
self.client_sessions
.get(&c_url)
.map(|item| item.value().clone())
.and_then(|item| item.is_running().then(|| item.value().clone()))
}
pub async fn disconnect_session_by_machine_id(
@@ -238,11 +304,14 @@ impl ClientManager {
if matches!(
status,
managed_config::ManagedConfigApplyStatus::Applied { .. }
) && let Some(config_revision) = config_revision
&& let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
) && self.storage.record_full_managed_config_change(
user_id,
machine_id,
config_revision.as_deref(),
) && let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
{
session
.notify_full_config_revision_changed(user_id, machine_id, config_revision)
.notify_managed_runtime_state_changed(user_id, machine_id)
.await;
}
Ok(())
@@ -259,7 +328,7 @@ impl ClientManager {
) -> anyhow::Result<()> {
let config_revision = config_revision.trim().to_string();
let expected_config_revision = expected_config_revision.trim().to_string();
let upsert_instance_ids = upserts
let mut dirty_instance_ids: HashSet<_> = upserts
.iter()
.map(|config| config.instance_id.clone())
.collect();
@@ -276,23 +345,26 @@ impl ClientManager {
if let managed_config::ManagedConfigApplyStatus::Applied {
deleted_web_instance_ids,
} = status
&& let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
{
session
.notify_patch_config_revision_changed(
user_id,
machine_id,
ManagedConfigRevisionDelta {
expected_revision: expected_config_revision,
target_revision: config_revision,
upsert_instance_ids,
delete_instance_ids: deleted_web_instance_ids
.into_iter()
.map(|instance_id| instance_id.to_string())
.collect(),
},
)
.await;
dirty_instance_ids.extend(
deleted_web_instance_ids
.into_iter()
.map(|instance_id| instance_id.to_string()),
);
let changed = self.storage.record_patch_managed_config_change(
user_id,
machine_id,
ManagedConfigPersistedChange {
expected_revision: expected_config_revision,
target_revision: config_revision,
dirty_instance_ids,
},
);
if changed && let Some(session) = self.get_session_by_machine_id(user_id, &machine_id) {
session
.notify_managed_runtime_state_changed(user_id, machine_id)
.await;
}
}
Ok(())
}
@@ -302,9 +374,13 @@ impl ClientManager {
user_id: UserIdInDb,
machine_id: uuid::Uuid,
) {
if let Some(session) = self.get_session_by_machine_id(user_id, &machine_id) {
if self
.storage
.invalidate_managed_runtime_state(user_id, machine_id)
&& let Some(session) = self.get_session_by_machine_id(user_id, &machine_id)
{
session
.invalidate_applied_config_revision(user_id, machine_id)
.notify_managed_runtime_state_changed(user_id, machine_id)
.await;
}
}
@@ -474,11 +550,36 @@ mod tests {
use sqlx::Executor;
use crate::{
FeatureFlags, client_manager::ClientManager, db::Db, webhook::ManagedNetworkConfig,
FeatureFlags,
client_manager::{ClientManager, HeartbeatPolicy, session::Session, storage::StorageToken},
db::Db,
webhook::ManagedNetworkConfig,
};
const MANAGED_CONFIG_TOKEN: &str = "managed-config-token";
#[test]
fn heartbeat_policy_validates_server_configuration() {
let policy = HeartbeatPolicy::from_millis(3_500, 15_000).unwrap();
let response = policy.response();
assert_eq!(response.heartbeat_interval_ms, Some(3_500));
assert_eq!(response.heartbeat_timeout_ms, Some(15_000));
assert_eq!(policy.session_rx_timeout(), Duration::from_secs(30));
let legacy_default = HeartbeatPolicy::from_millis(0, 15_000).unwrap();
assert_eq!(legacy_default.response().heartbeat_interval_ms, Some(3_500));
let slow = HeartbeatPolicy::from_millis(60_000, 65_000).unwrap();
assert_eq!(slow.session_rx_timeout(), Duration::from_secs(70));
assert_eq!(slow.legacy_response_delay(), Duration::from_millis(3_500));
assert!(HeartbeatPolicy::from_millis(999, 15_000).is_err());
assert!(HeartbeatPolicy::from_millis(60_001, 120_000).is_err());
assert!(HeartbeatPolicy::from_millis(3_500, 4_999).is_err());
assert!(HeartbeatPolicy::from_millis(60_000, 64_999).is_err());
assert!(HeartbeatPolicy::from_millis(3_500, 120_001).is_err());
}
async fn wait_for_condition<F, Fut>(mut condition: F, timeout: Duration)
where
F: FnMut() -> Fut,
@@ -655,7 +756,7 @@ mod tests {
let mut mgr = ClientManager::new(
Db::memory_db().await,
None,
Duration::ZERO,
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
Arc::new(FeatureFlags::default()),
webhook_config,
);
@@ -683,6 +784,51 @@ mod tests {
webhook_server.abort();
}
#[tokio::test]
async fn non_running_session_is_not_routable_by_machine_id() {
let db = Db::memory_db().await;
let mgr = ClientManager::new(
db.clone(),
None,
HeartbeatPolicy::default(),
Arc::new(FeatureFlags::default()),
Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
)),
);
let user_id = db.auto_create_user("token").await.unwrap().id;
let machine_id = uuid::Uuid::new_v4();
let client_url = url::Url::parse("udp://127.0.0.1:22020").unwrap();
mgr.storage.update_client(
StorageToken {
token: "token".to_string(),
client_url: client_url.clone(),
machine_id,
user_id,
},
1,
true,
);
let session = Arc::new(Session::new(
mgr.storage.weak_ref(),
client_url.clone(),
None,
HeartbeatPolicy::default(),
Arc::new(FeatureFlags::default()),
Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
)),
1,
));
assert!(!session.is_running());
mgr.client_sessions.insert(client_url, session);
assert!(
mgr.get_session_by_machine_id(user_id, &machine_id)
.is_none()
);
}
async fn wait_for_validated_user(mgr: &ClientManager, machine_id: uuid::Uuid) -> i32 {
tokio::time::timeout(Duration::from_secs(12), async {
loop {
@@ -1008,7 +1154,7 @@ mod tests {
let mut mgr = ClientManager::new(
Db::memory_db().await,
None,
Duration::ZERO,
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
Arc::new(FeatureFlags::default()),
Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
@@ -1075,7 +1221,7 @@ mod tests {
let mut mgr = ClientManager::new(
Db::memory_db().await,
None,
Duration::ZERO,
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
Arc::new(FeatureFlags::default()),
webhook_config,
);
@@ -1141,7 +1287,7 @@ mod tests {
let mut mgr = ClientManager::new(
Db::memory_db().await,
None,
Duration::ZERO,
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
Arc::new(FeatureFlags::default()),
webhook_config,
);
@@ -1333,7 +1479,7 @@ mod tests {
let mut mgr = ClientManager::new(
Db::memory_db().await,
None,
Duration::ZERO,
HeartbeatPolicy::from_millis(0, 15_000).unwrap(),
Arc::new(FeatureFlags::default()),
webhook_config,
);
easytier-web/src/client_manager/runtime_reconcile.rs
+297 -28
View File
@@ -28,7 +28,7 @@ use easytier::{
use super::session::{SessionConfigClient, SessionRpcClient};
pub(super) enum RuntimeReconcileAction {
None,
Unchanged(Box<NetworkConfig>),
Run {
config: Box<NetworkConfig>,
overwrite: bool,
@@ -269,27 +269,47 @@ fn normalized_managed_credentials(
Ok(NetworkConfig::new_from_config(config.gen_config()?)?.managed_credentials)
}
fn is_automatic_windows_dev_name(dev_name: &str) -> bool {
let Some((interface_count, suffix)) = dev_name
.strip_prefix("et_")
.and_then(|value| value.split_once('_'))
else {
return false;
};
!interface_count.is_empty()
&& interface_count.bytes().all(|byte| byte.is_ascii_digit())
&& suffix.len() == 4
&& suffix
.bytes()
.all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit())
}
fn web_source_runtime_patch(
current: &NetworkConfig,
desired: &NetworkConfig,
) -> anyhow::Result<Option<InstanceConfigPatch>> {
if let Some(desired_hostname) = desired
.hostname
.as_deref()
.filter(|hostname| !hostname.is_empty())
&& current.hostname.as_deref() != Some(desired_hostname)
{
return Ok(None);
}
let mut current_base = hot_patch_base(current)?;
let mut desired_base = hot_patch_base(desired)?;
current_base.hostname = None;
desired_base.hostname = None;
if desired.dev_name.is_none()
|| (desired.dev_name.as_deref() == Some("")
&& current
.dev_name
.as_deref()
.is_some_and(is_automatic_windows_dev_name))
{
current_base.dev_name = None;
desired_base.dev_name = None;
}
let current_hostname = current_base.hostname.take().unwrap_or_default();
let desired_hostname = desired_base.hostname.take().unwrap_or_default();
if current_base != desired_base {
return Ok(None);
}
let mut patch = InstanceConfigPatch::default();
if desired.hostname.is_some() && current_hostname != desired_hostname {
patch.hostname = Some(desired_hostname);
}
let current_acl = normalized_acl(&current.acl);
let desired_acl = normalized_acl(&desired.acl);
if current_acl != desired_acl {
@@ -358,14 +378,41 @@ fn web_source_runtime_patch(
Ok(Some(patch))
}
fn ensure_runtime_config_converged(
// Release 2.6.4 omits a configured hostname that matches the device
// hostname from config readback. After a successful hostname mutation the
// desired value must be restored into the observed config, otherwise every
// later plan re-sends the same hostname patch.
pub(super) fn restore_omitted_hostname(
current: &mut NetworkConfig,
desired: &NetworkConfig,
hostname_applied: bool,
) {
if hostname_applied && current.hostname.is_none() && desired.hostname.is_some() {
current.hostname = desired.hostname.clone();
}
}
pub(super) fn ensure_runtime_config_converged(
current: &NetworkConfig,
desired: &NetworkConfig,
hostname_applied: bool,
) -> anyhow::Result<()> {
let patch = web_source_runtime_patch(current, desired)?;
match patch {
Some(patch) if patch == InstanceConfigPatch::default() => Ok(()),
Some(_) => anyhow::bail!("runtime config still needs patch after reconcile"),
Some(mut patch) => {
// Release 2.6.4 omits a configured hostname when it equals
// the device hostname. The successful mutation is therefore
// authoritative for hostname, while every other field remains
// verified from the runtime readback.
if hostname_applied && current.hostname.is_none() {
patch.hostname = None;
}
if patch == InstanceConfigPatch::default() {
Ok(())
} else {
anyhow::bail!("runtime config still needs patch after reconcile")
}
}
None => anyhow::bail!("runtime config still needs full overwrite after reconcile"),
}
}
@@ -429,13 +476,19 @@ pub(super) fn prepare_web_source_runtime_reconcile_from_current(
desired_config: NetworkConfig,
) -> anyhow::Result<RuntimeReconcileAction> {
let Some(patch) = web_source_runtime_patch(current_config, &desired_config)? else {
let mut run_config = desired_config;
if run_config.hostname.is_none() {
run_config.hostname = current_config.hostname.clone();
}
return Ok(RuntimeReconcileAction::Run {
config: Box::new(desired_config),
config: Box::new(run_config),
overwrite: true,
});
};
if patch == InstanceConfigPatch::default() {
return Ok(RuntimeReconcileAction::None);
return Ok(RuntimeReconcileAction::Unchanged(Box::new(
current_config.clone(),
)));
}
Ok(RuntimeReconcileAction::Patch(Box::new(patch)))
@@ -449,12 +502,17 @@ pub(super) async fn apply_web_source_runtime_reconcile(
action: RuntimeReconcileAction,
) -> anyhow::Result<NetworkConfig> {
match action {
RuntimeReconcileAction::None => Ok(desired_config),
RuntimeReconcileAction::Unchanged(current_config) => Ok(*current_config),
RuntimeReconcileAction::Run { config, overwrite } => {
let hostname_applied = config.hostname.is_some();
run_web_source_instance(rpc_client, inst_id, *config, overwrite).await?;
Ok(desired_config)
let mut current_config = get_runtime_config(rpc_client, inst_id).await?;
ensure_runtime_config_converged(&current_config, &desired_config, hostname_applied)?;
restore_omitted_hostname(&mut current_config, &desired_config, hostname_applied);
Ok(current_config)
}
RuntimeReconcileAction::Patch(patch) => {
let hostname_applied = patch.hostname.is_some();
config_client
.patch_config(
BaseController::default(),
@@ -464,8 +522,9 @@ pub(super) async fn apply_web_source_runtime_reconcile(
},
)
.await?;
let current_config = get_runtime_config(rpc_client, inst_id).await?;
ensure_runtime_config_converged(&current_config, &desired_config)?;
let mut current_config = get_runtime_config(rpc_client, inst_id).await?;
ensure_runtime_config_converged(&current_config, &desired_config, hostname_applied)?;
restore_omitted_hostname(&mut current_config, &desired_config, hostname_applied);
Ok(current_config)
}
}
@@ -759,6 +818,63 @@ mod tests {
);
}
#[test]
fn runtime_reconcile_ignores_automatic_device_name_when_unmanaged() {
let mut current = config_with_port_forwards(Vec::new());
current.dev_name = Some("et_3_abcd".to_string());
let desired = config_with_port_forwards(Vec::new());
let action = prepare_web_source_runtime_reconcile_from_current(&current, desired)
.expect("prepare reconcile");
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
}
#[test]
fn runtime_reconcile_ignores_automatic_device_name_for_empty_desired_name() {
let mut current = config_with_port_forwards(Vec::new());
current.dev_name = Some("et_3_abcd".to_string());
let mut desired = config_with_port_forwards(Vec::new());
desired.dev_name = Some(String::new());
let action = prepare_web_source_runtime_reconcile_from_current(&current, desired)
.expect("prepare reconcile");
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
}
#[test]
fn runtime_reconcile_clears_explicit_device_name() {
let mut current = config_with_port_forwards(Vec::new());
current.dev_name = Some("managed-device".to_string());
let mut desired = config_with_port_forwards(Vec::new());
desired.dev_name = Some(String::new());
let action = prepare_web_source_runtime_reconcile_from_current(&current, desired)
.expect("prepare reconcile");
let RuntimeReconcileAction::Run { overwrite, .. } = action else {
panic!("clearing an explicit device name should require a full overwrite");
};
assert!(overwrite);
}
#[test]
fn runtime_reconcile_applies_explicit_device_name() {
let mut current = config_with_port_forwards(Vec::new());
current.dev_name = Some("et_3_abcd".to_string());
let mut desired = config_with_port_forwards(Vec::new());
desired.dev_name = Some("managed-device".to_string());
let action = prepare_web_source_runtime_reconcile_from_current(&current, desired)
.expect("prepare reconcile");
let RuntimeReconcileAction::Run { overwrite, .. } = action else {
panic!("explicit device name should require a full overwrite");
};
assert!(overwrite);
}
#[test]
fn runtime_convergence_rejects_stale_extra_port_forward() {
let current = config_with_port_forwards(vec![
@@ -769,7 +885,7 @@ mod tests {
let desired =
config_with_port_forwards(vec![port_forward(23000, 5174), port_forward(23007, 3389)]);
let err = ensure_runtime_config_converged(&current, &desired)
let err = ensure_runtime_config_converged(&current, &desired, false)
.expect_err("extra runtime port forward should not converge");
assert!(
@@ -791,13 +907,14 @@ mod tests {
.expect("hot patch");
assert_eq!(patch, InstanceConfigPatch::default());
ensure_runtime_config_converged(&current, &desired).expect("runtime converged");
ensure_runtime_config_converged(&current, &desired, false).expect("runtime converged");
}
#[test]
fn runtime_patch_rejects_non_hot_config_change() {
let current = config_with_port_forwards(Vec::new());
let mut desired = current.clone();
desired.network_secret = Some("new-secret".to_string());
let patch = web_source_runtime_patch(&current, &desired).expect("build patch");
@@ -805,6 +922,36 @@ mod tests {
assert!(patch.is_none());
}
#[test]
fn full_overwrite_preserves_unmanaged_hostname_for_later_explicit_clear() {
let mut current = config_with_port_forwards(Vec::new());
current.hostname = Some("runtime-host".to_string());
let mut unmanaged_desired = current.clone();
unmanaged_desired.hostname = None;
unmanaged_desired.network_secret = Some("new-secret".to_string());
let action =
prepare_web_source_runtime_reconcile_from_current(&current, unmanaged_desired.clone())
.expect("prepare full overwrite");
let RuntimeReconcileAction::Run { config, overwrite } = action else {
panic!("non-hot change should require a full overwrite");
};
assert!(overwrite);
assert_eq!(config.hostname.as_deref(), Some("runtime-host"));
let observed_after_run = *config;
let mut explicit_clear = unmanaged_desired;
explicit_clear.hostname = Some(String::new());
let action =
prepare_web_source_runtime_reconcile_from_current(&observed_after_run, explicit_clear)
.expect("prepare explicit clear");
let RuntimeReconcileAction::Patch(patch) = action else {
panic!("explicit clear should patch the preserved runtime hostname");
};
assert_eq!(patch.hostname.as_deref(), Some(""));
}
#[test]
fn runtime_patch_replaces_managed_credentials_without_full_run() {
let current = config_with_port_forwards(Vec::new());
@@ -1086,14 +1233,136 @@ mod tests {
}
#[test]
fn runtime_patch_rejects_explicit_desired_hostname_change() {
let current = config_with_port_forwards(vec![port_forward(23000, 5174)]);
let mut desired =
config_with_port_forwards(vec![port_forward(23000, 5174), port_forward(23007, 3389)]);
fn runtime_reconcile_hot_patches_explicit_desired_hostname_change() {
let current = config_with_port_forwards(Vec::new());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some("desired-host".to_string());
let patch = web_source_runtime_patch(&current, &desired).expect("build patch");
let action = prepare_web_source_runtime_reconcile_from_current(&current, desired)
.expect("prepare reconcile");
let RuntimeReconcileAction::Patch(patch) = action else {
panic!("hostname-only change should use a hot patch");
};
assert!(patch.is_none());
assert_eq!(patch.hostname.as_deref(), Some("desired-host"));
}
#[test]
fn runtime_convergence_accepts_hostname_only_readback_difference() {
let current = config_with_port_forwards(Vec::new());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some("device-host".to_string());
ensure_runtime_config_converged(&current, &desired, true)
.expect("hostname-only readback difference should be converged");
}
#[test]
fn runtime_convergence_rejects_omitted_hostname_before_apply() {
let current = config_with_port_forwards(Vec::new());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some("device-host".to_string());
let err = ensure_runtime_config_converged(&current, &desired, false)
.expect_err("omitted hostname before apply should not converge");
assert!(
err.to_string()
.contains("runtime config still needs patch after reconcile")
);
}
#[test]
fn runtime_convergence_rejects_explicit_wrong_hostname_after_apply() {
let mut current = config_with_port_forwards(Vec::new());
current.hostname = Some("wrong-host".to_string());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some("device-host".to_string());
let err = ensure_runtime_config_converged(&current, &desired, true)
.expect_err("explicit wrong hostname should not converge");
assert!(
err.to_string()
.contains("runtime config still needs patch after reconcile")
);
}
#[test]
fn runtime_patch_clears_explicit_desired_hostname() {
let mut current = config_with_port_forwards(Vec::new());
current.hostname = Some("runtime-host".to_string());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some(String::new());
let patch = web_source_runtime_patch(&current, &desired)
.expect("build patch")
.expect("hot patch");
assert_eq!(patch.hostname.as_deref(), Some(""));
}
#[test]
fn runtime_patch_normalizes_missing_runtime_hostname_for_explicit_clear() {
let current = config_with_port_forwards(Vec::new());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some(String::new());
let action = prepare_web_source_runtime_reconcile_from_current(&current, desired)
.expect("prepare reconcile");
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
}
#[test]
fn runtime_patch_skips_matching_explicit_hostname() {
let mut current = config_with_port_forwards(Vec::new());
current.hostname = Some("desired-host".to_string());
let desired = current.clone();
let action = prepare_web_source_runtime_reconcile_from_current(&current, desired)
.expect("prepare reconcile");
assert!(matches!(action, RuntimeReconcileAction::Unchanged(_)));
}
#[test]
fn runtime_patch_uses_core_normalized_hostname() {
let current = config_with_port_forwards(Vec::new());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some("a".repeat(33));
let patch = web_source_runtime_patch(&current, &desired)
.expect("build patch")
.expect("hot patch");
assert_eq!(patch.hostname.as_deref(), Some("a".repeat(32).as_str()));
}
#[test]
fn runtime_patch_removes_hostname_control_characters() {
let current = config_with_port_forwards(Vec::new());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some("node\u{7}-name".to_string());
let patch = web_source_runtime_patch(&current, &desired)
.expect("build patch")
.expect("hot patch");
assert_eq!(patch.hostname.as_deref(), Some("node-name"));
}
#[test]
fn runtime_patch_normalizes_control_only_hostname_to_clear() {
let mut current = config_with_port_forwards(Vec::new());
current.hostname = Some("runtime-host".to_string());
let mut desired = config_with_port_forwards(Vec::new());
desired.hostname = Some("\u{7}\n".to_string());
let patch = web_source_runtime_patch(&current, &desired)
.expect("build patch")
.expect("hot patch");
assert_eq!(patch.hostname.as_deref(), Some(""));
}
}
easytier-web/src/client_manager/session.rs
+978 -149
View File
File diff suppressed because it is too large. Load diff
easytier-web/src/client_manager/session/runtime_revision.rs
+822 -180
View File
File diff suppressed because it is too large. Load diff
easytier-web/src/client_manager/session/webhook_validation.rs
+490 -44
View File
@@ -25,6 +25,8 @@ pub(super) struct WebhookValidationInput {
pub(super) webhook_config: SharedWebhookConfig,
pub(super) client_url: url::Url,
pub(super) applied_config_revision: Option<String>,
pub(super) applied_config_revision_known: bool,
pub(super) failed_instance_ids: Vec<String>,
pub(super) req: HeartbeatRequest,
pub(super) machine_id: uuid::Uuid,
}
@@ -40,28 +42,39 @@ pub(super) fn retry_delay(machine_id: uuid::Uuid) -> Duration {
}
async fn request_heartbeat_validation(
webhook_config: &crate::webhook::WebhookConfig,
client_url: &url::Url,
input: &WebhookValidationInput,
persisted_config_revision: Option<&str>,
applied_config_revision: Option<&str>,
req: &HeartbeatRequest,
machine_id: uuid::Uuid,
) -> anyhow::Result<Option<WebhookHeartbeatValidation>> {
let webhook_req = crate::webhook::ValidateTokenRequest {
token: req.user_token.clone(),
machine_id: machine_id.to_string(),
public_ip: client_url.host_str().map(str::to_string),
hostname: req.hostname.clone(),
version: req.easytier_version.clone(),
os_type: req.device_os.as_ref().map(|info| info.os_type.clone()),
os_version: req.device_os.as_ref().map(|info| info.version.clone()),
os_distribution: req.device_os.as_ref().map(|info| info.distribution.clone()),
web_instance_id: webhook_config.web_instance_id.clone(),
web_instance_api_base_url: webhook_config.web_instance_api_base_url.clone(),
token: input.req.user_token.clone(),
machine_id: input.machine_id.to_string(),
public_ip: input.client_url.host_str().map(str::to_string),
hostname: input.req.hostname.clone(),
version: input.req.easytier_version.clone(),
os_type: input
.req
.device_os
.as_ref()
.map(|info| info.os_type.clone()),
os_version: input
.req
.device_os
.as_ref()
.map(|info| info.version.clone()),
os_distribution: input
.req
.device_os
.as_ref()
.map(|info| info.distribution.clone()),
web_instance_id: input.webhook_config.web_instance_id.clone(),
web_instance_api_base_url: input.webhook_config.web_instance_api_base_url.clone(),
persisted_config_revision: persisted_config_revision.map(str::to_string),
applied_config_revision: applied_config_revision.map(str::to_string),
applied_config_revision: input.applied_config_revision.as_deref().map(str::to_string),
applied_config_revision_known: input.applied_config_revision_known,
failed_instance_ids: input.failed_instance_ids.to_vec(),
};
let resp = webhook_config
let resp = input
.webhook_config
.validate_token(&webhook_req)
.await
.map_err(|e| anyhow::anyhow!("Webhook token validation failed: {:?}", e))?;
@@ -87,7 +100,7 @@ async fn resolve_user_id(storage: &Storage, token: &str) -> anyhow::Result<i32>
None => storage
.auto_create_user(token)
.await
.with_context(|| format!("Failed to auto-create webhook user: {:?}", token))?,
.with_context(|| "Failed to auto-create webhook user".to_string())?,
};
Ok(user_id)
@@ -115,7 +128,7 @@ async fn persisted_config_revision_for_token(
async fn wait_for_input(
session_data: std::sync::Weak<RwLock<SessionData>>,
) -> Option<WebhookValidationInput> {
) -> Option<(WebhookValidationInput, u64)> {
loop {
let notify = {
let session_data = session_data.upgrade()?;
@@ -133,14 +146,28 @@ async fn wait_for_input(
let req = data.req.clone()?;
let machine_id = req.machine_id.map(Into::into)?;
let storage = Storage::try_from(data.storage.clone()).ok()?;
return Some(WebhookValidationInput {
storage,
webhook_config: data.webhook_config.clone(),
client_url: data.client_url.clone(),
applied_config_revision: data.applied_config_revision.clone(),
req,
machine_id,
});
let (applied_config_revision, applied_config_revision_known) = {
let runtime = data.managed_runtime();
(
runtime.applied_config_revision.clone(),
runtime.applied_config_revision_known,
)
};
return Some((
WebhookValidationInput {
storage,
webhook_config: data.webhook_config.clone(),
client_url: data.client_url.clone(),
applied_config_revision,
applied_config_revision_known,
failed_instance_ids: SessionRpcService::sorted_failed_instance_ids_locked(
&data,
),
req,
machine_id,
},
data.webhook_validation_change_epoch,
));
}
data.webhook_validation_notify.clone()
};
@@ -148,17 +175,64 @@ async fn wait_for_input(
}
}
async fn wait_for_retry_or_state_change(
session_data: &std::sync::Weak<RwLock<SessionData>>,
machine_id: uuid::Uuid,
validation_change_epoch: u64,
delay: Duration,
) {
let retry_deadline = tokio::time::sleep(delay);
tokio::pin!(retry_deadline);
loop {
let notify = {
let Some(session_data) = session_data.upgrade() else {
return;
};
let data = session_data.read().await;
let Some(req) = data.req.as_ref() else {
return;
};
if req.machine_id.map(uuid::Uuid::from) != Some(machine_id)
|| matches!(data.auth_state, SessionAuthState::Invalid)
{
return;
}
if data.webhook_validation_change_epoch != validation_change_epoch {
return;
}
data.webhook_validation_notify.clone()
};
// Notify is only a wake-up hint. Periodic validation can set dirty,
// but only a meaningful validation-state change may bypass backoff.
// Recheck the epoch after every wake without resetting the deadline.
tokio::select! {
_ = &mut retry_deadline => {
mark_dirty_if_current(session_data, machine_id).await;
return;
}
_ = notify.notified() => {}
}
}
}
pub(super) async fn run_worker(session_data: std::sync::Weak<RwLock<SessionData>>) {
while let Some(input) = wait_for_input(session_data.clone()).await {
while let Some((input, validation_change_epoch)) = wait_for_input(session_data.clone()).await {
let machine_id = input.machine_id;
if let Err(error) = run_round(session_data.clone(), input).await {
if let Err(error) = run_round(session_data.clone(), input, validation_change_epoch).await {
tracing::warn!(
?machine_id,
%error,
"webhook validation failed, will retry later"
);
tokio::time::sleep(retry_delay(machine_id)).await;
mark_dirty_if_current(&session_data, machine_id).await;
wait_for_retry_or_state_change(
&session_data,
machine_id,
validation_change_epoch,
retry_delay(machine_id),
)
.await;
}
}
}
@@ -166,6 +240,7 @@ pub(super) async fn run_worker(session_data: std::sync::Weak<RwLock<SessionData>
pub(super) async fn run_round(
session_data: std::sync::Weak<RwLock<SessionData>>,
input: WebhookValidationInput,
validation_change_epoch: u64,
) -> anyhow::Result<()> {
let persisted_config_revision = persisted_config_revision_for_token(
&input.storage,
@@ -173,26 +248,53 @@ pub(super) async fn run_round(
input.machine_id,
)
.await?;
let validation = request_heartbeat_validation(
&input.webhook_config,
&input.client_url,
persisted_config_revision.as_deref(),
input.applied_config_revision.as_deref(),
&input.req,
input.machine_id,
)
.await?;
let validation =
request_heartbeat_validation(&input, persisted_config_revision.as_deref()).await?;
// The HTTP round trip can span heartbeats, revision updates, and
// failed-instance changes. Results older than the current epoch are
// discarded so a stale rejection cannot invalidate the session and a
// stale success cannot emit outdated connection transitions.
if !validation_results_are_current(&session_data, &input, validation_change_epoch).await {
return Ok(());
}
let Some(validation) = validation else {
apply_rejected(&session_data, &input).await;
apply_rejected(&session_data, &input, validation_change_epoch).await;
return Ok(());
};
let user_id = resolve_user_id(&input.storage, &input.req.user_token).await?;
apply_success(&session_data, input, validation, user_id).await;
apply_success(
&session_data,
input,
validation,
user_id,
validation_change_epoch,
)
.await;
Ok(())
}
async fn validation_results_are_current(
session_data: &std::sync::Weak<RwLock<SessionData>>,
input: &WebhookValidationInput,
validation_change_epoch: u64,
) -> bool {
let Some(session_data) = session_data.upgrade() else {
return false;
};
let data = session_data.read().await;
if data.webhook_validation_change_epoch != validation_change_epoch {
tracing::debug!(
machine_id = %input.machine_id,
"discard stale webhook validation result"
);
return false;
}
true
}
async fn mark_dirty_if_current(
session_data: &std::sync::Weak<RwLock<SessionData>>,
machine_id: uuid::Uuid,
@@ -224,6 +326,7 @@ async fn mark_dirty_if_current(
pub(super) async fn apply_rejected(
session_data: &std::sync::Weak<RwLock<SessionData>>,
input: &WebhookValidationInput,
validation_change_epoch: u64,
) {
let Some(session_data) = session_data.upgrade() else {
return;
@@ -239,6 +342,13 @@ pub(super) async fn apply_rejected(
}) {
return;
}
if data.webhook_validation_change_epoch != validation_change_epoch {
tracing::debug!(
machine_id = %input.machine_id,
"discard stale webhook validation rejection"
);
return;
}
tracing::info!(
machine_id = %input.machine_id,
client_url = %data.client_url,
@@ -247,8 +357,6 @@ pub(super) async fn apply_rejected(
data.auth_state = SessionAuthState::Invalid;
data.webhook_validation_dirty = false;
data.binding_version = None;
data.applied_config_revision = None;
data.pending_managed_config_delta = None;
let storage_token = data.storage_token.clone();
let disconnect_notification = storage_token.as_ref().and_then(|storage_token| {
data.webhook_connected_binding_version
@@ -282,6 +390,7 @@ pub(super) async fn apply_success(
input: WebhookValidationInput,
validation: WebhookHeartbeatValidation,
user_id: i32,
validation_change_epoch: u64,
) {
let WebhookHeartbeatValidation {
config_revision: _,
@@ -296,6 +405,7 @@ pub(super) async fn apply_success(
notifier,
disconnect_notification,
connect_notification,
validation_notify,
runtime_req,
session_epoch,
) = {
@@ -310,6 +420,13 @@ pub(super) async fn apply_success(
) {
return;
}
if data.webhook_validation_change_epoch != validation_change_epoch {
tracing::debug!(
machine_id = %input.machine_id,
"discard stale webhook validation success"
);
return;
}
if matches!(data.auth_state, SessionAuthState::Invalid) {
tracing::info!(
machine_id = %input.machine_id,
@@ -321,6 +438,20 @@ pub(super) async fn apply_success(
let previous_connected_binding_version = data.webhook_connected_binding_version;
let client_url = data.client_url.clone();
let is_new_storage_token = data.storage_token.is_none();
let mut restored_runtime_revision = false;
if is_new_storage_token {
data.managed_runtime = input.storage.bind_managed_runtime_state(
user_id,
input.machine_id,
SessionRpcService::heartbeat_runtime_id(&runtime_req),
data.session_epoch,
);
let runtime = data.managed_runtime();
restored_runtime_revision = runtime.applied_config_revision_known
&& (!input.applied_config_revision_known
|| input.applied_config_revision != runtime.applied_config_revision);
}
let storage_token = data.storage_token.get_or_insert_with(|| StorageToken {
token: runtime_req.user_token.clone(),
client_url,
@@ -330,6 +461,16 @@ pub(super) async fn apply_success(
let storage_token = storage_token.clone();
data.auth_state = SessionAuthState::Authorized;
data.binding_version = Some(binding_version);
if is_new_storage_token {
tracing::info!(
machine_id = %input.machine_id,
user_id,
session_epoch = data.session_epoch,
binding_version,
client_url = %data.client_url,
"session identity established"
);
}
let should_notify_connected = previous_connected_binding_version != Some(binding_version);
let disconnect_notification = previous_connected_binding_version
.filter(|previous_binding_version| *previous_binding_version != binding_version)
@@ -368,12 +509,15 @@ pub(super) async fn apply_success(
binding_version: Some(binding_version),
},
});
let validation_notify = restored_runtime_revision
.then(|| SessionRpcService::mark_webhook_validation_state_changed_locked(&mut data));
(
storage_token,
data.notifier.clone(),
disconnect_notification,
connect_notification,
validation_notify,
runtime_req,
data.session_epoch,
)
@@ -384,6 +528,10 @@ pub(super) async fn apply_success(
.storage
.update_session_client(storage_token, report_time, true, session_epoch);
if let Some(validation_notify) = validation_notify {
validation_notify.notify_one();
}
if disconnect_notification.is_some() || connect_notification.is_some() {
wait_webhook_connection_transition(
Arc::downgrade(&session_data),
@@ -410,3 +558,301 @@ async fn wait_webhook_connection_transition(
tracing::warn!(%error, "webhook connection transition task failed");
}
}
#[cfg(test)]
mod tests {
use super::*;
async fn validation_session(machine_id: uuid::Uuid) -> Arc<RwLock<SessionData>> {
let storage = Storage::new(crate::db::Db::memory_db().await);
let mut data = SessionData::new(
storage.weak_ref(),
url::Url::parse("http://127.0.0.1").unwrap(),
None,
Arc::new(crate::FeatureFlags::default()),
Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
)),
);
data.req = Some(HeartbeatRequest {
user_token: "token".to_string(),
machine_id: Some(machine_id.into()),
..Default::default()
});
data.auth_state = SessionAuthState::Authorized;
Arc::new(RwLock::new(data))
}
#[tokio::test]
async fn reconnect_immediately_reports_restored_runtime_revision() {
let storage = Storage::new(crate::db::Db::memory_db().await);
let user_id = storage.db().auto_create_user("token").await.unwrap().id;
let machine_id = uuid::Uuid::new_v4();
let runtime_id = uuid::Uuid::new_v4();
let shared = storage.bind_managed_runtime_state(user_id, machine_id, Some(runtime_id), 1);
{
let mut runtime = shared.lock().unwrap();
runtime.applied_config_revision = Some("rev-applied".to_string());
runtime.applied_config_revision_known = true;
}
let request = HeartbeatRequest {
user_token: "token".to_string(),
machine_id: Some(machine_id.into()),
inst_id: Some(runtime_id.into()),
..Default::default()
};
let mut data = SessionData::new(
storage.weak_ref(),
url::Url::parse("http://127.0.0.1").unwrap(),
None,
Arc::new(crate::FeatureFlags::default()),
Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
)),
);
data.req = Some(request.clone());
data.session_identity = Some(SessionRpcService::heartbeat_identity(&request, machine_id));
data.session_epoch = 2;
let session_data = Arc::new(RwLock::new(data));
let validation_change_epoch = session_data.read().await.webhook_validation_change_epoch;
apply_success(
&Arc::downgrade(&session_data),
WebhookValidationInput {
storage,
webhook_config: Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
)),
client_url: url::Url::parse("http://127.0.0.1").unwrap(),
applied_config_revision: None,
applied_config_revision_known: false,
failed_instance_ids: Vec::new(),
req: request,
machine_id,
},
WebhookHeartbeatValidation {
config_revision: "rev-applied".to_string(),
binding_version: 1,
},
user_id,
validation_change_epoch,
)
.await;
let data = session_data.read().await;
assert!(Arc::ptr_eq(&data.managed_runtime, &shared));
assert!(data.webhook_validation_dirty);
assert_eq!(data.webhook_validation_change_epoch, 1);
}
#[tokio::test]
async fn validation_input_carries_merged_failed_instance_ids() {
let machine_id = uuid::Uuid::new_v4();
let core_failed = uuid::Uuid::new_v4();
let local_failed = uuid::Uuid::new_v4().to_string();
let session_data = validation_session(machine_id).await;
let storage = Storage::new(crate::db::Db::memory_db().await);
{
let mut data = session_data.write().await;
data.storage = storage.weak_ref();
data.req
.as_mut()
.unwrap()
.failed_network_instances
.push(core_failed.into());
data.direct_run_failed_instance_ids
.insert(local_failed.clone());
data.webhook_validation_dirty = true;
}
let (input, _) = wait_for_input(Arc::downgrade(&session_data))
.await
.expect("validation input");
let mut expected = vec![core_failed.to_string(), local_failed];
expected.sort_unstable();
assert_eq!(input.failed_instance_ids, expected);
}
#[tokio::test]
async fn stale_notification_does_not_bypass_validation_retry_delay() {
let machine_id = uuid::Uuid::new_v4();
let session_data = validation_session(machine_id).await;
let notify = session_data.read().await.webhook_validation_notify.clone();
notify.notify_one();
let weak_session = Arc::downgrade(&session_data);
let wait =
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10));
tokio::pin!(wait);
assert!(
tokio::time::timeout(Duration::from_millis(20), &mut wait)
.await
.is_err()
);
assert!(!session_data.read().await.webhook_validation_dirty);
}
#[tokio::test]
async fn validation_retry_deadline_rearms_dirty_state() {
let machine_id = uuid::Uuid::new_v4();
let session_data = validation_session(machine_id).await;
let weak_session = Arc::downgrade(&session_data);
tokio::time::timeout(
Duration::from_secs(1),
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_millis(20)),
)
.await
.expect("retry deadline should eventually expire");
assert!(session_data.read().await.webhook_validation_dirty);
}
#[tokio::test]
async fn periodic_dirty_state_does_not_interrupt_validation_retry_delay() {
let machine_id = uuid::Uuid::new_v4();
let session_data = validation_session(machine_id).await;
let weak_session = Arc::downgrade(&session_data);
let wait =
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10));
tokio::pin!(wait);
assert!(
tokio::time::timeout(Duration::from_millis(20), &mut wait)
.await
.is_err()
);
mark_dirty_if_current(&weak_session, machine_id).await;
assert!(
tokio::time::timeout(Duration::from_millis(20), &mut wait)
.await
.is_err()
);
assert!(session_data.read().await.webhook_validation_dirty);
}
#[tokio::test]
async fn validation_state_change_interrupts_retry_delay() {
let machine_id = uuid::Uuid::new_v4();
let session_data = validation_session(machine_id).await;
let weak_session = Arc::downgrade(&session_data);
let wait =
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10));
tokio::pin!(wait);
assert!(
tokio::time::timeout(Duration::from_millis(20), &mut wait)
.await
.is_err()
);
let notify = {
let mut data = session_data.write().await;
SessionRpcService::mark_webhook_validation_state_changed_locked(&mut data)
};
notify.notify_one();
tokio::time::timeout(Duration::from_millis(500), &mut wait)
.await
.expect("validation state change should interrupt retry delay");
assert!(session_data.read().await.webhook_validation_dirty);
}
#[tokio::test]
async fn invalid_session_does_not_rearm_validation_retry() {
let machine_id = uuid::Uuid::new_v4();
let session_data = validation_session(machine_id).await;
session_data.write().await.auth_state = SessionAuthState::Invalid;
let weak_session = Arc::downgrade(&session_data);
tokio::time::timeout(
Duration::from_millis(100),
wait_for_retry_or_state_change(&weak_session, machine_id, 0, Duration::from_secs(10)),
)
.await
.expect("invalid session should stop waiting");
assert!(!session_data.read().await.webhook_validation_dirty);
}
#[tokio::test]
async fn validation_results_require_current_epoch_or_live_session() {
let machine_id = uuid::Uuid::new_v4();
let session_data = validation_session(machine_id).await;
let input = WebhookValidationInput {
storage: Storage::new(crate::db::Db::memory_db().await),
webhook_config: Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
)),
client_url: url::Url::parse("http://127.0.0.1").unwrap(),
applied_config_revision: None,
applied_config_revision_known: false,
failed_instance_ids: Vec::new(),
req: HeartbeatRequest::default(),
machine_id,
};
let weak_session = Arc::downgrade(&session_data);
assert!(
validation_results_are_current(&weak_session, &input, 0).await,
"matching epoch is current"
);
session_data.write().await.webhook_validation_change_epoch = 7;
assert!(
!validation_results_are_current(&weak_session, &input, 0).await,
"epoch bump discards stale results"
);
drop(session_data);
assert!(
!validation_results_are_current(&weak_session, &input, 7).await,
"dropped session discards results"
);
}
#[tokio::test]
async fn apply_paths_discard_results_from_stale_epochs() {
let machine_id = uuid::Uuid::new_v4();
let session_data = validation_session(machine_id).await;
session_data.write().await.webhook_connected_binding_version = Some(3);
let input = WebhookValidationInput {
storage: Storage::new(crate::db::Db::memory_db().await),
webhook_config: Arc::new(crate::webhook::WebhookConfig::new(
None, None, None, None, None,
)),
client_url: url::Url::parse("http://127.0.0.1").unwrap(),
applied_config_revision: None,
applied_config_revision_known: false,
failed_instance_ids: Vec::new(),
req: HeartbeatRequest {
user_token: "token".to_string(),
machine_id: Some(machine_id.into()),
..Default::default()
},
machine_id,
};
let stale_epoch = session_data.read().await.webhook_validation_change_epoch;
session_data.write().await.webhook_validation_change_epoch = stale_epoch + 1;
apply_rejected(&Arc::downgrade(&session_data), &input, stale_epoch).await;
let data = session_data.read().await;
assert_eq!(data.auth_state, SessionAuthState::Authorized);
assert_eq!(data.webhook_connected_binding_version, Some(3));
drop(data);
apply_success(
&Arc::downgrade(&session_data),
input,
WebhookHeartbeatValidation {
config_revision: "rev-1".to_string(),
binding_version: 9,
},
1,
stale_epoch,
)
.await;
let data = session_data.read().await;
assert_eq!(data.binding_version, None);
assert_eq!(data.webhook_connected_binding_version, Some(3));
}
}
easytier-web/src/client_manager/storage.rs
+343 -8
View File
@@ -1,9 +1,14 @@
use std::sync::{Arc, Weak};
use dashmap::DashMap;
use dashmap::{DashMap, mapref::entry::Entry};
use crate::db::{Db, UserIdInDb};
use super::session::{
ManagedConfigPersistedChange, ManagedConfigReconcileHint, ManagedRuntimeState,
SharedManagedRuntimeState, record_managed_config_reconcile_hint,
};
// use this to maintain Storage
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct StorageToken {
@@ -21,12 +26,46 @@ struct ClientInfo {
session_epoch: u64,
}
#[derive(Debug, Clone)]
struct ManagedRuntimeContinuity {
// Accepted trade-off: continuity assumes managed configuration is only
// mutated through easytier-web. A local management RPC can make Core
// drift without invalidating this state; detecting that would require a
// Core-wide mutation generation outside this compatibility path.
runtime_id: Option<uuid::Uuid>,
session_epoch: u64,
state: SharedManagedRuntimeState,
}
#[derive(Debug)]
pub struct StorageInner {
user_clients_map: DashMap<UserIdInDb, DashMap<uuid::Uuid, ClientInfo>>,
managed_runtime_states: DashMap<(UserIdInDb, uuid::Uuid), ManagedRuntimeContinuity>,
pub db: Db,
}
impl StorageInner {
pub(super) fn owns_authorized_session(
&self,
stoken: &StorageToken,
session_epoch: u64,
) -> bool {
self.user_clients_map
.get(&stoken.user_id)
.and_then(|clients| {
clients.get(&stoken.machine_id).map(|client| {
client.authorized
&& client.session_epoch == session_epoch
&& client.storage_token.token == stoken.token
&& client.storage_token.client_url == stoken.client_url
&& client.storage_token.user_id == stoken.user_id
&& client.storage_token.machine_id == stoken.machine_id
})
})
.unwrap_or(false)
}
}
#[derive(Debug, Clone)]
pub struct Storage(Arc<StorageInner>);
pub type WeakRefStorage = Weak<StorageInner>;
@@ -43,20 +82,149 @@ impl Storage {
pub fn new(db: Db) -> Self {
Storage(Arc::new(StorageInner {
user_clients_map: DashMap::new(),
managed_runtime_states: DashMap::new(),
db,
}))
}
pub(super) fn bind_managed_runtime_state(
&self,
user_id: UserIdInDb,
machine_id: uuid::Uuid,
runtime_id: Option<uuid::Uuid>,
session_epoch: u64,
) -> SharedManagedRuntimeState {
let new_state = || Arc::new(std::sync::Mutex::new(ManagedRuntimeState::default()));
match self.0.managed_runtime_states.entry((user_id, machine_id)) {
Entry::Occupied(mut entry) => {
let current = entry.get();
if runtime_id.is_some() && current.runtime_id == runtime_id {
let state = current.state.clone();
if session_epoch > current.session_epoch {
entry.get_mut().session_epoch = session_epoch;
}
return state;
}
if session_epoch < current.session_epoch {
return new_state();
}
let state = new_state();
entry.insert(ManagedRuntimeContinuity {
runtime_id,
session_epoch,
state: state.clone(),
});
state
}
Entry::Vacant(entry) => {
let state = new_state();
entry.insert(ManagedRuntimeContinuity {
runtime_id,
session_epoch,
state: state.clone(),
});
state
}
}
}
fn current_managed_runtime_state(
&self,
user_id: UserIdInDb,
machine_id: uuid::Uuid,
) -> Option<SharedManagedRuntimeState> {
self.0
.managed_runtime_states
.get(&(user_id, machine_id))
.map(|entry| entry.state.clone())
}
pub(super) fn record_full_managed_config_change(
&self,
user_id: UserIdInDb,
machine_id: uuid::Uuid,
config_revision: Option<&str>,
) -> bool {
let Some(state) = self.current_managed_runtime_state(user_id, machine_id) else {
return false;
};
let mut state = state.lock().expect("managed runtime state lock poisoned");
// Unrevisioned legacy updates carry no revision to compare against,
// so they always invalidate: record the hint and bump the epoch that
// fences in-flight reconcile rounds.
let target_already_applied = config_revision
.is_some_and(|revision| state.applied_config_revision.as_deref() == Some(revision));
if target_already_applied && state.pending_managed_config_reconcile.is_none() {
return false;
}
if !target_already_applied {
record_managed_config_reconcile_hint(
&mut state.pending_managed_config_reconcile,
ManagedConfigReconcileHint::Full,
);
}
state.runtime_config_epoch = state.runtime_config_epoch.wrapping_add(1);
true
}
pub(super) fn record_patch_managed_config_change(
&self,
user_id: UserIdInDb,
machine_id: uuid::Uuid,
change: ManagedConfigPersistedChange,
) -> bool {
let Some(state) = self.current_managed_runtime_state(user_id, machine_id) else {
return false;
};
let mut state = state.lock().expect("managed runtime state lock poisoned");
let target_already_applied =
state.applied_config_revision.as_deref() == Some(change.target_revision.as_str());
if target_already_applied && state.pending_managed_config_reconcile.is_none() {
return false;
}
if !target_already_applied {
record_managed_config_reconcile_hint(
&mut state.pending_managed_config_reconcile,
ManagedConfigReconcileHint::Dirty {
expected_revision: change.expected_revision,
target_revision: change.target_revision,
instance_ids: change.dirty_instance_ids,
},
);
}
state.runtime_config_epoch = state.runtime_config_epoch.wrapping_add(1);
true
}
pub(super) fn invalidate_managed_runtime_state(
&self,
user_id: UserIdInDb,
machine_id: uuid::Uuid,
) -> bool {
let Some(state) = self.current_managed_runtime_state(user_id, machine_id) else {
return false;
};
let mut state = state.lock().expect("managed runtime state lock poisoned");
state.applied_config_revision = None;
state.applied_config_revision_known = true;
state.known_runtime_base_revision = None;
state.pending_managed_config_reconcile = Some(ManagedConfigReconcileHint::Full);
state.runtime_config_epoch = state.runtime_config_epoch.wrapping_add(1);
state.runtime_config_cache_epoch = state.runtime_config_cache_epoch.wrapping_add(1);
true
}
fn remove_client_info_map(
map: &DashMap<uuid::Uuid, ClientInfo>,
stoken: &StorageToken,
session_epoch: u64,
) {
) -> bool {
map.remove_if(&stoken.machine_id, |_, v| {
v.storage_token.client_url == stoken.client_url
&& v.storage_token.user_id == stoken.user_id
&& v.session_epoch == session_epoch
});
})
.is_some()
}
fn update_client_info_map(map: &DashMap<uuid::Uuid, ClientInfo>, client_info: &ClientInfo) {
@@ -99,6 +267,20 @@ impl Storage {
authorized: bool,
session_epoch: u64,
) {
let mut continuity = self
.0
.managed_runtime_states
.entry((stoken.user_id, stoken.machine_id))
.or_insert_with(|| ManagedRuntimeContinuity {
runtime_id: None,
session_epoch,
state: Arc::new(std::sync::Mutex::new(ManagedRuntimeState::default())),
});
if session_epoch < continuity.session_epoch {
return;
}
continuity.session_epoch = session_epoch;
let inner = self.0.user_clients_map.entry(stoken.user_id).or_default();
let client_info = ClientInfo {
@@ -111,16 +293,30 @@ impl Storage {
}
pub fn remove_client(&self, stoken: &StorageToken) {
self.remove_session_client(stoken, 0);
let _ = self.remove_session_client(stoken, 0);
}
pub(super) fn remove_session_client(&self, stoken: &StorageToken, session_epoch: u64) {
pub(super) fn remove_session_client(&self, stoken: &StorageToken, session_epoch: u64) -> bool {
let Some(mut continuity) = self
.0
.managed_runtime_states
.get_mut(&(stoken.user_id, stoken.machine_id))
else {
return false;
};
if session_epoch < continuity.session_epoch {
return false;
}
continuity.session_epoch = session_epoch;
let mut removed = false;
self.0
.user_clients_map
.remove_if(&stoken.user_id, |_, set| {
Self::remove_client_info_map(set, stoken, session_epoch);
removed = Self::remove_client_info_map(set, stoken, session_epoch);
set.is_empty()
});
removed
}
pub fn weak_ref(&self) -> WeakRefStorage {
@@ -262,6 +458,36 @@ mod tests {
assert_eq!(storage.get_client_url_by_machine_id(2, &machine_id), None);
}
#[tokio::test]
async fn unrevisioned_full_change_always_fences_runtime_epochs() {
let storage = Storage::new(Db::memory_db().await);
let machine_id = uuid::Uuid::new_v4();
let state = storage.bind_managed_runtime_state(7, machine_id, None, 1);
{
let mut runtime = state.lock().unwrap();
runtime.applied_config_revision = Some("rev-1".to_string());
runtime.applied_config_revision_known = true;
}
assert!(storage.record_full_managed_config_change(7, machine_id, None));
{
let runtime = state.lock().unwrap();
assert_eq!(runtime.runtime_config_epoch, 1);
assert!(matches!(
runtime.pending_managed_config_reconcile,
Some(ManagedConfigReconcileHint::Full)
));
}
// Repeated unrevisioned updates keep fencing, while a revision that
// is already applied with no pending hint stays a no-op.
assert!(storage.record_full_managed_config_change(7, machine_id, None));
assert_eq!(state.lock().unwrap().runtime_config_epoch, 2);
state.lock().unwrap().pending_managed_config_reconcile = None;
assert!(!storage.record_full_managed_config_change(7, machine_id, Some("rev-1")));
assert_eq!(state.lock().unwrap().runtime_config_epoch, 2);
}
#[tokio::test]
async fn newer_session_epoch_owns_route_until_it_is_removed() {
let storage = Storage::new(Db::memory_db().await);
@@ -273,18 +499,127 @@ mod tests {
storage.update_session_client(current.clone(), 20, true, 2);
storage.update_session_client(old.clone(), 30, true, 1);
assert!(!storage.0.owns_authorized_session(&old, 1));
assert!(storage.0.owns_authorized_session(&current, 2));
assert_eq!(
storage.get_client_url_by_machine_id(1, &machine_id),
Some(current.client_url.clone())
);
storage.remove_session_client(&old, 1);
assert!(!storage.remove_session_client(&old, 1));
assert_eq!(
storage.get_client_url_by_machine_id(1, &machine_id),
Some(current.client_url.clone())
);
storage.remove_session_client(&current, 2);
assert!(storage.remove_session_client(&current, 2));
assert_eq!(storage.get_client_url_by_machine_id(1, &machine_id), None);
storage.update_session_client(old.clone(), 40, true, 1);
assert!(!storage.0.owns_authorized_session(&old, 1));
assert_eq!(storage.get_client_url_by_machine_id(1, &machine_id), None);
}
#[tokio::test]
async fn same_runtime_reuses_state_across_sessions() {
let storage = Storage::new(Db::memory_db().await);
let machine_id = uuid::Uuid::new_v4();
let runtime_id = uuid::Uuid::new_v4();
let first = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 1);
{
let mut state = first.lock().unwrap();
state.applied_config_revision = Some("rev-a".to_string());
state.applied_config_revision_known = true;
}
let reconnected = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 2);
assert!(Arc::ptr_eq(&first, &reconnected));
let state = reconnected.lock().unwrap();
assert_eq!(state.applied_config_revision.as_deref(), Some("rev-a"));
assert!(state.applied_config_revision_known);
}
#[tokio::test]
async fn changed_or_missing_runtime_id_starts_with_unknown_state() {
let storage = Storage::new(Db::memory_db().await);
let machine_id = uuid::Uuid::new_v4();
let first =
storage.bind_managed_runtime_state(1, machine_id, Some(uuid::Uuid::new_v4()), 1);
{
let mut state = first.lock().unwrap();
state.applied_config_revision = Some("rev-a".to_string());
state.applied_config_revision_known = true;
}
let restarted =
storage.bind_managed_runtime_state(1, machine_id, Some(uuid::Uuid::new_v4()), 2);
assert!(!Arc::ptr_eq(&first, &restarted));
assert!(!restarted.lock().unwrap().applied_config_revision_known);
let legacy = storage.bind_managed_runtime_state(1, machine_id, None, 3);
assert!(!Arc::ptr_eq(&restarted, &legacy));
let legacy_reconnected = storage.bind_managed_runtime_state(1, machine_id, None, 4);
assert!(!Arc::ptr_eq(&legacy, &legacy_reconnected));
assert!(
!legacy_reconnected
.lock()
.unwrap()
.applied_config_revision_known
);
}
#[tokio::test]
async fn stale_session_cannot_replace_current_runtime_state() {
let storage = Storage::new(Db::memory_db().await);
let machine_id = uuid::Uuid::new_v4();
let current_runtime_id = uuid::Uuid::new_v4();
let current =
storage.bind_managed_runtime_state(1, machine_id, Some(current_runtime_id), 2);
let stale =
storage.bind_managed_runtime_state(1, machine_id, Some(uuid::Uuid::new_v4()), 1);
assert!(!Arc::ptr_eq(&current, &stale));
let reconnected =
storage.bind_managed_runtime_state(1, machine_id, Some(current_runtime_id), 3);
assert!(Arc::ptr_eq(&current, &reconnected));
}
#[tokio::test]
async fn patch_hint_survives_disconnect_until_same_runtime_reconnects() {
let storage = Storage::new(Db::memory_db().await);
let machine_id = uuid::Uuid::new_v4();
let runtime_id = uuid::Uuid::new_v4();
let state = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 1);
{
let mut state = state.lock().unwrap();
state.applied_config_revision = Some("rev-a".to_string());
state.applied_config_revision_known = true;
state.known_runtime_base_revision = Some("rev-a".to_string());
}
assert!(storage.record_patch_managed_config_change(
1,
machine_id,
ManagedConfigPersistedChange {
expected_revision: "rev-a".to_string(),
target_revision: "rev-b".to_string(),
dirty_instance_ids: std::collections::HashSet::from(["instance-a".to_string(),]),
},
));
let reconnected = storage.bind_managed_runtime_state(1, machine_id, Some(runtime_id), 2);
let state = reconnected.lock().unwrap();
assert_eq!(
state.pending_managed_config_reconcile,
Some(ManagedConfigReconcileHint::Dirty {
expected_revision: "rev-a".to_string(),
target_revision: "rev-b".to_string(),
instance_ids: std::collections::HashSet::from(["instance-a".to_string(),]),
})
);
assert_eq!(state.runtime_config_epoch, 1);
}
#[tokio::test]
easytier-web/src/main.rs
+34 -4
View File
@@ -3,8 +3,8 @@
#[macro_use]
extern crate rust_i18n;
use std::net::IpAddr;
use std::sync::Arc;
use std::{net::IpAddr, time::Duration};
use clap::Parser;
use easytier::tunnel::websocket::WsTunnelListener;
@@ -116,11 +116,19 @@ struct Cli {
#[arg(
long,
env = "ET_HEARTBEAT_MIN_RESPONSE_MS",
default_value = "0",
default_value = "3500",
help = t!("cli.heartbeat_min_response_ms").to_string(),
)]
heartbeat_min_response_ms: u64,
#[arg(
long,
env = "ET_HEARTBEAT_TIMEOUT_MS",
default_value = "15000",
help = t!("cli.heartbeat_timeout_ms").to_string(),
)]
heartbeat_timeout_ms: u64,
#[cfg(feature = "embed")]
#[arg(
long,
@@ -288,7 +296,21 @@ async fn main() {
setup_panic_handler();
let cli = Cli::parse();
log::init(&cli, false).unwrap();
log::init_with_default_console_targets(&cli, false, &["CORE", "easytier_web"]).unwrap();
tracing::info!(
version = EASYTIER_VERSION,
web_instance_id = ?cli.webhook.web_instance_id,
api_address = %cli.api_server_addr,
api_port = cli.api_server_port,
config_protocol = %cli.config_server_protocol,
config_port = cli.config_server_port,
heartbeat_min_response_ms = cli.heartbeat_min_response_ms,
heartbeat_timeout_ms = cli.heartbeat_timeout_ms,
webhook_enabled = cli.webhook.webhook_url.as_deref().is_some_and(|url| !url.trim().is_empty()),
rust_log_override = std::env::var_os("RUST_LOG").is_some(),
console_log_override = cli.console_log_level.is_some(),
"easytier-web starting"
);
// Validate OIDC configuration: check split-deploy specific requirements
// Basic OIDC parameter validation is handled in OidcConfig::from_params
@@ -326,10 +348,18 @@ async fn main() {
cli.webhook.web_instance_id,
cli.webhook.web_instance_api_base_url,
));
let heartbeat_policy = client_manager::HeartbeatPolicy::from_millis(
cli.heartbeat_min_response_ms,
cli.heartbeat_timeout_ms,
)
.unwrap_or_else(|error| {
eprintln!("Invalid heartbeat configuration: {error}");
std::process::exit(2);
});
let mut mgr = client_manager::ClientManager::new(
db.clone(),
cli.geoip_db,
Duration::from_millis(cli.heartbeat_min_response_ms),
heartbeat_policy,
feature_flags.clone(),
webhook_config.clone(),
);
easytier-web/src/webhook.rs
+112 -9
View File
@@ -308,6 +308,8 @@ pub struct ValidateTokenRequest {
pub persisted_config_revision: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub applied_config_revision: Option<String>,
pub applied_config_revision_known: bool,
pub failed_instance_ids: Vec<String>,
}
#[derive(Debug, Deserialize)]
@@ -350,6 +352,26 @@ pub struct NodeDisconnectedRequest {
pub binding_version: Option<u64>,
}
#[derive(Debug, thiserror::Error)]
pub(crate) enum WebhookDeliveryError {
#[error("webhook endpoint is invalid: {0}")]
Configuration(#[source] anyhow::Error),
#[error("webhook request failed: {0}")]
Transport(#[source] reqwest::Error),
#[error("webhook returned status {0}")]
ResponseStatus(reqwest::StatusCode),
}
impl WebhookDeliveryError {
pub(crate) fn is_retryable(&self) -> bool {
match self {
Self::Transport(_) => true,
Self::ResponseStatus(status) => status.is_server_error(),
Self::Configuration(_) => false,
}
}
}
// --- Webhook client ---
impl WebhookConfig {
@@ -384,7 +406,10 @@ impl WebhookConfig {
http_timeout: Duration,
) -> anyhow::Result<ValidateTokenResponse> {
let url = self.webhook_endpoint("validate-token")?;
let started_at = Instant::now();
let permit = self.validate_limiter.acquire().await;
let queue_elapsed = started_at.elapsed();
let http_started_at = Instant::now();
let ret = match tokio::time::timeout(http_timeout, async {
let resp = self
.client
@@ -406,25 +431,45 @@ impl WebhookConfig {
Err(_) => Err(anyhow::anyhow!("webhook validate-token timed out")),
};
permit.complete(ret.is_ok());
let http_elapsed = http_started_at.elapsed();
let elapsed = started_at.elapsed();
if queue_elapsed >= Duration::from_secs(1) || http_elapsed >= VALIDATE_TOKEN_SLOW_THRESHOLD
{
tracing::warn!(
machine_id = %req.machine_id,
queue_ms = queue_elapsed.as_millis(),
http_ms = http_elapsed.as_millis(),
elapsed_ms = elapsed.as_millis(),
success = ret.is_ok(),
"validate-token completed slowly"
);
}
ret
}
/// Notify the webhook receiver that a node has connected.
pub async fn notify_node_connected(&self, req: &NodeConnectedRequest) {
pub(crate) async fn notify_node_connected(
&self,
req: &NodeConnectedRequest,
) -> Result<(), WebhookDeliveryError> {
if !self.is_enabled() {
return;
return Ok(());
}
let Ok(url) = self.webhook_endpoint("webhook/node-connected") else {
tracing::warn!("skip node-connected webhook because webhook_url is not configured");
return;
};
let _ = self
let url = self
.webhook_endpoint("webhook/node-connected")
.map_err(WebhookDeliveryError::Configuration)?;
let response = self
.client
.post(&url)
.header("X-Internal-Auth", self.webhook_auth_secret())
.json(req)
.send()
.await;
.await
.map_err(WebhookDeliveryError::Transport)?;
if !response.status().is_success() {
return Err(WebhookDeliveryError::ResponseStatus(response.status()));
}
Ok(())
}
/// Notify the webhook receiver that a node has disconnected.
@@ -436,13 +481,35 @@ impl WebhookConfig {
tracing::warn!("skip node-disconnected webhook because webhook_url is not configured");
return;
};
let _ = self
let started_at = Instant::now();
let result = self
.client
.post(&url)
.header("X-Internal-Auth", self.webhook_auth_secret())
.json(req)
.send()
.await;
let elapsed = started_at.elapsed();
match result {
Err(error) => tracing::warn!(
machine_id = %req.machine_id,
elapsed_ms = elapsed.as_millis(),
%error,
"node-disconnected webhook delivery failed"
),
Ok(response) if !response.status().is_success() => tracing::warn!(
machine_id = %req.machine_id,
status = %response.status(),
elapsed_ms = elapsed.as_millis(),
"node-disconnected webhook returned failure status"
),
Ok(_) if elapsed >= VALIDATE_TOKEN_SLOW_THRESHOLD => tracing::warn!(
machine_id = %req.machine_id,
elapsed_ms = elapsed.as_millis(),
"node-disconnected webhook completed slowly"
),
Ok(_) => {}
}
}
fn webhook_auth_secret(&self) -> &str {
@@ -461,6 +528,21 @@ mod tests {
use axum::{Json, Router, routing::post};
use serde_json::json;
fn node_connected_request() -> NodeConnectedRequest {
NodeConnectedRequest {
machine_id: uuid::Uuid::new_v4().to_string(),
token: "token".to_string(),
user_id: Some(1),
hostname: String::new(),
version: String::new(),
os_type: None,
os_version: None,
os_distribution: None,
web_instance_id: None,
binding_version: Some(1),
}
}
#[test]
fn adaptive_validate_limiter_increases_under_queue_pressure() {
let now = Instant::now();
@@ -736,6 +818,8 @@ mod tests {
web_instance_api_base_url: None,
persisted_config_revision: None,
applied_config_revision: None,
applied_config_revision_known: false,
failed_instance_ids: Vec::new(),
};
validate_webhook
.validate_token_with_http_timeout(&req, Duration::from_millis(20))
@@ -773,4 +857,23 @@ mod tests {
assert!(resp.valid);
assert!(resp.config_revision.is_empty());
}
#[tokio::test]
async fn node_connected_transport_error_is_retryable() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
drop(listener);
let webhook = WebhookConfig::new(Some(format!("http://{addr}")), None, None, None, None);
let error = tokio::time::timeout(
Duration::from_secs(1),
webhook.notify_node_connected(&node_connected_request()),
)
.await
.unwrap()
.unwrap_err();
assert!(matches!(error, WebhookDeliveryError::Transport(_)));
assert!(error.is_retryable());
}
}