EasyTier/docs/core-architecture.md

EasyTier Core Architecture

Status and scope

This document describes the current architecture after the portable-core refactor. It is the source of truth for ownership, dependency direction, feature boundaries, and validation. It intentionally records the resulting design rather than the migration history.

The refactor has three principal crate roles:

• easytier-core owns portable EasyTier configuration, protocol state, routing, peer state, connectivity orchestration, packet processing, and instance lifecycle.
• easytier is the native composition root. It owns operating-system resources, native protocol engines, process integration, CLI and native presentation.
• easytier-proto owns generated protobuf and RPC types, descriptor data, and the feature slices needed by core and presentation users.

easytier-core is designed to compile without direct operating-system network access. It supports native hosts through Rust traits and has a target-only WASI adapter and ABI implementation under easytier-core/src/wasi.

This architecture does not require compatibility with old internal module paths. Wire compatibility, configuration compatibility, management semantics, and externally used application behaviour remain compatibility requirements.

Architectural vocabulary

The following terms have specific meanings in this document:

• Module: an interface and the implementation hidden behind it.
• Host: the process or runtime embedding core and owning platform resources.
• Host capability: an operation core may request but must not implement with direct OS calls.
• Adapter: a concrete implementation of a Host capability or protocol extension.
• Composition root: code that creates core configuration, Host Adapters, instances, and process-level services.
• Runtime configuration: the authoritative normalized state used after an instance starts.
• Packet plane: portable packet classification, routing, transformation, proxy/NAT state, and forwarding decisions.

New abstractions should pass a deletion test: deleting a useful deep Module should force non-trivial policy or lifecycle logic to reappear in multiple callers. A pass-through wrapper with no independent invariant is not an architectural boundary.

Crate dependency direction

The principal dependency direction is:

easytier-proto <- easytier-core <- easytier

Presentation crates and platform integrations consume these crates. Portable policy must not move outward merely because one current consumer is native. Conversely, core must not absorb an OS mechanism or a protocol engine whose dependencies cannot satisfy the core target contract.

easytier-proto

The protobuf crate is split by public Cargo features:

• core provides the common wire messages, peer RPC messages, generated RPC runtime, and descriptor bytes needed by core.
• api adds management API messages.
• protocol-specific features add only their generated message modules.
• json-rpc enables the well-known protobuf JSON types used by the management plane.
• full is the compatibility aggregate used by complete products.

The core crate depends on easytier-proto with default features disabled and enables only core, adding API or JSON-RPC types through its own management features.

The main core/native path has no prost-reflect dependency. OSPF route reflection uses the focused wire editor in peers/route/route_peer_wire.rs. It retains the original encoded RoutePeerInfo, replaces only the fields credential filtering is allowed to change, and leaves all other top-level and nested fields intact. This is required so unknown fields survive mixed-version, multi-hop propagation. Generated Rust types remain responsible for normal message construction and validation.

Descriptor sets are still generated and embedded by easytier-proto; removing runtime reflection did not remove descriptor data used by configuration and RPC tooling. The OHOS integration has its own schema service and dependency policy and is outside this replacement.

easytier-core

Core owns portable behaviour and exposes capability seams. Its normal dependencies use Tokio runtime, time, synchronization, and I/O traits without requiring the full Tokio feature set.

Core may depend on optional portable engines when their owning feature is enabled. It does not create real native TCP/UDP sockets, alter routes, open a TUN device, enter a network namespace, configure system DNS, manage a service, or invoke UPnP/NAT-PMP directly.

easytier

The native crate owns:

• process startup, shutdown, signals, service management, and allocators;
• filesystem configuration input and persistence;
• real TCP/UDP, DNS, TUN, raw-socket, route, interface, namespace, and socket option operations;
• UPnP and NAT-PMP operations;
• Unix and FakeTCP resources;
• WebSocket/WSS, QUIC, WireGuard, and KCP concrete engines;
• native Magic DNS serving and system DNS integration;
• CLI, web, GUI, FFI, and native management presentation.

Native code may translate values and assemble Adapters. It must not maintain a second peer graph, reproduce core routing or hole-punch policy, or invent an alternative instance lifecycle.

Internal core layers

The physical module layout follows this downward order:

foundation
  <- config / packet
  <- socket
  <- host
  <- tunnel
  <- listener / connectivity
  <- peers / rpc
  <- gateway
  <- instance
  <- management

process_runtime is a process- or module-scoped owner shared by instances. wasi is target integration and is compiled only for tests or the WASI target; it is not an additional portable domain layer.

Foundation

foundation/ contains task supervision, the time facade, rate limiting, statistics primitives, and the domain-neutral external operation broker. The broker owns asynchronous operation lifecycle and completion storage while the calling domain owns operation kinds, outcomes, resources, and errors. Foundation must not depend on a domain layer.

Configuration and packets

config/ owns:

• the complete TomlConfig model;
• parsing, serialization, and validation;
• OS-independent defaults;
• peer, encryption, gateway, and API input models;
• normalized runtime snapshots and the live runtime configuration store.

The Host supplies platform facts through CoreInstanceHostConfig. Core applies the policy that combines those facts with TOML input. This is especially important for a WASI build: the compile-time guest target cannot be used as a proxy for the Host operating system.

packet/ owns EasyTier packet structures, compression, STUN and hole-punch wire codecs. It does not own socket I/O or connection policy.

Socket and Host seams

socket/ contains transport-neutral primitives:

• SocketContext, including IP-family policy, optional socket mark, and an opaque network-namespace token;
• virtual TCP socket, listener, and factory traits;
• virtual UDP socket and factory traits;
• UDP session multiplexing, classification, and lifecycle;
• in-process Ring sockets.

UDP sessions expose a scoped hole-punch source observer keyed by transaction ID. Listeners carry a registration handle for their existing receive loop, which reports the first matching source; dropping the observer removes the registration. Cone clients optionally prepare that transaction in the listener-selection RPC; the response echoes its ID only after registration. The registration retains early UDP sources until the cone-sending RPC claims it. Unclaimed registrations expire after 30 seconds through the existing listener cleanup loop, covering local socket and STUN preparation without a separate receive or sending task. Clients omitting the optional transaction retain registration at the sending RPC; older servers omit the echoed ID and do not confirm early registration. Connectivity owns preparation expiry, the active RPC lifetime, candidate selection and bounded sending to the signaled and observed addresses. Cone clients start the existing UDP handshake without requiring a reverse hole-punch packet, and the session layer remains the sole receiver.

host/ is the single home of Host capability seams:

• DNS and DNS record resolution;
• connector environment observations;
• packet ingress and egress;
• Host socket operation bridges and handle-based TCP/UDP/listener adapters;
• host-owned, message-preserving tunnel endpoints.

Core owns scheduling, backpressure, cancellation, UDP session state, and protocol state even when each actual operation crosses a Host Adapter. A Host Adapter owns the real resource and performs the OS operation.

For host-owned tunnels, ownership crosses the guest ABI only after a bounded Host listener queue accepts the tunnel. The queue is registered through CoreHostAdapters and consumed by the normal CoreListenerRuntime, so the tunnel still reaches PeerAcceptedTunnelHandler and cannot bypass peer handshake, admission, events, or routing policy. Each Host receive produces one complete DummyTunnel payload; no stream framing is added. The Host owns transport-specific message validation and maps a clean close to tunnel EOF.

Hosts without outbound sockets select CoreConnectivityMode::InboundOnly for one instance. That startup plan retains listeners and the peer/router while omitting STUN, outbound connectors, and hole punching. Cargo features only compile Host Adapters; they do not change CoreInstance fields, lifecycle, or management semantics.

The native NativeHostRuntime is process-wide and does not retain an instance GlobalCtx, namespace guard, socket mark, or connectivity state. Differences between instances travel in each request's SocketContext. A narrow instance-host projection may expose listener and interface facts, but it does not become another socket factory.

Tunnel and listener

A socket is a raw communication endpoint. A Tunnel is an EasyTier connection created by adding framing, metadata, handshakes, and protocol lifecycle.

Core owns:

• raw TCP framing and upgrade;
• UDP tunnel/session framing and classification;
• Ring Tunnel identity and registry state;
• encryption and secure-datagram policy that is portable;
• client/server protocol selection interfaces;
• listener planning, optional/required listener policy, retry, accept scheduling, running-listener registry, and orderly shutdown.

Native protocol Adapters own WebSocket/WSS, QUIC, WireGuard, and KCP engines. Unix and FakeTCP are socket resources that feed a core protocol upgrader; they are not independent owners of EasyTier peer state.

Each protocol registration must provide a coherent client/server Adapter. Unavailable configured transports must be rejected during validation or protocol selection in the standard runtime, rather than silently falling back to another transport. A compact compatibility Host may instead retain the desired value for management readback and omit it from normalized runtime state; it must not advertise or partially activate the unavailable transport.

Connectivity

connectivity/ owns:

• manual connection and endpoint discovery policy;
• direct candidate selection;
• retry, backoff, blacklists, and listener reuse;
• STUN requests, responses, probing, NAT inference, and published endpoint state;
• TCP and UDP hole-punch state machines;
• UDP port-mapping policy and lease lifecycle;
• conversion of successful sockets into protocol-upgrade requests.

The Host owns DNS execution, socket syscalls, interface enumeration, bind device/mark/namespace operations, and concrete UPnP/NAT-PMP calls. STUN-only hole punching remains available when the Host does not supply a port-mapping Adapter.

Some connectivity files intentionally implement peer-facing adapter traits for PeerManagerCore. These are localized integration edges between adjacent domains, not permission for lower socket or Host layers to depend on peers.

Peers and RPC

peers/ is the authoritative owner of:

• admission and connection sessions;
• peer maps and connection lifecycle;
• ACL and whitelist decisions;
• OSPF route calculation and graph algorithms;
• peer and credential RPC registration;
• foreign-network admission, identity, relay, and lifecycle;
• peer-center state and public IPv6 policy;
• traffic metrics and peer snapshots.

Submodules progress from kernel types and utilities, through ACL/context, connection state, route state, manager services, and finally foreign-network and peer-center composition. Callers consume the public surface declared by the domain rather than reaching into a parallel native peer owner.

rpc/ owns the peer-flavoured RPC transport, packet fragmentation, client and server lifecycle, handler registry, and standalone listener/client lifecycle. Generated service descriptors and message types remain in easytier-proto.

Gateway

gateway/ owns portable packet-plane features:

• proxy CIDR state and monitoring policy;
• packet parsing, reassembly, NAT/proxy state, and TCP/UDP/ICMP decisions;
• the smoltcp-backed portable dataplane selected by its feature;
• SOCKS5 framing, authentication, association, routing, and session state;
• wrapped-transport planning and session state used by KCP and QUIC Adapters;
• DHCP allocation policy;
• Magic DNS route and response policy;
• VPN portal client/session policy;
• UDP broadcast classification and rewrite policy.

Each VPN portal client is normalized to one attached-peer IPv4 CIDR. The portable gateway owns that client address and prefix; the hosting network manager's DHCP or static address is not a source of portal client routing facts.

TUN, raw sockets, transparent-destination lookup, concrete protocol engines, native DNS servers, namespace operations, and route application stay in native Adapters.

Optional gateway capabilities are selected by cohesive Modules. Disabled implementations retain stable lifecycle calls and report unsupported configuration in the standard runtime. A compact compatibility Host may silently normalize those settings to no-ops while preserving the desired TOML model; disabled implementations do not duplicate portable policy.

The instance-scoped DataPlaneSession composes the foundation operation broker under the same session lock as its resource and quota state. The broker owns generic completion, cancellation, free, drain, and take transitions. The data plane retains TCP/UDP resource ownership, operation metadata, route deadlines, and error semantics.

The proposed restructuring of the smoltcp data plane, SOCKS5 and port-forward Adapters, portable KCP engine, event-driven FFI/WASI completion model, and Go Host integration is tracked in data-plane-runtime-plan.md. That document is a future implementation plan; this document remains the source of truth for the currently implemented architecture until the plan is completed.

Instance and management

CoreInstance::new(CoreInstanceConfig, CoreHostAdapters) is the sole direct construction path for a normalized instance. CoreInstance::from_toml uses the same normalization and construction path. Core constructs the peer graph, runtime store, STUN collector, connectivity managers, listener runtime, packet plane, gateway runtimes, and lifecycle owners.

A core instance:

• owns all mutable portable state for one network;
• is one-shot after stop;
• exposes one complete start and one stop lifecycle interface;
• starts Modules in a fixed serial composition order without cross-Module started flags or staged activation;
• installs initial ACL, proxy CIDR, and manual-peer inputs before startup;
• serializes lifecycle operations with one instance-level operation lock;
• owns cooperative cancellation and component shutdown order;
• exposes CorePacketPlane as the narrow packet/route projection used by Host dataplane Adapters;
• treats its normalized runtime store as authoritative after construction.

CoreHostAdapters contains the required Host, DNS, packet sink, and CoreProcessRuntime, plus optional protocol and platform capabilities. The bundle carries capabilities, not preconstructed portable managers.

The standard native factory owns one continuously driven Tokio executor per instance. multi_thread selects a multi-thread or current-thread runtime; multi_thread_count sets the worker count with the historical minimum of two. Construction, start/stop, and FFI data-plane tasks use that executor, independent of the process management runtime. Dropping its Host owner shuts down the executor without blocking a worker on its own termination. Compact and WASI Hosts retain their externally supplied execution model.

Each Module owns partial-start cleanup for its internal resources. CoreInstance has one outer cancellation and recovery path for the complete serial startup. Running therefore means the Host runtime and every enabled portable Module have started successfully; there is no separate post-Host activation state. Host packet tasks stop before PeerManager resources are cleared.

InstanceManager<F> is the canonical UUID-indexed instance collection for one Host composition. Its InstanceFactory constructs one complete record before the manager performs an atomic uniqueness check. The manager owns collection membership; it does not own startup order, persistence, daemon policy, cached errors, ABI handles, or RPC projections.

management/ consumes the canonical manager and instances. It owns:

• stable UUID/name selection;
• read-only instance and peer management RPC;
• full process mutation and configuration transactions when enabled;
• persistence and logger-control capability interfaces;
• management listener/client lifecycle and JSON-RPC presentation.

There is one process-level management entry. Instances and the manager do not depend on management response projections.

Process-scoped state

CoreProcessRuntime owns portable resources shared across instances in one process or instantiated module:

• the Ring Tunnel registry and namespace;
• a reference-counted protected TCP-port registry.

The composition root creates and shares one runtime. Management listener ports are protected before bind and held by leases after the concrete port is known. Native and target adapters supply bound resources but do not implement a second protected-port registry.

Process-global capability objects may contain stateless or shared platform mechanisms. They must not contain instance-specific peer, route, configuration, or connectivity state.

Native shared TUN ownership

The native composition owns SharedVirtualNicRegistry. Named desktop TUNs are shared within a network namespace; mobile instances attach to one supplied TUN descriptor. Members own address, route, and MTU claims, not the physical device. Bootstrap route cleanup runs only when that device is created. Mobile IPv6 claims are initialized before forwarding and follow public IPv6 updates. Mobile VPN configuration is projected from admitted members' native claims. Admission keeps unchanged attached members ahead of conflicting newcomers or reconfigured members. Rejected members are reported individually, while the remaining members stay attached. Each member's initial claims are applied as one batch before it can open or replace the shared descriptor. Mobile claims update dispatcher tables without configuring the OS interface; the platform VPN owns its addresses, routes, and MTU. If applying native claims fails, the shared NIC is invalidated and its dispatcher is stopped so every member can recreate the device. Restoring only the recorded claims would leave partially applied OS configuration in use.

Desktop and mobile shared TUNs use the same lazily initialized native Tokio runtime, independent of every member's executor. It creates the device's async I/O registration, drives dispatchers, and completes member claim cleanup even after the member executor stops or the registration is dropped outside Tokio. The runtime has two workers on desktop to allow independent TUN dispatchers to run concurrently, and retains one worker on mobile. It owns no registry or member claims; those remain with each shared NIC. Cleanup remains asynchronous, and the last member's cleanup stops the dispatcher and releases the desktop device. A process-lifetime runtime does not extend the device's lifetime. Unnamed desktop TUNs and instance-local work retain their existing executors.

Only the member running the Magic DNS resolver claims its shared fake-IP route. The claim is removed before resolver shutdown and acquired by the next serving member after takeover. Client-only runners do not claim that route.

The dispatcher preserves desktop kernel-forwarded subnet sources. It corrects wrong-member local sources and, on mobile, synthetic sources. Native local flows and translated flows reserve member-scoped transport tuples so address translation cannot merge sockets using the same port and remote endpoint. Collisions remap the source port or ICMP Echo identifier; replies restore the original tuple. Unrelated route updates and member removals retain mappings. All Echo sequence numbers for one identifier share a reservation; translation preserves the sequence field in each request and reply. Removing an involved address or member releases its mappings. A full member queue drops that packet without blocking other members or control messages.

Reservations are bounded at 4096 flows. Activity in either direction refreshes them; table pressure reclaims reservations idle for at least 180 seconds. If all slots remain active, new tracked flows are dropped rather than evicting a live mapping. Desktop external-source forwarding bypasses this table. As with other idle-expiring NAT state, very late replies after tuple reuse cannot be distinguished from replies to the new flow.

Translation supports unfragmented IPv4 and ordinary IPv6 TCP, UDP, and ICMP Echo traffic. IPv4 fragment and IPv6 extension-header translation remain unsupported. ICMP errors quoting a translated flow are not reverse-translated; in particular, a smaller downstream MTU can prevent PMTU discovery for such flows. This limitation does not affect unmodified flows and is intentionally retained rather than extending the dispatcher into a full ICMP/fragment NAT.

Runtime configuration authority

TomlConfig is the authoritative desired configuration used for management readback and patch transactions. Compact Hosts keep unsupported accepted values there so controllers observe the configuration they submitted.

The separately typed, normalized core runtime store is authoritative for live behavior:

• peer feature flags and routing policy;
• listeners and initial peers;
• ACL and whitelist inputs;
• manual and VPN portal CIDRs;
• gateway and connectivity settings;
• runtime configuration patches.

Host persistence is an effect following a successful core transaction. A Host Adapter must not call back into an instance to obtain a hidden configuration snapshot while core is applying an operation.

Non-serializable resources such as TUN descriptors, packet sinks, execution domains, and native protocol engines are construction context, not TOML fields.

Logging

The main native runtime uses a small logger implemented in easytier/src/common/log:

• log records and tracing events share console and file sinks;
• timestamps, compact formatting, optional terminal colours, NO_COLOR, and basic RUST_LOG target/level filters are implemented directly;
• file rotation uses the existing EasyTier rolling appender;
• management RPC can reload the file level;
• an atomic maximum-level gate rejects disabled events before target matching or file-filter locking;
• concurrent file-level reload serializes the filter and atomic-level update.

File logging and no-file logging are separate selected backends. The default tracing backend records events and deliberately ignores span trees. The optional tracing feature selects the tokio-console subscriber integration; only that diagnostic profile pulls the main crate's tracing-subscriber and console-subscriber dependencies.

Contrib applications and platform integrations may have independent logging requirements and are not implicitly wired to the native process logger.

Feature model

Features represent coherent capabilities, not arbitrary source fragments. Important core feature relationships are:

• management-rpc enables generated management API types and read-only management services.
• management adds configuration writes, full management composition, rich errors, and JSON-RPC.
• proxy-packet enables portable packet parsing/proxy machinery and the required smoltcp packet features.
• proxy-smoltcp-stack adds the async TCP/UDP smoltcp stack.
• dns-resolver is the shared Hickory resolver leaf used by endpoint discovery and Magic DNS without coupling either capability to the other.
• endpoint-discovery adds HTTPS endpoint discovery dependencies.
• magic-dns enables its DNS server, management wire messages, and portable packet-query integration.
• tcp-hole-punch enables the TCP hole-punch runtime.
• dhcp-ipv4, public-ipv6-provider, vpn-portal, wrapped-transport, and proxy-cidr-monitor are independent gateway or platform-policy leaves.
• extended-services is the compatibility aggregate for those leaves.
• encryption and compression engines remain independently selectable.

The native crate maps product features to the core and protocol features it actually consumes. A protocol feature must not accidentally enable unrelated gateway or management capabilities.

Production feature and platform selection belongs at Module or Adapter boundaries rather than inside shared implementations. The logger demonstrates the intended pattern: file and tracing variants are complete backend modules with one stable interface, so shared event processing contains no feature branches.

Module boundaries

The dependency directions in this document define the intended module boundaries. Changes that require a new upward edge must first define a stable lower-layer interface or explicitly revise this architecture.

Modules are pub(crate) by default. Each domain's mod.rs declares its outward surface. Public visibility is used for real cross-crate Host, configuration, management, packet-plane, or test-support interfaces.

Architectural invariants

1. Portable EasyTier policy has one owner in easytier-core.
2. Core does not perform real OS socket, DNS, TUN, route, filesystem configuration, process, or service-manager operations.
3. Host-OS policy is runtime input; a WASI compile target is not Host policy.
4. Every real socket and DNS operation crosses a Host capability seam.
5. Core owns socket scheduling, backpressure, protocol state, and cancellation.
6. Dial, accept, and hole-punch paths produce sockets before protocol upgrade.
7. Peer admission consumes upgraded transports and does not create OS resources.
8. Each instance owns its mutable peer, route, connectivity, gateway, and runtime configuration state.
9. One Host composition has one canonical UUID-to-instance manager.
10. Process-level runtimes do not capture instance state.
11. CoreInstance::new is the sole normalized direct construction entry.
12. The manager owns membership, not lifecycle or presentation.
13. Management consumes the manager; the manager does not return management projections.
14. Unknown protobuf fields in reflected route information survive forwarding and credential filtering.
15. Feature selection is localized at cohesive Module/Adapter boundaries.
16. The standard runtime rejects unsupported configured capabilities. Compact compatibility Hosts may preserve them as runtime no-ops, but never change wire protocol, advertise them, or silently fall back to an unsafe mode.

Validation

Changes to these boundaries should run, at minimum:

cargo fmt --all -- --check
cargo check -p easytier-core -p easytier-proto -p easytier --features full
cargo test -p easytier-core --lib

Feature work should add focused checks for the changed no-default, isolated, default, full, and cross-target profiles. Socket, TUN, namespace, protocol engine, and multi-node changes require the relevant Docker integration tests. WASI ABI or Adapter changes require a wasm32-wasip1 build and target-side tests. These compiler-resolved profiles are the authority for feature and target boundaries.

CI path filters include easytier-core, easytier-proto, native, web, GUI Tauri, and contrib. The archived Rust test suite contains both easytier and easytier-core.

Known limitations and debt

• Some production feature and platform gates still select fields or statements inside shared implementations. New code should prefer complete Module or Adapter variants, and existing cases should move only when their owning Module is changed.
• Connectivity retains localized Adapter implementations that name PeerManagerCore; further decoupling requires an interface extraction, not a visibility-only move.
• Native Linux namespace guards exist in paths that can cross async suspension. Because setns is thread-local, those operations should eventually be kept on one non-migrating execution context.
• QUIC session retirement after failed or exhausted accepted sessions remains separate native-engine correctness work; it must preserve multiple connections sharing one QUIC endpoint/session.

These limitations are not reasons to add fallback owners or parallel state. Fixes should preserve the ownership rules above and address the responsible Module directly.