mirror of
https://github.com/EasyTier/EasyTier.git
synced 2026-10-13 07:46:15 -08:00
Add shared virtual NIC core with per-member routing
Share a named TUN across instances while keeping unnamed devices dedicated. Track member addresses, routes, and MTU, and reject overlapping destinations except the common Magic DNS host route. Dispatch packets between the physical TUN and member tunnels. Keep IPv4 and ordinary IPv6 source translation for mobile wrong-source traffic. Update platform configuration and runtime host integration, with unit and network namespace coverage for routing and lifecycle. Use smoltcp for packet parsing and checksum updates. Keep core CI matrix jobs independent. Reserve native inbound flows before source translations allocate their wire tuples. Keep active reservations under table pressure and leave forwarded subnet traffic outside the local flow table. Complete interface claim updates on the shared runtime and retain member registrations until the transaction commits. Keep address, route, and MTU cleanup ordered after cancelled callers, including device initialization. Project mobile VPN addresses, routes, and DNS from admitted native members. Prefer unchanged attached claims and report conflicting members individually. Apply each member's complete initial claims before opening the shared fd, so conflicts leave healthy members usable. Cover inbound IPv4/IPv6 collisions, cancellation during native changes and dispatcher acknowledgments, admission priority, and atomic mobile claim rejection with regression tests. Use scoped TestNetwork fixtures for shared TUN bridge setup and cleanup. Resolve public IPv6 namespace aliases through the lab's actual names so the new tests remain isolated under parallel upstream test runs.
This commit is contained in:
1 parent
1c6992d6cb
commit
ad795429c6
26 files changed
+8502
-339
No files matched your search
@@ -144,7 +144,7 @@ jobs:
|
||||
easytier-web/frontend/dist/*
|
||||
build:
|
||||
strategy:
|
||||
fail-fast: true
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- TARGET: x86_64-unknown-linux-musl
|
||||
|
||||
@@ -413,6 +413,64 @@ Process-global capability objects may contain stateless or shared platform
|
||||
mechanisms. They must not contain instance-specific peer, route,
|
||||
configuration, or connectivity state.
|
||||
|
||||
### Native shared TUN ownership
|
||||
|
||||
The native composition owns `SharedVirtualNicRegistry`. Named desktop TUNs
|
||||
are shared within a network namespace; mobile instances attach to one supplied
|
||||
TUN descriptor. Members own address, route, and MTU claims, not the physical
|
||||
device. Bootstrap route cleanup runs only when that device is created. Mobile
|
||||
IPv6 claims are initialized before forwarding and follow public IPv6 updates.
|
||||
Mobile VPN configuration is projected from admitted members' native claims.
|
||||
Admission keeps unchanged attached members ahead of conflicting newcomers or
|
||||
reconfigured members. Rejected members are reported individually, while the
|
||||
remaining members stay attached. Each member's initial claims are applied as
|
||||
one batch before it can open or replace the shared descriptor.
|
||||
Mobile claims update dispatcher tables without configuring the OS interface;
|
||||
the platform VPN owns its addresses, routes, and MTU.
|
||||
If applying native claims fails, the shared NIC is invalidated and its
|
||||
dispatcher is stopped so every member can recreate the device. Restoring only
|
||||
the recorded claims would leave partially applied OS configuration in use.
|
||||
|
||||
Desktop and mobile shared TUNs use the same lazily initialized native Tokio
|
||||
runtime, independent of every member's executor. It creates the device's async
|
||||
I/O registration, drives dispatchers, and completes member claim cleanup even
|
||||
after the member executor stops or the registration is dropped outside Tokio.
|
||||
The runtime has two workers on desktop to allow independent TUN dispatchers
|
||||
to run concurrently, and retains one worker on mobile. It owns no registry or
|
||||
member claims; those remain with each shared NIC. Cleanup remains asynchronous,
|
||||
and the last member's cleanup stops the dispatcher and releases the desktop
|
||||
device. A process-lifetime runtime does not extend the device's lifetime.
|
||||
Unnamed desktop TUNs and instance-local work retain their existing executors.
|
||||
|
||||
Only the member running the Magic DNS resolver claims its shared fake-IP
|
||||
route. The claim is removed before resolver shutdown and acquired by the next
|
||||
serving member after takeover. Client-only runners do not claim that route.
|
||||
|
||||
The dispatcher preserves desktop kernel-forwarded subnet sources. It corrects
|
||||
wrong-member local sources and, on mobile, synthetic sources. Native local
|
||||
flows and translated flows reserve member-scoped transport tuples so address
|
||||
translation cannot merge sockets using the same port and remote endpoint.
|
||||
Collisions remap the source port or ICMP Echo identifier; replies restore the
|
||||
original tuple. Unrelated route updates and member removals retain mappings.
|
||||
All Echo sequence numbers for one identifier share a reservation; translation
|
||||
preserves the sequence field in each request and reply.
|
||||
Removing an involved address or member releases its mappings. A full member
|
||||
queue drops that packet without blocking other members or control messages.
|
||||
|
||||
Reservations are bounded at 4096 flows. Activity in either direction refreshes
|
||||
them; table pressure reclaims reservations idle for at least 180 seconds. If
|
||||
all slots remain active, new tracked flows are dropped rather than evicting a
|
||||
live mapping. Desktop external-source forwarding bypasses this table. As with
|
||||
other idle-expiring NAT state, very late replies after tuple reuse cannot be
|
||||
distinguished from replies to the new flow.
|
||||
|
||||
Translation supports unfragmented IPv4 and ordinary IPv6 TCP, UDP, and ICMP
|
||||
Echo traffic. IPv4 fragment and IPv6 extension-header translation remain
|
||||
unsupported. ICMP errors quoting a translated flow are not reverse-translated;
|
||||
in particular, a smaller downstream MTU can prevent PMTU discovery for such
|
||||
flows. This limitation does not affect unmodified flows and is intentionally
|
||||
retained rather than extending the dispatcher into a full ICMP/fragment NAT.
|
||||
|
||||
## Runtime configuration authority
|
||||
|
||||
`TomlConfig` is the authoritative desired configuration used for management
|
||||
|
||||
+1
-1
@@ -365,7 +365,7 @@ mimalloc = ["dep:mimalloc"]
|
||||
aes-gcm = ["easytier-core/aes-gcm"]
|
||||
openssl-crypto = ["easytier-core/openssl-crypto"]
|
||||
ring-crypto = ["easytier-core/ring-crypto"]
|
||||
tun = ["dep:tun", "linux-netlink"]
|
||||
tun = ["dep:tun", "dep:smoltcp", "linux-netlink", "tokio/rt-multi-thread"]
|
||||
linux-netlink = ["dep:netlink-sys"]
|
||||
proxy-cidr-monitor = ["easytier-core/proxy-cidr-monitor"]
|
||||
websocket = [
|
||||
|
||||
@@ -1,10 +1,38 @@
|
||||
use std::net::Ipv4Addr;
|
||||
use std::{collections::BTreeSet, net::Ipv4Addr};
|
||||
|
||||
use super::{Error, IfConfiguerTrait, cidr_to_subnet_mask, run_shell_cmd};
|
||||
use async_trait::async_trait;
|
||||
use cidr::{Ipv4Inet, Ipv6Inet};
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct MacIfConfiger {
|
||||
configured_ipv4: Mutex<BTreeSet<Ipv4Inet>>,
|
||||
}
|
||||
|
||||
impl MacIfConfiger {
|
||||
fn build_add_ipv4_cmd(name: &str, addr: Ipv4Inet, has_configured_ipv4: bool) -> String {
|
||||
let address = addr.address();
|
||||
if has_configured_ipv4 {
|
||||
format!(
|
||||
"ifconfig {} alias {:?} {:?} netmask {}",
|
||||
name,
|
||||
address,
|
||||
address,
|
||||
cidr_to_subnet_mask(addr.network_length())
|
||||
)
|
||||
} else {
|
||||
format!(
|
||||
"ifconfig {} {:?}/{:?} {:?} up",
|
||||
name,
|
||||
address,
|
||||
addr.network_length(),
|
||||
address,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub struct MacIfConfiger {}
|
||||
#[async_trait]
|
||||
impl IfConfiguerTrait for MacIfConfiger {
|
||||
async fn add_ipv4_route(
|
||||
@@ -14,12 +42,28 @@ impl IfConfiguerTrait for MacIfConfiger {
|
||||
cidr_prefix: u8,
|
||||
cost: Option<i32>,
|
||||
) -> Result<(), Error> {
|
||||
self.add_ipv4_route_with_source_hint(name, address, cidr_prefix, cost, None)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn add_ipv4_route_with_source_hint(
|
||||
&self,
|
||||
name: &str,
|
||||
address: Ipv4Addr,
|
||||
cidr_prefix: u8,
|
||||
cost: Option<i32>,
|
||||
source_hint: Option<Ipv4Addr>,
|
||||
) -> Result<(), Error> {
|
||||
let source_hint = source_hint
|
||||
.map(|source| format!(" -ifa {}", source))
|
||||
.unwrap_or_default();
|
||||
run_shell_cmd(
|
||||
format!(
|
||||
"route -n add {} -netmask {} -interface {} -hopcount {}",
|
||||
"route -n add {} -netmask {} -interface {}{} -hopcount {}",
|
||||
address,
|
||||
cidr_to_subnet_mask(cidr_prefix),
|
||||
name,
|
||||
source_hint,
|
||||
cost.unwrap_or(7)
|
||||
)
|
||||
.as_str(),
|
||||
@@ -51,14 +95,15 @@ impl IfConfiguerTrait for MacIfConfiger {
|
||||
address: Ipv4Addr,
|
||||
cidr_prefix: u8,
|
||||
) -> Result<(), Error> {
|
||||
run_shell_cmd(
|
||||
format!(
|
||||
"ifconfig {} {:?}/{:?} {:?} up",
|
||||
name, address, cidr_prefix, address,
|
||||
)
|
||||
.as_str(),
|
||||
)
|
||||
.await
|
||||
let addr = Ipv4Inet::new(address, cidr_prefix).map_err(|err| {
|
||||
anyhow::anyhow!("invalid IPv4 address {address}/{cidr_prefix}: {err:?}")
|
||||
})?;
|
||||
let mut configured_ipv4 = self.configured_ipv4.lock().await;
|
||||
let cmd = Self::build_add_ipv4_cmd(name, addr, !configured_ipv4.is_empty());
|
||||
|
||||
run_shell_cmd(cmd.as_str()).await?;
|
||||
configured_ipv4.insert(addr);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn set_link_status(&self, name: &str, up: bool) -> Result<(), Error> {
|
||||
@@ -67,11 +112,16 @@ impl IfConfiguerTrait for MacIfConfiger {
|
||||
}
|
||||
|
||||
async fn remove_ip(&self, name: &str, ip: Option<Ipv4Inet>) -> Result<(), Error> {
|
||||
let mut configured_ipv4 = self.configured_ipv4.lock().await;
|
||||
if let Some(ip) = ip {
|
||||
run_shell_cmd(format!("ifconfig {} inet {} delete", name, ip.address()).as_str()).await
|
||||
run_shell_cmd(format!("ifconfig {} inet {} delete", name, ip.address()).as_str())
|
||||
.await?;
|
||||
configured_ipv4.remove(&ip);
|
||||
} else {
|
||||
run_shell_cmd(format!("ifconfig {} inet delete", name).as_str()).await
|
||||
run_shell_cmd(format!("ifconfig {} inet delete", name).as_str()).await?;
|
||||
configured_ipv4.clear();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn set_mtu(&self, name: &str, mtu: u32) -> Result<(), Error> {
|
||||
|
||||
@@ -40,6 +40,16 @@ pub trait IfConfiguerTrait: Send + Sync {
|
||||
) -> Result<(), Error> {
|
||||
Ok(())
|
||||
}
|
||||
async fn add_ipv4_route_with_source_hint(
|
||||
&self,
|
||||
name: &str,
|
||||
address: Ipv4Addr,
|
||||
cidr_prefix: u8,
|
||||
cost: Option<i32>,
|
||||
_source_hint: Option<Ipv4Addr>,
|
||||
) -> Result<(), Error> {
|
||||
self.add_ipv4_route(name, address, cidr_prefix, cost).await
|
||||
}
|
||||
async fn remove_ipv4_route(
|
||||
&self,
|
||||
_name: &str,
|
||||
@@ -48,6 +58,16 @@ pub trait IfConfiguerTrait: Send + Sync {
|
||||
) -> Result<(), Error> {
|
||||
Ok(())
|
||||
}
|
||||
async fn remove_ipv4_route_with_cost_and_source_hint(
|
||||
&self,
|
||||
name: &str,
|
||||
address: Ipv4Addr,
|
||||
cidr_prefix: u8,
|
||||
_cost: Option<i32>,
|
||||
_source_hint: Option<Ipv4Addr>,
|
||||
) -> Result<(), Error> {
|
||||
self.remove_ipv4_route(name, address, cidr_prefix).await
|
||||
}
|
||||
async fn add_ipv4_ip(
|
||||
&self,
|
||||
_name: &str,
|
||||
@@ -157,6 +177,7 @@ async fn run_shell_cmd(cmd: &str) -> Result<(), Error> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct DummyIfConfiger {}
|
||||
#[async_trait]
|
||||
impl IfConfiguerTrait for DummyIfConfiger {}
|
||||
|
||||
@@ -133,6 +133,7 @@ fn dump_netlink_messages<T: NetlinkDecode>(
|
||||
receive_netlink_dump(builder)
|
||||
}
|
||||
|
||||
#[derive(Default)]
|
||||
pub struct NetlinkIfConfiger {}
|
||||
|
||||
impl NetlinkIfConfiger {
|
||||
@@ -335,6 +336,27 @@ impl NetlinkIfConfiger {
|
||||
})
|
||||
.collect())
|
||||
}
|
||||
|
||||
fn ipv4_route_message(
|
||||
ifindex: u32,
|
||||
address: Ipv4Addr,
|
||||
cidr_prefix: u8,
|
||||
cost: Option<i32>,
|
||||
source_hint: Option<Ipv4Addr>,
|
||||
) -> RouteMessage {
|
||||
let mut builder = RouteMessageBuilder::new(libc::AF_INET as u8)
|
||||
.destination(IpAddr::V4(address), cidr_prefix)
|
||||
.oif(ifindex)
|
||||
.priority(cost.unwrap_or(65535) as u32)
|
||||
.table(libc::RT_TABLE_MAIN.into())
|
||||
.static_protocol()
|
||||
.universe_scope()
|
||||
.route_type(RouteType::Unicast);
|
||||
if let Some(source_hint) = source_hint {
|
||||
builder = builder.preferred_source(IpAddr::V4(source_hint));
|
||||
}
|
||||
builder.build()
|
||||
}
|
||||
}
|
||||
|
||||
#[async_trait]
|
||||
@@ -346,15 +368,25 @@ impl IfConfiguerTrait for NetlinkIfConfiger {
|
||||
cidr_prefix: u8,
|
||||
cost: Option<i32>,
|
||||
) -> Result<(), Error> {
|
||||
let message = RouteMessageBuilder::new(libc::AF_INET as u8)
|
||||
.destination(IpAddr::V4(address), cidr_prefix)
|
||||
.oif(Self::get_interface_index(name)?)
|
||||
.priority(cost.unwrap_or(65535) as u32)
|
||||
.table(libc::RT_TABLE_MAIN.into())
|
||||
.static_protocol()
|
||||
.universe_scope()
|
||||
.route_type(RouteType::Unicast)
|
||||
.build();
|
||||
self.add_ipv4_route_with_source_hint(name, address, cidr_prefix, cost, None)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn add_ipv4_route_with_source_hint(
|
||||
&self,
|
||||
name: &str,
|
||||
address: Ipv4Addr,
|
||||
cidr_prefix: u8,
|
||||
cost: Option<i32>,
|
||||
source_hint: Option<Ipv4Addr>,
|
||||
) -> Result<(), Error> {
|
||||
let message = NetlinkIfConfiger::ipv4_route_message(
|
||||
NetlinkIfConfiger::get_interface_index(name)?,
|
||||
address,
|
||||
cidr_prefix,
|
||||
cost,
|
||||
source_hint,
|
||||
);
|
||||
let request = message_request(
|
||||
RTM_NEWROUTE,
|
||||
NLM_F_ACK | NLM_F_CREATE | NLM_F_EXCL | NLM_F_REQUEST,
|
||||
@@ -390,6 +422,28 @@ impl IfConfiguerTrait for NetlinkIfConfiger {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn remove_ipv4_route_with_cost_and_source_hint(
|
||||
&self,
|
||||
name: &str,
|
||||
address: Ipv4Addr,
|
||||
cidr_prefix: u8,
|
||||
cost: Option<i32>,
|
||||
source_hint: Option<Ipv4Addr>,
|
||||
) -> Result<(), Error> {
|
||||
let message = Self::ipv4_route_message(
|
||||
Self::get_interface_index(name)?,
|
||||
address,
|
||||
cidr_prefix,
|
||||
cost,
|
||||
source_hint,
|
||||
);
|
||||
let request = message_request(RTM_DELROUTE, NLM_F_ACK | NLM_F_REQUEST, &message)?;
|
||||
match send_netlink_req_and_wait_ack(request) {
|
||||
Err(Error::IOError(err)) if err.raw_os_error() == Some(libc::ESRCH) => Ok(()),
|
||||
result => result,
|
||||
}
|
||||
}
|
||||
|
||||
async fn add_ipv4_ip(
|
||||
&self,
|
||||
name: &str,
|
||||
@@ -696,6 +750,49 @@ mod tests {
|
||||
assert!(!routes.contains(&IpAddr::V4("10.5.5.0".parse().unwrap())));
|
||||
}
|
||||
|
||||
#[serial_test::serial]
|
||||
#[tokio::test]
|
||||
async fn remove_ipv4_route_with_source_hint_keeps_other_metric() {
|
||||
let iface = test_iface_name("rm");
|
||||
let _link = ScopedDummyLink::new(&iface);
|
||||
let ifcfg = NetlinkIfConfiger {};
|
||||
let address = "10.231.1.1".parse().unwrap();
|
||||
let destination = "10.99.0.0".parse().unwrap();
|
||||
|
||||
ifcfg.add_ipv4_ip(&iface, address, 24).await.unwrap();
|
||||
for cost in [123, 124] {
|
||||
ifcfg
|
||||
.add_ipv4_route_with_source_hint(&iface, destination, 24, Some(cost), Some(address))
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
ifcfg
|
||||
.remove_ipv4_route_with_cost_and_source_hint(
|
||||
&iface,
|
||||
destination,
|
||||
24,
|
||||
Some(123),
|
||||
Some(address),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let routes = run_cmd(&format!("ip -4 route show 10.99.0.0/24 dev {iface}"));
|
||||
assert!(!routes.contains("metric 123"));
|
||||
assert!(routes.contains("metric 124"));
|
||||
|
||||
ifcfg
|
||||
.remove_ipv4_route_with_cost_and_source_hint(
|
||||
&iface,
|
||||
destination,
|
||||
24,
|
||||
Some(123),
|
||||
Some(address),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[serial_test::serial]
|
||||
#[tokio::test]
|
||||
async fn ipv6_addr_readback_test() {
|
||||
|
||||
@@ -37,6 +37,7 @@ const RTA_DST: u16 = 1;
|
||||
const RTA_SRC: u16 = 2;
|
||||
const RTA_OIF: u16 = 4;
|
||||
const RTA_PRIORITY: u16 = 6;
|
||||
const RTA_PREFSRC: u16 = 7;
|
||||
const RTA_TABLE: u16 = 15;
|
||||
|
||||
const NDA_DST: u16 = 1;
|
||||
@@ -488,6 +489,13 @@ impl RouteMessageBuilder {
|
||||
self
|
||||
}
|
||||
|
||||
pub(crate) fn preferred_source(mut self, address: IpAddr) -> Self {
|
||||
self.message
|
||||
.attributes
|
||||
.push(Attribute::new(RTA_PREFSRC, ip_bytes(address)));
|
||||
self
|
||||
}
|
||||
|
||||
pub(crate) fn table(mut self, table: u32) -> Self {
|
||||
if let Ok(table) = u8::try_from(table) {
|
||||
self.message.table = table;
|
||||
@@ -632,6 +640,22 @@ mod tests {
|
||||
assert_eq!(encode(&decoded), bytes);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn route_builder_encodes_preferred_source() {
|
||||
let source = "10.231.1.1".parse().unwrap();
|
||||
let message = RouteMessageBuilder::new(libc::AF_INET as u8)
|
||||
.destination("10.99.0.0".parse().unwrap(), 24)
|
||||
.preferred_source(source)
|
||||
.oif(7)
|
||||
.table(libc::RT_TABLE_MAIN.into())
|
||||
.build();
|
||||
let bytes = encode(&message);
|
||||
let attributes = parse_attributes(&bytes[12..]).unwrap();
|
||||
assert!(attributes.iter().any(|attribute| {
|
||||
attribute.kind == RTA_PREFSRC && attribute.value == ip_bytes(source)
|
||||
}));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn route_parser_reads_ipv6_source_prefix() {
|
||||
let mut bytes = vec![
|
||||
|
||||
@@ -22,6 +22,7 @@ use winreg::{
|
||||
};
|
||||
|
||||
use super::{Error, IfConfiguerTrait};
|
||||
#[derive(Default)]
|
||||
pub struct WindowsIfConfiger {}
|
||||
|
||||
fn format_win_error(error: u32) -> String {
|
||||
|
||||
@@ -31,6 +31,8 @@ use crate::{
|
||||
instance::runtime_host::NativeInstanceRuntimeHost,
|
||||
};
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
use super::shared_virtual_nic::ArcSharedVirtualNicRegistry;
|
||||
use super::{
|
||||
host::{NativeInstanceHost, native_instance_host},
|
||||
runtime_executor::NativeInstanceExecutor,
|
||||
@@ -48,6 +50,7 @@ pub(crate) type NativeCoreInstance = CoreInstance<NativeInstanceHost>;
|
||||
pub(super) fn compose_native_core_instance(
|
||||
toml_config: TomlConfig,
|
||||
process_runtime: Arc<CoreProcessRuntime>,
|
||||
#[cfg(feature = "tun")] shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
compact_runtime: bool,
|
||||
executor: Option<NativeInstanceExecutor>,
|
||||
) -> anyhow::Result<Arc<NativeCoreInstance>> {
|
||||
@@ -62,7 +65,12 @@ pub(super) fn compose_native_core_instance(
|
||||
&normalized,
|
||||
&host_config,
|
||||
));
|
||||
let runtime_host = NativeInstanceRuntimeHost::new_with_executor(global_ctx.clone(), executor);
|
||||
let runtime_host = NativeInstanceRuntimeHost::new_with_executor(
|
||||
global_ctx.clone(),
|
||||
#[cfg(feature = "tun")]
|
||||
shared_virtual_nic_registry,
|
||||
executor,
|
||||
);
|
||||
let mut adapters = runtime_core_host_adapters_with_packet_egress_and_config(
|
||||
global_ctx.clone(),
|
||||
process_runtime,
|
||||
|
||||
@@ -5,7 +5,15 @@ use std::{net::Ipv4Addr, sync::Arc, time::Duration};
|
||||
|
||||
use easytier_core::instance::CorePacketPlane;
|
||||
|
||||
use crate::common::global_ctx::ArcGlobalCtx;
|
||||
use crate::{
|
||||
common::{
|
||||
error::Error as EtError,
|
||||
global_ctx::ArcGlobalCtx,
|
||||
ifcfg::{IfConfiger, IfConfiguerTrait},
|
||||
netns::NetNS,
|
||||
},
|
||||
instance::virtual_nic::NicBackend,
|
||||
};
|
||||
|
||||
use super::{client_instance::MagicDnsClientInstance, server_instance::MagicDnsServerInstance};
|
||||
|
||||
@@ -17,6 +25,52 @@ pub struct DnsRunner {
|
||||
tun_dev: Option<String>,
|
||||
tun_inet: Ipv4Inet,
|
||||
fake_ip: Ipv4Addr,
|
||||
shared_route_backend: Option<NicBackend>,
|
||||
active_fake_ip_route: Option<MagicDnsFakeIpRouteClaim>,
|
||||
}
|
||||
|
||||
struct MagicDnsFakeIpRouteClaim {
|
||||
tun_dev: Option<String>,
|
||||
net_ns: NetNS,
|
||||
fake_ip: Ipv4Addr,
|
||||
route_backend: NicBackend,
|
||||
}
|
||||
|
||||
impl MagicDnsFakeIpRouteClaim {
|
||||
async fn add(&self) -> anyhow::Result<()> {
|
||||
let cost = if cfg!(target_os = "windows") {
|
||||
Some(4)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
match self
|
||||
.route_backend
|
||||
.add_route_with_cost(self.fake_ip, 32, cost)
|
||||
.await
|
||||
{
|
||||
Err(EtError::IOError(err))
|
||||
if err.kind() == std::io::ErrorKind::AlreadyExists && self.tun_dev.is_some() =>
|
||||
{
|
||||
let ifcfg = IfConfiger::default();
|
||||
let _guard = self.net_ns.guard();
|
||||
ifcfg
|
||||
.remove_ipv4_route(self.tun_dev.as_deref().unwrap(), self.fake_ip, 32)
|
||||
.await?;
|
||||
self.route_backend
|
||||
.add_route_with_cost(self.fake_ip, 32, cost)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
result => result.map_err(Into::into),
|
||||
}
|
||||
}
|
||||
|
||||
async fn remove(&self) {
|
||||
if let Err(err) = self.route_backend.remove_route(self.fake_ip, 32).await {
|
||||
tracing::warn!(?err, fake_ip = ?self.fake_ip, "remove magic dns route failed");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl DnsRunner {
|
||||
@@ -35,29 +89,76 @@ impl DnsRunner {
|
||||
tun_dev,
|
||||
tun_inet,
|
||||
fake_ip,
|
||||
shared_route_backend: None,
|
||||
active_fake_ip_route: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn with_shared_route_backend(mut self, route_backend: Option<NicBackend>) -> Self {
|
||||
self.shared_route_backend = route_backend;
|
||||
self
|
||||
}
|
||||
|
||||
async fn clean_env(&mut self) {
|
||||
if let Some(claim) = self.active_fake_ip_route.take() {
|
||||
claim.remove().await;
|
||||
}
|
||||
if let Some(server) = self.server.take() {
|
||||
server.clean_env().await;
|
||||
}
|
||||
self.client.take();
|
||||
}
|
||||
|
||||
async fn run_once(&mut self) -> anyhow::Result<()> {
|
||||
fn should_manage_fake_ip_route_externally(&self) -> bool {
|
||||
self.shared_route_backend.is_some() && !self.tun_inet.contains(&self.fake_ip)
|
||||
}
|
||||
|
||||
fn fake_ip_route_claim(&self) -> Option<MagicDnsFakeIpRouteClaim> {
|
||||
if !self.should_manage_fake_ip_route_externally() {
|
||||
return None;
|
||||
}
|
||||
|
||||
Some(MagicDnsFakeIpRouteClaim {
|
||||
tun_dev: self.tun_dev.clone(),
|
||||
net_ns: self.global_ctx.net_ns.clone(),
|
||||
fake_ip: self.fake_ip,
|
||||
route_backend: self.shared_route_backend.clone()?,
|
||||
})
|
||||
}
|
||||
|
||||
async fn run_once(&mut self, cancel_token: &CancellationToken) -> anyhow::Result<()> {
|
||||
// try server first
|
||||
match MagicDnsServerInstance::new(
|
||||
self.packet_plane.clone(),
|
||||
self.global_ctx.clone(),
|
||||
self.tun_dev.clone(),
|
||||
self.tun_inet,
|
||||
self.fake_ip,
|
||||
)
|
||||
.await
|
||||
{
|
||||
let server_result = if self.should_manage_fake_ip_route_externally() {
|
||||
MagicDnsServerInstance::new_with_external_fake_ip_route(
|
||||
self.packet_plane.clone(),
|
||||
self.global_ctx.clone(),
|
||||
self.tun_dev.clone(),
|
||||
self.tun_inet,
|
||||
self.fake_ip,
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
MagicDnsServerInstance::new(
|
||||
self.packet_plane.clone(),
|
||||
self.global_ctx.clone(),
|
||||
self.tun_dev.clone(),
|
||||
self.tun_inet,
|
||||
self.fake_ip,
|
||||
)
|
||||
.await
|
||||
};
|
||||
|
||||
match server_result {
|
||||
Ok(server) => {
|
||||
self.server = Some(server);
|
||||
// Keep the initialized resolver owned by the runner before adding
|
||||
// its route, so route failure still reaches async server cleanup.
|
||||
self.active_fake_ip_route = self.fake_ip_route_claim();
|
||||
if let Some(claim) = &self.active_fake_ip_route {
|
||||
claim.add().await.map_err(|err| {
|
||||
anyhow::anyhow!("failed to add magic dns fake-ip route: {err}")
|
||||
})?;
|
||||
}
|
||||
tracing::info!("DnsRunner::run_once: server started");
|
||||
}
|
||||
Err(e) => {
|
||||
@@ -65,34 +166,37 @@ impl DnsRunner {
|
||||
}
|
||||
}
|
||||
|
||||
// every runner must run a client
|
||||
let client = MagicDnsClientInstance::new(self.packet_plane.clone()).await?;
|
||||
self.client = Some(client);
|
||||
self.client.as_mut().unwrap().run_and_wait().await;
|
||||
|
||||
Err(anyhow::anyhow!("Client instance exit"))
|
||||
}
|
||||
|
||||
pub async fn run(&mut self, canel_token: CancellationToken) {
|
||||
loop {
|
||||
tracing::info!("DnsRunner::run: start");
|
||||
tokio::select! {
|
||||
_ = canel_token.cancelled() => {
|
||||
self.clean_env().await;
|
||||
tracing::info!("DnsRunner::run: cancelled");
|
||||
return;
|
||||
}
|
||||
|
||||
ret = self.run_once() => {
|
||||
self.clean_env().await;
|
||||
if let Err(e) = ret {
|
||||
tracing::error!("DnsRunner::run: {:?}", e);
|
||||
} else {
|
||||
tracing::info!("DnsRunner::run: unexpected exit, server may be down");
|
||||
}
|
||||
tokio::time::sleep(Duration::from_millis(500)).await;
|
||||
}
|
||||
}
|
||||
// Every runner must run a client. Only this phase is cancellable: server
|
||||
// initialization and route updates must finish before async cleanup.
|
||||
tokio::select! {
|
||||
_ = cancel_token.cancelled() => Ok(()),
|
||||
ret = async {
|
||||
let client = MagicDnsClientInstance::new(self.packet_plane.clone()).await?;
|
||||
self.client = Some(client);
|
||||
self.client.as_mut().unwrap().run_and_wait().await;
|
||||
Err(anyhow::anyhow!("Client instance exit"))
|
||||
} => ret,
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn run(&mut self, cancel_token: CancellationToken) {
|
||||
while !cancel_token.is_cancelled() {
|
||||
tracing::info!("DnsRunner::run: start");
|
||||
let ret = self.run_once(&cancel_token).await;
|
||||
self.clean_env().await;
|
||||
if cancel_token.is_cancelled() {
|
||||
break;
|
||||
}
|
||||
if let Err(e) = ret {
|
||||
tracing::error!("DnsRunner::run: {:?}", e);
|
||||
} else {
|
||||
tracing::info!("DnsRunner::run: unexpected exit, server may be down");
|
||||
}
|
||||
tokio::select! {
|
||||
_ = cancel_token.cancelled() => break,
|
||||
_ = tokio::time::sleep(Duration::from_millis(500)) => {}
|
||||
}
|
||||
}
|
||||
tracing::info!("DnsRunner::run: cancelled");
|
||||
}
|
||||
}
|
||||
@@ -14,8 +14,10 @@ use super::{
|
||||
};
|
||||
use crate::{
|
||||
common::{
|
||||
error::Error as EtError,
|
||||
global_ctx::ArcGlobalCtx,
|
||||
ifcfg::{IfConfiger, IfConfiguerTrait},
|
||||
netns::NetNS,
|
||||
},
|
||||
instance::dns_server::{
|
||||
config::{Record, RecordBuilder, RecordType},
|
||||
@@ -51,7 +53,9 @@ use std::{collections::BTreeMap, io, net::Ipv4Addr, str::FromStr, sync::Arc, tim
|
||||
pub(super) struct MagicDnsServerInstanceData {
|
||||
dns_server: Server,
|
||||
tun_dev: Option<String>,
|
||||
net_ns: NetNS,
|
||||
fake_ip: Ipv4Addr,
|
||||
manage_fake_ip_route: bool,
|
||||
route_store: MagicDnsRecordStore,
|
||||
record_apply: tokio::sync::Mutex<()>,
|
||||
|
||||
@@ -356,12 +360,66 @@ fn get_system_config(
|
||||
}
|
||||
|
||||
impl MagicDnsServerInstance {
|
||||
async fn add_fake_ip_route(
|
||||
tun_dev_name: &str,
|
||||
fake_ip: Ipv4Addr,
|
||||
net_ns: &NetNS,
|
||||
cost: Option<i32>,
|
||||
) -> Result<(), anyhow::Error> {
|
||||
let ifcfg = IfConfiger::default();
|
||||
let _guard = net_ns.guard();
|
||||
match ifcfg.add_ipv4_route(tun_dev_name, fake_ip, 32, cost).await {
|
||||
Err(EtError::IOError(err)) if err.kind() == io::ErrorKind::AlreadyExists => {
|
||||
ifcfg.remove_ipv4_route(tun_dev_name, fake_ip, 32).await?;
|
||||
ifcfg
|
||||
.add_ipv4_route(tun_dev_name, fake_ip, 32, cost)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
ret => ret.map_err(Into::into),
|
||||
}
|
||||
}
|
||||
|
||||
async fn remove_fake_ip_route(tun_dev_name: &str, fake_ip: Ipv4Addr, net_ns: &NetNS) {
|
||||
let ifcfg = IfConfiger::default();
|
||||
let _guard = net_ns.guard();
|
||||
if let Err(err) = ifcfg.remove_ipv4_route(tun_dev_name, fake_ip, 32).await {
|
||||
tracing::warn!(
|
||||
?err,
|
||||
?tun_dev_name,
|
||||
?fake_ip,
|
||||
"remove magic dns route failed"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) async fn new(
|
||||
packet_plane: Arc<CorePacketPlane>,
|
||||
global_ctx: ArcGlobalCtx,
|
||||
tun_dev: Option<String>,
|
||||
tun_inet: Ipv4Inet,
|
||||
fake_ip: Ipv4Addr,
|
||||
) -> Result<Self, anyhow::Error> {
|
||||
Self::new_inner(packet_plane, global_ctx, tun_dev, tun_inet, fake_ip, true).await
|
||||
}
|
||||
|
||||
pub(crate) async fn new_with_external_fake_ip_route(
|
||||
packet_plane: Arc<CorePacketPlane>,
|
||||
global_ctx: ArcGlobalCtx,
|
||||
tun_dev: Option<String>,
|
||||
tun_inet: Ipv4Inet,
|
||||
fake_ip: Ipv4Addr,
|
||||
) -> Result<Self, anyhow::Error> {
|
||||
Self::new_inner(packet_plane, global_ctx, tun_dev, tun_inet, fake_ip, false).await
|
||||
}
|
||||
|
||||
async fn new_inner(
|
||||
packet_plane: Arc<CorePacketPlane>,
|
||||
global_ctx: ArcGlobalCtx,
|
||||
tun_dev: Option<String>,
|
||||
tun_inet: Ipv4Inet,
|
||||
fake_ip: Ipv4Addr,
|
||||
manage_fake_ip_route: bool,
|
||||
) -> Result<Self, anyhow::Error> {
|
||||
let tcp_listener = runtime_rpc_listener(MAGIC_DNS_INSTANCE_SOCKET_ADDR.parse()?);
|
||||
let mut rpc_server = StandAloneServer::new(tcp_listener);
|
||||
@@ -374,7 +432,8 @@ impl MagicDnsServerInstance {
|
||||
let mut dns_server = Server::new(dns_config);
|
||||
dns_server.run().await?;
|
||||
|
||||
if !tun_inet.contains(&fake_ip)
|
||||
if manage_fake_ip_route
|
||||
&& !tun_inet.contains(&fake_ip)
|
||||
&& let Some(tun_dev_name) = &tun_dev
|
||||
{
|
||||
let cost = if cfg!(target_os = "windows") {
|
||||
@@ -382,16 +441,15 @@ impl MagicDnsServerInstance {
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let ifcfg = IfConfiger {};
|
||||
ifcfg
|
||||
.add_ipv4_route(tun_dev_name, fake_ip, 32, cost)
|
||||
.await?;
|
||||
Self::add_fake_ip_route(tun_dev_name, fake_ip, &global_ctx.net_ns, cost).await?;
|
||||
}
|
||||
|
||||
let data = Arc::new(MagicDnsServerInstanceData {
|
||||
dns_server,
|
||||
tun_dev: tun_dev.clone(),
|
||||
net_ns: global_ctx.net_ns.clone(),
|
||||
fake_ip,
|
||||
manage_fake_ip_route,
|
||||
route_store: MagicDnsRecordStore::default(),
|
||||
record_apply: tokio::sync::Mutex::new(()),
|
||||
system_config: get_system_config(tun_dev.as_deref())?,
|
||||
@@ -436,14 +494,13 @@ impl MagicDnsServerInstance {
|
||||
if let Err(e) = ret {
|
||||
tracing::error!("Failed to close system config: {:?}", e);
|
||||
}
|
||||
if !self.tun_inet.contains(&self.data.fake_ip)
|
||||
&& let Some(tun_dev_name) = &self.data.tun_dev
|
||||
{
|
||||
let ifcfg = IfConfiger {};
|
||||
let _ = ifcfg
|
||||
.remove_ipv4_route(tun_dev_name, self.data.fake_ip, 32)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
if self.data.manage_fake_ip_route
|
||||
&& !self.tun_inet.contains(&self.data.fake_ip)
|
||||
&& let Some(tun_dev_name) = &self.data.tun_dev
|
||||
{
|
||||
Self::remove_fake_ip_route(tun_dev_name, self.data.fake_ip, &self.data.net_ns).await;
|
||||
}
|
||||
|
||||
self.packet_filter.close().await;
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
#[cfg(any(feature = "management-rpc", test))]
|
||||
use easytier_core::instance::manager::InstanceManager;
|
||||
#[cfg(feature = "management-rpc")]
|
||||
@@ -12,12 +15,20 @@ use easytier_core::{
|
||||
|
||||
use crate::common::global_ctx::EventBusSubscriber;
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
use super::shared_virtual_nic::{ArcSharedVirtualNicRegistry, SharedVirtualNicRegistry};
|
||||
use super::{
|
||||
composition::compose_native_core_instance, host::NativeInstanceHost,
|
||||
runtime_executor::NativeInstanceExecutor, runtime_host::NativeInstanceRuntimeHost,
|
||||
};
|
||||
|
||||
pub type NativeCoreInstance = CoreInstance<NativeInstanceHost>;
|
||||
#[cfg(all(feature = "management-rpc", feature = "tun"))]
|
||||
mod mobile_tun;
|
||||
#[cfg(all(feature = "management-rpc", feature = "tun"))]
|
||||
pub use mobile_tun::MobileTunConfig;
|
||||
#[cfg(all(feature = "management-rpc", feature = "tun", mobile))]
|
||||
pub use mobile_tun::{attach_mobile_tun_fd, get_mobile_tun_config};
|
||||
#[cfg(feature = "management-rpc")]
|
||||
pub type NativeInstanceManager = InstanceManager<NativeInstanceFactory>;
|
||||
#[cfg(feature = "management")]
|
||||
@@ -97,6 +108,8 @@ fn native_instance_manager_with_optional_runtime(
|
||||
/// Native construction Adapter for the canonical core InstanceManager.
|
||||
pub struct NativeInstanceFactory {
|
||||
process_runtime: Arc<CoreProcessRuntime>,
|
||||
#[cfg(feature = "tun")]
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
runtime_handle: Option<tokio::runtime::Handle>,
|
||||
compact_runtime: bool,
|
||||
#[cfg(feature = "logging")]
|
||||
@@ -107,6 +120,8 @@ impl NativeInstanceFactory {
|
||||
pub fn new(process_runtime: Arc<CoreProcessRuntime>) -> Self {
|
||||
Self {
|
||||
process_runtime,
|
||||
#[cfg(feature = "tun")]
|
||||
shared_virtual_nic_registry: Arc::new(Mutex::new(SharedVirtualNicRegistry::new())),
|
||||
runtime_handle: None,
|
||||
compact_runtime: false,
|
||||
#[cfg(feature = "logging")]
|
||||
@@ -126,6 +141,7 @@ impl NativeInstanceFactory {
|
||||
self
|
||||
}
|
||||
|
||||
#[cfg(feature = "management-rpc")]
|
||||
fn with_compact_runtime(mut self) -> Self {
|
||||
self.compact_runtime = true;
|
||||
self
|
||||
@@ -159,6 +175,8 @@ impl InstanceFactory for NativeInstanceFactory {
|
||||
let instance = compose_native_core_instance(
|
||||
config,
|
||||
self.process_runtime.clone(),
|
||||
#[cfg(feature = "tun")]
|
||||
self.shared_virtual_nic_registry.clone(),
|
||||
self.compact_runtime,
|
||||
executor,
|
||||
)?;
|
||||
|
||||
@@ -0,0 +1,393 @@
|
||||
#[cfg(any(mobile, test))]
|
||||
use std::collections::BTreeSet;
|
||||
|
||||
#[cfg(mobile)]
|
||||
use super::{NativeInstanceManager, NativeInstanceRuntimeHost};
|
||||
#[cfg(any(mobile, test))]
|
||||
use crate::instance::shared_virtual_nic::{
|
||||
SharedIfConfig, SharedIfConfigClaims, SharedIfConfigConflict, SharedIpv4Route,
|
||||
};
|
||||
#[cfg(mobile)]
|
||||
use easytier_core::instance::CoreInstanceState;
|
||||
|
||||
#[derive(Clone, Debug, Default, serde::Serialize)]
|
||||
pub struct MobileTunConfig {
|
||||
pub instance_ids: Vec<uuid::Uuid>,
|
||||
pub addresses: Vec<String>,
|
||||
pub routes: Vec<String>,
|
||||
pub dns: Option<String>,
|
||||
pub rejected: Vec<MobileTunMemberError>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, serde::Serialize)]
|
||||
pub struct MobileTunMemberError {
|
||||
pub instance_id: uuid::Uuid,
|
||||
pub error: String,
|
||||
}
|
||||
|
||||
#[cfg(any(mobile, test))]
|
||||
struct MobileTunMember {
|
||||
instance_id: uuid::Uuid,
|
||||
claims: SharedIfConfigClaims,
|
||||
current_claims: Option<SharedIfConfigClaims>,
|
||||
dns: bool,
|
||||
}
|
||||
|
||||
#[cfg(any(mobile, test))]
|
||||
fn magic_dns_route() -> SharedIpv4Route {
|
||||
SharedIpv4Route::new(
|
||||
crate::instance::dns_server::MAGIC_DNS_FAKE_IP
|
||||
.parse()
|
||||
.unwrap(),
|
||||
32,
|
||||
Some(1),
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(any(mobile, test))]
|
||||
fn comparable_claims(mut claims: SharedIfConfigClaims) -> SharedIfConfigClaims {
|
||||
claims.mtu = None;
|
||||
let magic_dns = magic_dns_route();
|
||||
claims
|
||||
.ipv4_routes
|
||||
.retain(|route| route.address != magic_dns.address || route.prefix != magic_dns.prefix);
|
||||
claims
|
||||
}
|
||||
|
||||
#[cfg(any(mobile, test))]
|
||||
fn select_members(
|
||||
mut members: Vec<MobileTunMember>,
|
||||
mut rejected: Vec<MobileTunMemberError>,
|
||||
) -> (Vec<MobileTunMember>, MobileTunConfig) {
|
||||
// Keep unchanged attached members ahead of new or reconfigured members.
|
||||
members.sort_by_key(|member| {
|
||||
!member.current_claims.as_ref().is_some_and(|current| {
|
||||
comparable_claims(current.clone()) == comparable_claims(member.claims.clone())
|
||||
})
|
||||
});
|
||||
let mut ifcfg = SharedIfConfig::default();
|
||||
let mut accepted = Vec::new();
|
||||
for member in members {
|
||||
let mut claims = member.claims.clone();
|
||||
if member.dns {
|
||||
claims.ipv4_routes.insert(magic_dns_route());
|
||||
}
|
||||
if let Err(error) = ifcfg.ensure_disjoint(member.instance_id, &claims) {
|
||||
rejected.push(MobileTunMemberError {
|
||||
instance_id: member.instance_id,
|
||||
error: error.to_string(),
|
||||
});
|
||||
continue;
|
||||
}
|
||||
ifcfg.apply_member_claims(member.instance_id, claims);
|
||||
accepted.push(member);
|
||||
}
|
||||
let config = config_for_members(&accepted, rejected);
|
||||
(accepted, config)
|
||||
}
|
||||
|
||||
#[cfg(any(mobile, test))]
|
||||
fn config_for_members(
|
||||
members: &[MobileTunMember],
|
||||
rejected: Vec<MobileTunMemberError>,
|
||||
) -> MobileTunConfig {
|
||||
let mut config = MobileTunConfig {
|
||||
rejected,
|
||||
..Default::default()
|
||||
};
|
||||
let mut addresses = BTreeSet::new();
|
||||
let mut routes = BTreeSet::new();
|
||||
for member in members {
|
||||
config.instance_ids.push(member.instance_id);
|
||||
for address in &member.claims.ipv4_addresses {
|
||||
addresses.insert(format!(
|
||||
"{}/{}",
|
||||
address.address(),
|
||||
address.network_length()
|
||||
));
|
||||
routes.insert(format!(
|
||||
"{}/{}",
|
||||
address.network().first_address(),
|
||||
address.network_length()
|
||||
));
|
||||
}
|
||||
for address in &member.claims.ipv6_addresses {
|
||||
addresses.insert(format!(
|
||||
"{}/{}",
|
||||
address.address(),
|
||||
address.network_length()
|
||||
));
|
||||
routes.insert(format!(
|
||||
"{}/{}",
|
||||
address.network().first_address(),
|
||||
address.network_length()
|
||||
));
|
||||
}
|
||||
for route in &member.claims.ipv4_routes {
|
||||
if let Ok(route) = cidr::Ipv4Inet::new(route.address, route.prefix) {
|
||||
routes.insert(format!(
|
||||
"{}/{}",
|
||||
route.network().first_address(),
|
||||
route.network_length()
|
||||
));
|
||||
}
|
||||
}
|
||||
for route in &member.claims.ipv6_routes {
|
||||
if let Ok(route) = cidr::Ipv6Inet::new(route.address, route.prefix) {
|
||||
routes.insert(format!(
|
||||
"{}/{}",
|
||||
route.network().first_address(),
|
||||
route.network_length()
|
||||
));
|
||||
}
|
||||
}
|
||||
if member.dns {
|
||||
config.dns = Some(crate::instance::dns_server::MAGIC_DNS_FAKE_IP.to_owned());
|
||||
routes.insert(format!(
|
||||
"{}/32",
|
||||
crate::instance::dns_server::MAGIC_DNS_FAKE_IP
|
||||
));
|
||||
}
|
||||
}
|
||||
config.instance_ids.sort();
|
||||
config.addresses = addresses.into_iter().collect();
|
||||
config.routes = routes.into_iter().collect();
|
||||
config
|
||||
}
|
||||
|
||||
#[cfg(mobile)]
|
||||
async fn collect_members(
|
||||
manager: &NativeInstanceManager,
|
||||
instance_ids: &[uuid::Uuid],
|
||||
) -> (Vec<MobileTunMember>, Vec<MobileTunMemberError>) {
|
||||
let mut members = Vec::new();
|
||||
let mut rejected = Vec::new();
|
||||
let mut seen = BTreeSet::new();
|
||||
for &instance_id in instance_ids {
|
||||
if !seen.insert(instance_id) {
|
||||
continue;
|
||||
}
|
||||
let instance = manager.instance(instance_id);
|
||||
let runtime = instance
|
||||
.as_ref()
|
||||
.filter(|instance| instance.state() == CoreInstanceState::Running)
|
||||
.and_then(|instance| instance.runtime_host::<NativeInstanceRuntimeHost>())
|
||||
.filter(|runtime| runtime.tun_enabled());
|
||||
let Some(runtime) = runtime else {
|
||||
rejected.push(MobileTunMemberError {
|
||||
instance_id,
|
||||
error: "TUN-enabled instance is not running".to_owned(),
|
||||
});
|
||||
continue;
|
||||
};
|
||||
let (claims, current_claims, dns) = runtime
|
||||
.mobile_tun_claims(&instance.as_ref().unwrap().packet_plane())
|
||||
.await;
|
||||
members.push(MobileTunMember {
|
||||
instance_id,
|
||||
claims,
|
||||
current_claims,
|
||||
dns,
|
||||
});
|
||||
}
|
||||
(members, rejected)
|
||||
}
|
||||
|
||||
#[cfg(mobile)]
|
||||
pub async fn get_mobile_tun_config(
|
||||
manager: &NativeInstanceManager,
|
||||
instance_ids: &[uuid::Uuid],
|
||||
) -> MobileTunConfig {
|
||||
let (members, rejected) = collect_members(manager, instance_ids).await;
|
||||
select_members(members, rejected).1
|
||||
}
|
||||
|
||||
#[cfg(any(mobile, test))]
|
||||
fn is_claim_conflict(error: &anyhow::Error) -> bool {
|
||||
error
|
||||
.chain()
|
||||
.any(|error| error.is::<SharedIfConfigConflict>())
|
||||
}
|
||||
|
||||
#[cfg(mobile)]
|
||||
pub async fn attach_mobile_tun_fd(
|
||||
manager: &NativeInstanceManager,
|
||||
fd: i32,
|
||||
instance_ids: Option<&[uuid::Uuid]>,
|
||||
) -> anyhow::Result<MobileTunConfig> {
|
||||
let all_ids = manager
|
||||
.instances()
|
||||
.into_iter()
|
||||
.filter(|instance| {
|
||||
instance.state() == CoreInstanceState::Running
|
||||
&& instance
|
||||
.runtime_host::<NativeInstanceRuntimeHost>()
|
||||
.is_some_and(NativeInstanceRuntimeHost::tun_enabled)
|
||||
})
|
||||
.map(|instance| instance.instance_id())
|
||||
.collect::<BTreeSet<_>>()
|
||||
.into_iter()
|
||||
.collect::<Vec<_>>();
|
||||
if fd <= 0 {
|
||||
let mut errors = Vec::new();
|
||||
for id in all_ids {
|
||||
if let Err(error) = attach_member(manager, id, 0, false, None).await {
|
||||
errors.push(error.to_string());
|
||||
}
|
||||
}
|
||||
anyhow::ensure!(
|
||||
errors.is_empty(),
|
||||
"failed to detach mobile TUN fd: {}",
|
||||
errors.join("; ")
|
||||
);
|
||||
return Ok(MobileTunConfig::default());
|
||||
}
|
||||
let (members, rejected) = collect_members(manager, instance_ids.unwrap_or(&all_ids)).await;
|
||||
let (members, mut config) = select_members(members, rejected);
|
||||
let mut attached = Vec::new();
|
||||
// Open once; a rejected member never replaces or tears down this shared fd.
|
||||
for member in members {
|
||||
match attach_member(
|
||||
manager,
|
||||
member.instance_id,
|
||||
fd,
|
||||
attached.is_empty(),
|
||||
Some(member.claims.clone()),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(()) => attached.push(member),
|
||||
Err(error) if is_claim_conflict(&error) => {
|
||||
attach_member(manager, member.instance_id, 0, false, None).await?;
|
||||
config.rejected.push(MobileTunMemberError {
|
||||
instance_id: member.instance_id,
|
||||
error: error.to_string(),
|
||||
});
|
||||
}
|
||||
Err(error) => {
|
||||
for id in &all_ids {
|
||||
let _ = attach_member(manager, *id, 0, false, None).await;
|
||||
}
|
||||
return Err(error.context("failed to attach mobile TUN fd"));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(config_for_members(&attached, config.rejected))
|
||||
}
|
||||
|
||||
#[cfg(mobile)]
|
||||
async fn attach_member(
|
||||
manager: &NativeInstanceManager,
|
||||
instance_id: uuid::Uuid,
|
||||
fd: i32,
|
||||
replace_tun_fd: bool,
|
||||
claims: Option<SharedIfConfigClaims>,
|
||||
) -> anyhow::Result<()> {
|
||||
let instance = manager
|
||||
.instance(instance_id)
|
||||
.ok_or_else(|| anyhow::anyhow!("instance {instance_id} is unavailable"))?;
|
||||
let runtime = instance
|
||||
.runtime_host::<NativeInstanceRuntimeHost>()
|
||||
.ok_or_else(|| anyhow::anyhow!("native runtime host is unavailable"))?;
|
||||
runtime
|
||||
.attach_mobile_tun_fd(fd, replace_tun_fd, claims)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::instance::shared_virtual_nic::SharedIpv6Route;
|
||||
|
||||
fn member(id: u128, address: &str) -> MobileTunMember {
|
||||
MobileTunMember {
|
||||
instance_id: uuid::Uuid::from_u128(id),
|
||||
claims: SharedIfConfigClaims {
|
||||
ipv4_addresses: BTreeSet::from([address.parse().unwrap()]),
|
||||
..Default::default()
|
||||
},
|
||||
current_claims: None,
|
||||
dns: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn keeps_attached_member_when_bootstrap_subnets_overlap() {
|
||||
let mut existing = member(2, "10.126.126.2/24");
|
||||
existing.current_claims = Some(existing.claims.clone());
|
||||
let (_, config) = select_members(vec![member(1, "10.126.126.1/24"), existing], Vec::new());
|
||||
assert_eq!(config.instance_ids, vec![uuid::Uuid::from_u128(2)]);
|
||||
assert_eq!(config.addresses, vec!["10.126.126.2/24"]);
|
||||
assert_eq!(config.routes, vec!["10.126.126.0/24"]);
|
||||
assert_eq!(config.rejected[0].instance_id, uuid::Uuid::from_u128(1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reconfigured_member_does_not_displace_unchanged_member() {
|
||||
let mut changed = member(1, "10.0.2.1/24");
|
||||
changed.current_claims = Some(member(1, "10.0.1.1/24").claims);
|
||||
let mut unchanged = member(2, "10.0.2.2/24");
|
||||
unchanged.current_claims = Some(unchanged.claims.clone());
|
||||
unchanged.current_claims.as_mut().unwrap().mtu = Some(1300);
|
||||
let (_, config) = select_members(vec![changed, unchanged], Vec::new());
|
||||
assert_eq!(config.instance_ids, vec![uuid::Uuid::from_u128(2)]);
|
||||
assert_eq!(config.rejected[0].instance_id, uuid::Uuid::from_u128(1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn projects_ipv6_and_shared_dns_and_rejects_overlapping_proxy_routes() {
|
||||
let mut first = member(1, "10.0.1.1/24");
|
||||
first.dns = true;
|
||||
first
|
||||
.claims
|
||||
.ipv6_addresses
|
||||
.insert("2001:db8::1/128".parse().unwrap());
|
||||
first
|
||||
.claims
|
||||
.ipv6_routes
|
||||
.insert(SharedIpv6Route::new("::".parse().unwrap(), 0, Some(5)));
|
||||
let mut second = member(2, "10.0.2.1/24");
|
||||
second.dns = true;
|
||||
let (_, config) = select_members(
|
||||
vec![first, second, member(3, "100.100.100.101/32")],
|
||||
Vec::new(),
|
||||
);
|
||||
assert_eq!(config.instance_ids.len(), 2);
|
||||
assert!(config.addresses.contains(&"2001:db8::1/128".to_owned()));
|
||||
assert!(config.routes.contains(&"::/0".to_owned()));
|
||||
assert_eq!(
|
||||
config
|
||||
.routes
|
||||
.iter()
|
||||
.filter(|route| *route == "100.100.100.101/32")
|
||||
.count(),
|
||||
1
|
||||
);
|
||||
assert_eq!(config.dns.as_deref(), Some("100.100.100.101"));
|
||||
assert_eq!(config.rejected[0].instance_id, uuid::Uuid::from_u128(3));
|
||||
let mut route_owner = member(2, "10.0.2.1/24");
|
||||
route_owner.claims.ipv4_routes.insert(SharedIpv4Route::new(
|
||||
"10.0.1.0".parse().unwrap(),
|
||||
24,
|
||||
None,
|
||||
));
|
||||
let (_, config) = select_members(vec![member(1, "10.0.1.1/24"), route_owner], Vec::new());
|
||||
assert_eq!(config.instance_ids.len(), 1);
|
||||
assert_eq!(config.rejected[0].instance_id, uuid::Uuid::from_u128(2));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn distinguishes_claim_conflicts_from_device_failures_through_context() {
|
||||
let mut claims = SharedIfConfig::default();
|
||||
claims.apply_member_claims(uuid::Uuid::from_u128(1), member(1, "10.0.1.1/24").claims);
|
||||
let conflict = claims
|
||||
.ensure_disjoint(uuid::Uuid::from_u128(2), &member(2, "10.0.1.2/24").claims)
|
||||
.unwrap_err();
|
||||
let error = anyhow::Error::new(crate::common::error::Error::from(conflict))
|
||||
.context("attach mobile TUN failed");
|
||||
assert!(is_claim_conflict(&error));
|
||||
assert!(!is_claim_conflict(&anyhow::Error::new(
|
||||
std::io::Error::from_raw_os_error(19)
|
||||
)));
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,9 @@ pub(crate) mod listeners;
|
||||
#[cfg(feature = "public-ipv6-provider")]
|
||||
pub(crate) mod public_ipv6_provider;
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
pub mod shared_virtual_nic;
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
pub mod virtual_nic;
|
||||
|
||||
|
||||
@@ -1,13 +1,16 @@
|
||||
use std::sync::Arc;
|
||||
|
||||
#[cfg(feature = "web-client")]
|
||||
use easytier_core::config::runtime::CoreInstanceRuntimeConfig;
|
||||
use easytier_core::{
|
||||
config::runtime::CoreInstanceRuntimeConfig, gateway::dhcp::DhcpIpv4Host,
|
||||
host::packet::HostPacketReceiver, instance::CorePacketPlane,
|
||||
gateway::dhcp::DhcpIpv4Host, host::packet::HostPacketReceiver, instance::CorePacketPlane,
|
||||
};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use crate::common::global_ctx::ArcGlobalCtx;
|
||||
#[cfg(feature = "tun")]
|
||||
use crate::instance::shared_virtual_nic::ArcSharedVirtualNicRegistry;
|
||||
|
||||
mod event_journal;
|
||||
mod implementation;
|
||||
@@ -42,16 +45,30 @@ pub(crate) struct NativeInstanceRuntimeHost {
|
||||
|
||||
impl NativeInstanceRuntimeHost {
|
||||
#[cfg(test)]
|
||||
pub(crate) fn new(global_ctx: ArcGlobalCtx) -> Arc<Self> {
|
||||
Self::new_with_executor(global_ctx, None)
|
||||
pub(crate) fn new(
|
||||
global_ctx: ArcGlobalCtx,
|
||||
#[cfg(feature = "tun")] shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
) -> Arc<Self> {
|
||||
Self::new_with_executor(
|
||||
global_ctx,
|
||||
#[cfg(feature = "tun")]
|
||||
shared_virtual_nic_registry,
|
||||
None,
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) fn new_with_executor(
|
||||
global_ctx: ArcGlobalCtx,
|
||||
#[cfg(feature = "tun")] shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
executor: Option<NativeInstanceExecutor>,
|
||||
) -> Arc<Self> {
|
||||
let cancel = CancellationToken::new();
|
||||
let tun = NativeTunRuntime::new(global_ctx.clone(), cancel.clone());
|
||||
let tun = NativeTunRuntime::new(
|
||||
global_ctx.clone(),
|
||||
cancel.clone(),
|
||||
#[cfg(feature = "tun")]
|
||||
shared_virtual_nic_registry,
|
||||
);
|
||||
let event_journal = EventJournal::new(&global_ctx);
|
||||
Arc::new(Self {
|
||||
global_ctx,
|
||||
@@ -128,10 +145,44 @@ impl NativeInstanceRuntimeHost {
|
||||
self.global_ctx.subscribe()
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "tun", mobile))]
|
||||
pub(crate) fn tun_enabled(&self) -> bool {
|
||||
!self.global_ctx.get_flags().no_tun
|
||||
}
|
||||
|
||||
fn attach_runtime_tun_fd(&self, fd: i32) -> anyhow::Result<()> {
|
||||
self.tun.attach_fd(fd)
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "tun", mobile))]
|
||||
pub(crate) async fn attach_mobile_tun_fd(
|
||||
&self,
|
||||
fd: i32,
|
||||
replace_tun_fd: bool,
|
||||
claims: Option<crate::instance::shared_virtual_nic::SharedIfConfigClaims>,
|
||||
) -> anyhow::Result<()> {
|
||||
self.tun.attach_mobile_fd(fd, replace_tun_fd, claims).await
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "tun", mobile))]
|
||||
pub(crate) async fn mobile_tun_claims(
|
||||
&self,
|
||||
packet_plane: &CorePacketPlane,
|
||||
) -> (
|
||||
crate::instance::shared_virtual_nic::SharedIfConfigClaims,
|
||||
Option<crate::instance::shared_virtual_nic::SharedIfConfigClaims>,
|
||||
bool,
|
||||
) {
|
||||
(
|
||||
crate::instance::virtual_nic::NicCtx::mobile_tun_claims(&self.global_ctx, packet_plane)
|
||||
.await,
|
||||
self.tun.current_mobile_tun_claims().await,
|
||||
cfg!(feature = "magic-dns")
|
||||
&& self.global_ctx.get_flags().accept_dns
|
||||
&& self.global_ctx.get_ipv4().is_some(),
|
||||
)
|
||||
}
|
||||
|
||||
fn install_packet_receiver(&self, receiver: HostPacketReceiver) -> anyhow::Result<()> {
|
||||
self.tun.install_packet_receiver(receiver)
|
||||
}
|
||||
@@ -145,6 +196,16 @@ mod tests {
|
||||
global_ctx::{GlobalCtx, GlobalCtxEvent},
|
||||
};
|
||||
|
||||
fn runtime_host(global_ctx: ArcGlobalCtx) -> Arc<NativeInstanceRuntimeHost> {
|
||||
NativeInstanceRuntimeHost::new(
|
||||
global_ctx,
|
||||
#[cfg(feature = "tun")]
|
||||
Arc::new(tokio::sync::Mutex::new(
|
||||
crate::instance::shared_virtual_nic::SharedVirtualNicRegistry::new(),
|
||||
)),
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(feature = "web-client")]
|
||||
fn runtime_config(config: &TomlConfig) -> CoreInstanceRuntimeConfig {
|
||||
let normalized = easytier_core::instance::CoreInstanceConfig::from_toml(config).unwrap();
|
||||
@@ -157,7 +218,7 @@ mod tests {
|
||||
#[test]
|
||||
fn runtime_host_owns_event_subscription_context() {
|
||||
let global_ctx = Arc::new(GlobalCtx::new(TomlConfig::default()));
|
||||
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone());
|
||||
let runtime_host = runtime_host(global_ctx.clone());
|
||||
let mut events = runtime_host.subscribe_event();
|
||||
|
||||
global_ctx.issue_event(GlobalCtxEvent::CredentialChanged);
|
||||
@@ -178,7 +239,7 @@ mod tests {
|
||||
config.set_ipv4(Some("10.20.0.1/24".parse().unwrap()));
|
||||
config.set_ipv6(Some("fd00::1/64".parse().unwrap()));
|
||||
let global_ctx = Arc::new(GlobalCtx::new(config.clone()));
|
||||
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone());
|
||||
let runtime_host = runtime_host(global_ctx.clone());
|
||||
|
||||
assert_eq!(global_ctx.get_hostname(), "before");
|
||||
assert_eq!(global_ctx.get_ipv4(), Some("10.20.0.1/24".parse().unwrap()));
|
||||
@@ -220,7 +281,7 @@ mod tests {
|
||||
let config = TomlConfig::default();
|
||||
config.set_dhcp(true);
|
||||
let global_ctx = Arc::new(GlobalCtx::new(config.clone()));
|
||||
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone());
|
||||
let runtime_host = runtime_host(global_ctx.clone());
|
||||
let lease = "10.20.0.7/24".parse().unwrap();
|
||||
global_ctx.set_ipv4(Some(lease));
|
||||
|
||||
|
||||
@@ -3,9 +3,9 @@ use easytier_core::instance::CorePacketPlane;
|
||||
#[cfg(feature = "magic-dns")]
|
||||
use tokio_util::{sync::CancellationToken, task::AbortOnDropHandle};
|
||||
|
||||
use crate::common::global_ctx::ArcGlobalCtx;
|
||||
#[cfg(feature = "magic-dns")]
|
||||
use crate::instance::dns_server::{MAGIC_DNS_FAKE_IP, runner::DnsRunner};
|
||||
use crate::{common::global_ctx::ArcGlobalCtx, instance::virtual_nic::NicBackend};
|
||||
|
||||
#[derive(Default)]
|
||||
pub(super) struct MagicDnsRuntime {
|
||||
@@ -26,6 +26,7 @@ impl MagicDnsRuntime {
|
||||
packet_plane: std::sync::Arc<CorePacketPlane>,
|
||||
tun_dev: Option<String>,
|
||||
tun_ip: Ipv4Inet,
|
||||
shared_route_backend: Option<NicBackend>,
|
||||
) -> Self {
|
||||
let active = global_ctx.get_flags().accept_dns.then(|| {
|
||||
let mut runner = DnsRunner::new(
|
||||
@@ -34,7 +35,8 @@ impl MagicDnsRuntime {
|
||||
tun_dev,
|
||||
tun_ip,
|
||||
MAGIC_DNS_FAKE_IP.parse().unwrap(),
|
||||
);
|
||||
)
|
||||
.with_shared_route_backend(shared_route_backend);
|
||||
let cancel = CancellationToken::new();
|
||||
let task_cancel = cancel.clone();
|
||||
let task = tokio::spawn(async move {
|
||||
@@ -54,6 +56,7 @@ impl MagicDnsRuntime {
|
||||
_packet_plane: std::sync::Arc<CorePacketPlane>,
|
||||
_tun_dev: Option<String>,
|
||||
_tun_ip: Ipv4Inet,
|
||||
_shared_route_backend: Option<NicBackend>,
|
||||
) -> Self {
|
||||
Self::default()
|
||||
}
|
||||
|
||||
@@ -3,11 +3,43 @@ use std::{
|
||||
sync::{Arc, OnceLock},
|
||||
};
|
||||
|
||||
use easytier_core::host::packet::HostPacketReceiver;
|
||||
use easytier_core::{host::packet::HostPacketReceiver, instance::CorePacketPlane};
|
||||
use tokio::{sync::Mutex, task::JoinSet};
|
||||
|
||||
use super::MagicDnsRuntime;
|
||||
use crate::instance::virtual_nic::NicCtx;
|
||||
use crate::{
|
||||
common::{error::Error, global_ctx::ArcGlobalCtx},
|
||||
instance::{shared_virtual_nic::ArcSharedVirtualNicRegistry, virtual_nic::NicCtx},
|
||||
};
|
||||
|
||||
pub(super) async fn create_nic_ctx(
|
||||
global_ctx: ArcGlobalCtx,
|
||||
packet_plane: Arc<CorePacketPlane>,
|
||||
receiver: Arc<Mutex<HostPacketReceiver>>,
|
||||
close_notifier: Arc<tokio::sync::Notify>,
|
||||
registry: ArcSharedVirtualNicRegistry,
|
||||
) -> Result<NicCtx, Error> {
|
||||
#[cfg(not(mobile))]
|
||||
if global_ctx.get_flags().dev_name.is_empty() {
|
||||
return Ok(NicCtx::new(
|
||||
global_ctx,
|
||||
packet_plane,
|
||||
receiver,
|
||||
close_notifier,
|
||||
));
|
||||
}
|
||||
|
||||
let member_id = global_ctx.get_id();
|
||||
NicCtx::new_shared(
|
||||
global_ctx,
|
||||
packet_plane,
|
||||
receiver,
|
||||
close_notifier,
|
||||
registry,
|
||||
member_id,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
struct NicCtxContainer {
|
||||
_nic_ctx: Option<Box<dyn Any + Send>>,
|
||||
@@ -65,6 +97,15 @@ impl TunNicState {
|
||||
drop(old);
|
||||
}
|
||||
|
||||
#[cfg(mobile)]
|
||||
pub(super) async fn mobile_tun_claims(
|
||||
&self,
|
||||
) -> Option<crate::instance::shared_virtual_nic::SharedIfConfigClaims> {
|
||||
let nic = self.nic_ctx.lock().await;
|
||||
let nic = nic.as_ref()?._nic_ctx.as_ref()?.downcast_ref::<NicCtx>()?;
|
||||
nic.current_mobile_tun_claims().await
|
||||
}
|
||||
|
||||
pub(super) async fn drain(&self) {
|
||||
self.stop().await;
|
||||
let receiver = self.receiver();
|
||||
|
||||
@@ -14,28 +14,37 @@ use tokio::{
|
||||
};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use super::{MagicDnsRuntime, tun_common::TunNicState};
|
||||
use super::{
|
||||
MagicDnsRuntime,
|
||||
tun_common::{TunNicState, create_nic_ctx},
|
||||
};
|
||||
use crate::{
|
||||
common::{
|
||||
error::Error,
|
||||
global_ctx::{ArcGlobalCtx, GlobalCtxEvent},
|
||||
},
|
||||
instance::virtual_nic::NicCtx,
|
||||
instance::shared_virtual_nic::ArcSharedVirtualNicRegistry,
|
||||
};
|
||||
|
||||
pub(super) struct NativeTunRuntime {
|
||||
global_ctx: ArcGlobalCtx,
|
||||
cancel: CancellationToken,
|
||||
nic: TunNicState,
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
static_ip_task: Mutex<Option<JoinHandle<()>>>,
|
||||
}
|
||||
|
||||
impl NativeTunRuntime {
|
||||
pub(super) fn new(global_ctx: ArcGlobalCtx, cancel: CancellationToken) -> Self {
|
||||
pub(super) fn new(
|
||||
global_ctx: ArcGlobalCtx,
|
||||
cancel: CancellationToken,
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
) -> Self {
|
||||
Self {
|
||||
global_ctx,
|
||||
cancel,
|
||||
nic: TunNicState::empty(),
|
||||
shared_virtual_nic_registry,
|
||||
static_ip_task: Mutex::new(None),
|
||||
}
|
||||
}
|
||||
@@ -67,6 +76,7 @@ impl NativeTunRuntime {
|
||||
let cancel = self.cancel.clone();
|
||||
let global_ctx = self.global_ctx.clone();
|
||||
let receiver = self.nic.receiver();
|
||||
let shared_virtual_nic_registry = self.shared_virtual_nic_registry.clone();
|
||||
let (output, first_round) = oneshot::channel();
|
||||
let task = tokio::spawn(async move {
|
||||
let mut output = Some(output);
|
||||
@@ -76,12 +86,29 @@ impl NativeTunRuntime {
|
||||
return;
|
||||
}
|
||||
let closed = Arc::new(Notify::new());
|
||||
let mut nic = NicCtx::new(
|
||||
let mut nic = match create_nic_ctx(
|
||||
global_ctx.clone(),
|
||||
packet_plane.clone(),
|
||||
receiver.clone(),
|
||||
closed.clone(),
|
||||
);
|
||||
shared_virtual_nic_registry.clone(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(nic) => nic,
|
||||
Err(error) => {
|
||||
if let Some(output) = output.take() {
|
||||
let _ = output.send(Err(error));
|
||||
return;
|
||||
}
|
||||
tracing::error!(?error, "failed to create native interface context");
|
||||
tokio::select! {
|
||||
_ = cancel.cancelled() => return,
|
||||
_ = tokio::time::sleep(Duration::from_secs(1)) => {}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let result = tokio::select! {
|
||||
biased;
|
||||
_ = cancel.cancelled() => {
|
||||
@@ -104,11 +131,13 @@ impl NativeTunRuntime {
|
||||
}
|
||||
|
||||
let magic_dns = if let Some(ip) = ipv4 {
|
||||
let shared_route_backend = nic.shared_route_backend_for_dns();
|
||||
MagicDnsRuntime::start(
|
||||
global_ctx.clone(),
|
||||
packet_plane.clone(),
|
||||
nic.ifname().await,
|
||||
ip,
|
||||
shared_route_backend,
|
||||
)
|
||||
} else {
|
||||
MagicDnsRuntime::default()
|
||||
@@ -161,6 +190,7 @@ impl NativeTunRuntime {
|
||||
nic: self.nic.clone(),
|
||||
closed: Arc::new(Notify::new()),
|
||||
packet_plane,
|
||||
shared_virtual_nic_registry: self.shared_virtual_nic_registry.clone(),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -172,6 +202,7 @@ struct NativeDhcpIpv4Host {
|
||||
nic: TunNicState,
|
||||
closed: Arc<Notify>,
|
||||
packet_plane: Arc<CorePacketPlane>,
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
}
|
||||
|
||||
impl NativeDhcpIpv4Host {
|
||||
@@ -199,21 +230,25 @@ impl NativeDhcpIpv4Host {
|
||||
return Ok(Some(ip));
|
||||
}
|
||||
|
||||
let mut nic = NicCtx::new(
|
||||
let mut nic = create_nic_ctx(
|
||||
self.global_ctx.clone(),
|
||||
self.packet_plane.clone(),
|
||||
self.nic.receiver(),
|
||||
self.closed.clone(),
|
||||
);
|
||||
self.shared_virtual_nic_registry.clone(),
|
||||
)
|
||||
.await?;
|
||||
tokio::select! {
|
||||
_ = self.cancel.cancelled() => anyhow::bail!("instance is closing; DHCP update cancelled"),
|
||||
result = nic.run(Some(ip), self.global_ctx.get_ipv6()) => result?,
|
||||
}
|
||||
let shared_route_backend = nic.shared_route_backend_for_dns();
|
||||
let magic_dns = MagicDnsRuntime::start(
|
||||
self.global_ctx.clone(),
|
||||
self.packet_plane.clone(),
|
||||
nic.ifname().await,
|
||||
ip,
|
||||
shared_route_backend,
|
||||
);
|
||||
self.nic.install(nic, magic_dns).await;
|
||||
self.global_ctx.set_ipv4(Some(ip));
|
||||
|
||||
@@ -8,26 +8,41 @@ use easytier_core::{
|
||||
instance::CorePacketPlane,
|
||||
};
|
||||
use futures::FutureExt as _;
|
||||
use tokio::sync::{Mutex, Notify, mpsc};
|
||||
use tokio::sync::{Mutex, Notify, mpsc, oneshot};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
|
||||
use super::{MagicDnsRuntime, tun_common::TunNicState};
|
||||
use super::{
|
||||
MagicDnsRuntime,
|
||||
tun_common::{TunNicState, create_nic_ctx},
|
||||
};
|
||||
use crate::{
|
||||
common::global_ctx::{ArcGlobalCtx, GlobalCtxEvent},
|
||||
instance::virtual_nic::NicCtx,
|
||||
instance::shared_virtual_nic::ArcSharedVirtualNicRegistry,
|
||||
};
|
||||
|
||||
struct MobileTunAttachment {
|
||||
fd: i32,
|
||||
replace_tun_fd: bool,
|
||||
completion: Option<oneshot::Sender<anyhow::Result<()>>>,
|
||||
claims: Option<crate::instance::shared_virtual_nic::SharedIfConfigClaims>,
|
||||
}
|
||||
|
||||
pub(super) struct NativeTunRuntime {
|
||||
global_ctx: ArcGlobalCtx,
|
||||
cancel: CancellationToken,
|
||||
nic: TunNicState,
|
||||
tun_fd: mpsc::Sender<i32>,
|
||||
tun_fd_receiver: Mutex<Option<mpsc::Receiver<i32>>>,
|
||||
tun_fd: mpsc::Sender<MobileTunAttachment>,
|
||||
tun_fd_receiver: Mutex<Option<mpsc::Receiver<MobileTunAttachment>>>,
|
||||
task: Mutex<Option<tokio::task::JoinHandle<()>>>,
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
}
|
||||
|
||||
impl NativeTunRuntime {
|
||||
pub(super) fn new(global_ctx: ArcGlobalCtx, cancel: CancellationToken) -> Self {
|
||||
pub(super) fn new(
|
||||
global_ctx: ArcGlobalCtx,
|
||||
cancel: CancellationToken,
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
) -> Self {
|
||||
let (tun_fd, tun_fd_receiver) = mpsc::channel(16);
|
||||
Self {
|
||||
global_ctx,
|
||||
@@ -36,6 +51,7 @@ impl NativeTunRuntime {
|
||||
tun_fd,
|
||||
tun_fd_receiver: Mutex::new(Some(tun_fd_receiver)),
|
||||
task: Mutex::new(None),
|
||||
shared_virtual_nic_registry,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -51,23 +67,39 @@ impl NativeTunRuntime {
|
||||
global_ctx: ArcGlobalCtx,
|
||||
packet_plane: Arc<CorePacketPlane>,
|
||||
fd: i32,
|
||||
replace_tun_fd: bool,
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
claims: Option<crate::instance::shared_virtual_nic::SharedIfConfigClaims>,
|
||||
) -> anyhow::Result<()> {
|
||||
nic_state.drain().await;
|
||||
if fd <= 0 {
|
||||
return Ok(());
|
||||
}
|
||||
let closed = Arc::new(Notify::new());
|
||||
let mut nic = NicCtx::new(
|
||||
let mut nic = create_nic_ctx(
|
||||
global_ctx.clone(),
|
||||
packet_plane.clone(),
|
||||
nic_state.receiver(),
|
||||
closed,
|
||||
);
|
||||
nic.run_for_mobile(fd).await.context("add ip failed")?;
|
||||
let magic_dns = global_ctx
|
||||
.get_ipv4()
|
||||
.map(|ip| MagicDnsRuntime::start(global_ctx, packet_plane, None, ip))
|
||||
.unwrap_or_default();
|
||||
shared_virtual_nic_registry,
|
||||
)
|
||||
.await?;
|
||||
let claims = match claims {
|
||||
Some(claims) => claims,
|
||||
None => {
|
||||
crate::instance::virtual_nic::NicCtx::mobile_tun_claims(&global_ctx, &packet_plane)
|
||||
.await
|
||||
}
|
||||
};
|
||||
nic.run_for_mobile(fd, replace_tun_fd, claims)
|
||||
.await
|
||||
.context("attach mobile TUN failed")?;
|
||||
let magic_dns = if let Some(ip) = global_ctx.get_ipv4() {
|
||||
let shared_route_backend = nic.shared_route_backend_for_dns();
|
||||
MagicDnsRuntime::start(global_ctx, packet_plane, None, ip, shared_route_backend)
|
||||
} else {
|
||||
MagicDnsRuntime::default()
|
||||
};
|
||||
nic_state.install(nic, magic_dns).await;
|
||||
Ok(())
|
||||
}
|
||||
@@ -80,28 +112,44 @@ impl NativeTunRuntime {
|
||||
let nic_state = self.nic.clone();
|
||||
let global_ctx = self.global_ctx.clone();
|
||||
let cancel = self.cancel.clone();
|
||||
let shared_virtual_nic_registry = self.shared_virtual_nic_registry.clone();
|
||||
self.task.lock().await.replace(tokio::spawn(async move {
|
||||
loop {
|
||||
let fd = tokio::select! {
|
||||
let attachment = tokio::select! {
|
||||
_ = cancel.cancelled() => return,
|
||||
fd = tun_fds.recv() => match fd { Some(fd) => fd, None => return },
|
||||
attachment = tun_fds.recv() => match attachment {
|
||||
Some(attachment) => attachment,
|
||||
None => return,
|
||||
},
|
||||
};
|
||||
if let Err(error) = Self::install_mobile_tun(
|
||||
nic_state.clone(),
|
||||
global_ctx.clone(),
|
||||
packet_plane.clone(),
|
||||
fd,
|
||||
)
|
||||
.await
|
||||
{
|
||||
let result = if attachment.fd <= 0 {
|
||||
nic_state.drain().await;
|
||||
Ok(())
|
||||
} else {
|
||||
Self::install_mobile_tun(
|
||||
nic_state.clone(),
|
||||
global_ctx.clone(),
|
||||
packet_plane.clone(),
|
||||
attachment.fd,
|
||||
attachment.replace_tun_fd,
|
||||
shared_virtual_nic_registry.clone(),
|
||||
attachment.claims,
|
||||
)
|
||||
.await
|
||||
};
|
||||
if let Err(error) = &result {
|
||||
tracing::error!(?error, "failed to attach mobile TUN fd");
|
||||
}
|
||||
if let Some(completion) = attachment.completion {
|
||||
let _ = completion.send(result);
|
||||
}
|
||||
}
|
||||
}));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) async fn shutdown(&self) {
|
||||
self.cancel.cancel();
|
||||
if let Some(task) = self.task.lock().await.take() {
|
||||
let _ = task.await;
|
||||
}
|
||||
@@ -110,10 +158,43 @@ impl NativeTunRuntime {
|
||||
|
||||
pub(super) fn attach_fd(&self, fd: i32) -> anyhow::Result<()> {
|
||||
self.tun_fd
|
||||
.try_send(fd)
|
||||
.try_send(MobileTunAttachment {
|
||||
fd,
|
||||
replace_tun_fd: true,
|
||||
completion: None,
|
||||
claims: None,
|
||||
})
|
||||
.map_err(|error| anyhow::anyhow!("failed to send TUN fd: {error}"))
|
||||
}
|
||||
|
||||
pub(super) async fn attach_mobile_fd(
|
||||
&self,
|
||||
fd: i32,
|
||||
replace_tun_fd: bool,
|
||||
claims: Option<crate::instance::shared_virtual_nic::SharedIfConfigClaims>,
|
||||
) -> anyhow::Result<()> {
|
||||
if self.task.lock().await.is_none() {
|
||||
anyhow::bail!("mobile TUN runtime is not running");
|
||||
}
|
||||
|
||||
let (completion, result) = oneshot::channel();
|
||||
tokio::select! {
|
||||
_ = self.cancel.cancelled() => anyhow::bail!("instance is closing; TUN attachment cancelled"),
|
||||
send_result = self.tun_fd.send(MobileTunAttachment {
|
||||
fd,
|
||||
replace_tun_fd,
|
||||
completion: Some(completion),
|
||||
claims,
|
||||
}) => send_result.map_err(|error| anyhow::anyhow!("failed to send TUN fd: {error}"))?,
|
||||
}
|
||||
|
||||
tokio::select! {
|
||||
_ = self.cancel.cancelled() => anyhow::bail!("instance is closing; TUN attachment cancelled"),
|
||||
result = result => result
|
||||
.map_err(|_| anyhow::anyhow!("mobile TUN runtime stopped before attachment completed"))?,
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn dhcp_host(
|
||||
&self,
|
||||
operation: Arc<Mutex<()>>,
|
||||
@@ -127,6 +208,12 @@ impl NativeTunRuntime {
|
||||
closed: Arc::new(Notify::new()),
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) async fn current_mobile_tun_claims(
|
||||
&self,
|
||||
) -> Option<crate::instance::shared_virtual_nic::SharedIfConfigClaims> {
|
||||
self.nic.mobile_tun_claims().await
|
||||
}
|
||||
}
|
||||
|
||||
struct NativeDhcpIpv4Host {
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -7,6 +7,8 @@ use easytier_core::{
|
||||
process_runtime::CoreProcessRuntime,
|
||||
};
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
use crate::instance::shared_virtual_nic::{ArcSharedVirtualNicRegistry, SharedVirtualNicRegistry};
|
||||
use crate::{
|
||||
common::global_ctx::{ArcGlobalCtx, GlobalCtx},
|
||||
instance::{
|
||||
@@ -15,6 +17,8 @@ use crate::{
|
||||
},
|
||||
socket::udp::RuntimeUdpSocket,
|
||||
};
|
||||
#[cfg(feature = "tun")]
|
||||
use tokio::sync::Mutex;
|
||||
|
||||
pub(crate) struct TestInstance {
|
||||
core: Arc<NativeCoreInstance>,
|
||||
@@ -26,7 +30,27 @@ impl TestInstance {
|
||||
config: TomlConfig,
|
||||
process_runtime: Arc<CoreProcessRuntime>,
|
||||
) -> Self {
|
||||
Self::compose(config, process_runtime, |_| {})
|
||||
Self::compose(
|
||||
config,
|
||||
process_runtime,
|
||||
#[cfg(feature = "tun")]
|
||||
Arc::new(Mutex::new(SharedVirtualNicRegistry::new())),
|
||||
|_| {},
|
||||
)
|
||||
}
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
pub fn new_with_process_runtime_and_shared_virtual_nic_registry(
|
||||
config: TomlConfig,
|
||||
process_runtime: Arc<CoreProcessRuntime>,
|
||||
shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
) -> Self {
|
||||
Self::compose(config, process_runtime, shared_virtual_nic_registry, |_| {})
|
||||
}
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
pub fn new_shared_virtual_nic_registry() -> ArcSharedVirtualNicRegistry {
|
||||
Arc::new(Mutex::new(SharedVirtualNicRegistry::new()))
|
||||
}
|
||||
|
||||
pub fn new_with_process_runtime_and_stun_provider(
|
||||
@@ -35,14 +59,21 @@ impl TestInstance {
|
||||
provider: Box<dyn StunSocketMapper<RuntimeUdpSocket>>,
|
||||
) -> Self {
|
||||
let provider: Arc<dyn StunSocketMapper<RuntimeUdpSocket>> = Arc::from(provider);
|
||||
Self::compose(config, process_runtime, move |adapters| {
|
||||
adapters.replace_stun_provider(provider);
|
||||
})
|
||||
Self::compose(
|
||||
config,
|
||||
process_runtime,
|
||||
#[cfg(feature = "tun")]
|
||||
Arc::new(Mutex::new(SharedVirtualNicRegistry::new())),
|
||||
move |adapters| {
|
||||
adapters.replace_stun_provider(provider);
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
fn compose(
|
||||
config: TomlConfig,
|
||||
process_runtime: Arc<CoreProcessRuntime>,
|
||||
#[cfg(feature = "tun")] shared_virtual_nic_registry: ArcSharedVirtualNicRegistry,
|
||||
customize: impl FnOnce(
|
||||
&mut easytier_core::instance::CoreHostAdapters<
|
||||
crate::instance::host::NativeInstanceHost,
|
||||
@@ -50,7 +81,11 @@ impl TestInstance {
|
||||
),
|
||||
) -> Self {
|
||||
let global_ctx = Arc::new(GlobalCtx::new(config.clone()));
|
||||
let runtime_host = NativeInstanceRuntimeHost::new(global_ctx.clone());
|
||||
let runtime_host = NativeInstanceRuntimeHost::new(
|
||||
global_ctx.clone(),
|
||||
#[cfg(feature = "tun")]
|
||||
shared_virtual_nic_registry,
|
||||
);
|
||||
let mut adapters = runtime_core_host_adapters_with_packet_egress(
|
||||
global_ctx.clone(),
|
||||
process_runtime,
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -4,6 +4,9 @@ mod three_node;
|
||||
#[cfg(target_os = "linux")]
|
||||
mod network;
|
||||
|
||||
#[cfg(all(target_os = "linux", feature = "tun"))]
|
||||
mod shared_virtual_nic;
|
||||
|
||||
mod ipv6_test;
|
||||
|
||||
#[cfg(target_os = "linux")]
|
||||
|
||||
@@ -0,0 +1,874 @@
|
||||
use std::{net::Ipv4Addr, process::Command, sync::Arc, time::Duration};
|
||||
|
||||
use easytier_core::{config::PeerId, process_runtime::CoreProcessRuntime};
|
||||
|
||||
use super::{
|
||||
InstanceTestExt as _, create_netns, del_netns, drop_insts, network::TestNetwork, ping_test,
|
||||
};
|
||||
use crate::{
|
||||
common::{
|
||||
config::{ConfigLoader, NetworkIdentity, TomlConfigLoader},
|
||||
netns::{NetNS, ROOT_NETNS_NAME},
|
||||
},
|
||||
instance::{
|
||||
shared_virtual_nic::{ArcSharedVirtualNicRegistry, SharedIpv4Route},
|
||||
test_instance::TestInstance as Instance,
|
||||
},
|
||||
tunnel::common::tests::wait_for_condition,
|
||||
};
|
||||
|
||||
const PROXY_CIDR: &str = "10.1.2.0/24";
|
||||
const WAIT: Duration = Duration::from_secs(10);
|
||||
|
||||
#[derive(Clone)]
|
||||
struct SharedTestRuntime {
|
||||
process: Arc<CoreProcessRuntime>,
|
||||
registry: ArcSharedVirtualNicRegistry,
|
||||
}
|
||||
|
||||
impl SharedTestRuntime {
|
||||
fn new() -> Self {
|
||||
Self {
|
||||
process: CoreProcessRuntime::new(),
|
||||
registry: Instance::new_shared_virtual_nic_registry(),
|
||||
}
|
||||
}
|
||||
|
||||
fn instance(&self, config: TomlConfigLoader) -> Instance {
|
||||
Instance::new_with_process_runtime_and_shared_virtual_nic_registry(
|
||||
config,
|
||||
self.process.clone(),
|
||||
self.registry.clone(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
fn test_config(
|
||||
instance_name: &str,
|
||||
network_name: &str,
|
||||
network_secret: &str,
|
||||
netns: Option<&str>,
|
||||
dev_name: Option<&str>,
|
||||
ipv4: &str,
|
||||
) -> TomlConfigLoader {
|
||||
let config = TomlConfigLoader::default();
|
||||
config.set_inst_name(instance_name.to_owned());
|
||||
config.set_network_identity(NetworkIdentity::new(
|
||||
network_name.to_owned(),
|
||||
network_secret.to_owned(),
|
||||
));
|
||||
config.set_netns(netns.map(str::to_owned));
|
||||
config.set_ipv4(Some(ipv4.parse().unwrap()));
|
||||
config.set_ipv6(None);
|
||||
config.set_dhcp(false);
|
||||
config.set_listeners(vec![]);
|
||||
config.set_socks5_portal(None);
|
||||
|
||||
let mut flags = config.get_flags();
|
||||
flags.dev_name = dev_name.unwrap_or_default().to_owned();
|
||||
flags.enable_ipv6 = false;
|
||||
config.set_flags(flags);
|
||||
config
|
||||
}
|
||||
|
||||
fn test_dev_name() -> String {
|
||||
format!("st{:08x}", rand::random::<u32>())
|
||||
}
|
||||
|
||||
fn short_name(prefix: &str) -> String {
|
||||
format!("{prefix}{:04x}", rand::random::<u16>())
|
||||
}
|
||||
|
||||
struct TestNetnsGuard {
|
||||
name: String,
|
||||
}
|
||||
|
||||
impl TestNetnsGuard {
|
||||
fn new(name: String, ipv4: &str, ipv6: &str) -> Self {
|
||||
let guard = Self { name };
|
||||
del_netns(&guard.name);
|
||||
create_netns(&guard.name, ipv4, ipv6);
|
||||
guard
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for TestNetnsGuard {
|
||||
fn drop(&mut self) {
|
||||
del_netns(&self.name);
|
||||
}
|
||||
}
|
||||
|
||||
struct ProxyLab {
|
||||
network: TestNetwork,
|
||||
}
|
||||
|
||||
impl ProxyLab {
|
||||
fn new() -> Self {
|
||||
let mut network = TestNetwork::new();
|
||||
network.create_namespace("source", "10.1.1.1/24", "fd11::1/64");
|
||||
network.create_namespace("owner", "10.1.2.3/24", "fd12::3/64");
|
||||
network.create_namespace("target", "10.1.2.4/24", "fd12::4/64");
|
||||
network.create_bridge("bridge");
|
||||
network.add_to_bridge("bridge", "owner");
|
||||
network.add_to_bridge("bridge", "target");
|
||||
Self { network }
|
||||
}
|
||||
}
|
||||
|
||||
async fn wait_tun_ready(instance: &Instance, expected: &str) {
|
||||
wait_for_condition(
|
||||
|| async { instance.get_global_ctx().get_tun_device_name().as_deref() == Some(expected) },
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
fn proxy_route_exists(
|
||||
routes: &[easytier_proto::core_peer::peer::Route],
|
||||
peer_id: PeerId,
|
||||
proxy_cidr: &str,
|
||||
) -> bool {
|
||||
routes
|
||||
.iter()
|
||||
.any(|route| route.peer_id == peer_id && route.proxy_cidrs.iter().any(|c| c == proxy_cidr))
|
||||
}
|
||||
|
||||
async fn wait_proxy_route(instance: &Instance, peer_id: PeerId, proxy_cidr: &str) {
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
proxy_route_exists(
|
||||
&instance.get_core_instance().route_snapshots().await,
|
||||
peer_id,
|
||||
proxy_cidr,
|
||||
)
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
async fn wait_proxy_route_absent(instance: &Instance, peer_id: PeerId, proxy_cidr: &str) {
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
!proxy_route_exists(
|
||||
&instance.get_core_instance().route_snapshots().await,
|
||||
peer_id,
|
||||
proxy_cidr,
|
||||
)
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
fn ipv4_route_exists_in_ns(ns: &str, needle: &str) -> bool {
|
||||
let _root = NetNS::new(Some(ROOT_NETNS_NAME.to_owned())).guard();
|
||||
let output = Command::new("ip")
|
||||
.args(["netns", "exec", ns, "ip", "route", "show"])
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"failed to list IPv4 routes in {ns}: {}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
String::from_utf8_lossy(&output.stdout)
|
||||
.lines()
|
||||
.any(|line| line.contains(needle))
|
||||
}
|
||||
|
||||
fn tun_ifindex(ns: &str, dev_name: &str) -> Option<u64> {
|
||||
let output = Command::new("ip")
|
||||
.args(["-n", ns, "-j", "link", "show", "dev", dev_name])
|
||||
.output()
|
||||
.unwrap();
|
||||
if !output.status.success() {
|
||||
return None;
|
||||
}
|
||||
let links: serde_json::Value = serde_json::from_slice(&output.stdout).unwrap();
|
||||
links[0]["ifindex"].as_u64()
|
||||
}
|
||||
|
||||
#[cfg(feature = "proxy-cidr-monitor")]
|
||||
async fn patch_proxy_cidr(
|
||||
instance: &Instance,
|
||||
action: crate::proto::api::config::ConfigPatchAction,
|
||||
) {
|
||||
use crate::proto::api::config::{InstanceConfigPatch, ProxyNetworkPatch};
|
||||
|
||||
instance
|
||||
.get_config_patcher()
|
||||
.apply_patch(InstanceConfigPatch {
|
||||
proxy_networks: vec![ProxyNetworkPatch {
|
||||
action: action as i32,
|
||||
cidr: Some(PROXY_CIDR.parse().unwrap()),
|
||||
mapped_cidr: None,
|
||||
}],
|
||||
..Default::default()
|
||||
})
|
||||
.await
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
async fn shared_route_owner_count(
|
||||
registry: &ArcSharedVirtualNicRegistry,
|
||||
dev_name: &str,
|
||||
route: &SharedIpv4Route,
|
||||
) -> usize {
|
||||
let nic = {
|
||||
let registry = registry.lock().await;
|
||||
registry.get_by_dev_name_for_test(dev_name)
|
||||
};
|
||||
let Some(nic) = nic else {
|
||||
return 0;
|
||||
};
|
||||
nic.lock().await.ifcfg().owners_of_ipv4_route(route).len()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn failed_claim_update_closes_all_members_and_allows_recreation() {
|
||||
use crate::instance::{
|
||||
shared_virtual_nic::SharedVirtualNicRegistry, virtual_nic::VirtualNicConfig,
|
||||
};
|
||||
use tokio::sync::Notify;
|
||||
|
||||
let netns = TestNetnsGuard::new(short_name("svf"), "10.233.1.2/24", "fd43::2/64");
|
||||
let dev_name = test_dev_name();
|
||||
let config =
|
||||
|| VirtualNicConfig::new(dev_name.clone(), 1400, NetNS::new(Some(netns.name.clone())));
|
||||
let mut registry = SharedVirtualNicRegistry::new();
|
||||
let first_closed = Arc::new(Notify::new());
|
||||
let second_closed = Arc::new(Notify::new());
|
||||
let first_id = uuid::Uuid::new_v4();
|
||||
let first = registry.create_member(dev_name.clone(), config(), first_id, first_closed.clone());
|
||||
let second = registry.create_member(
|
||||
dev_name.clone(),
|
||||
config(),
|
||||
uuid::Uuid::new_v4(),
|
||||
second_closed.clone(),
|
||||
);
|
||||
let first_tunnel = first.create_dev().await.unwrap();
|
||||
let second_tunnel = second.create_dev().await.unwrap();
|
||||
first.link_up().await.unwrap();
|
||||
first
|
||||
.add_ip("10.233.97.1".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap();
|
||||
second
|
||||
.add_ip("10.233.98.1".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap();
|
||||
let destination = "10.233.90.0".parse().unwrap();
|
||||
first.add_route(destination, 24).await.unwrap();
|
||||
assert!(ipv4_route_exists_in_ns(&netns.name, "10.233.90.0/24"));
|
||||
|
||||
// An externally configured, better source makes the address add fail only
|
||||
// after the existing route has been removed to update its preferred source.
|
||||
let output = Command::new("ip")
|
||||
.args([
|
||||
"netns",
|
||||
"exec",
|
||||
&netns.name,
|
||||
"ip",
|
||||
"addr",
|
||||
"add",
|
||||
"10.233.96.1/24",
|
||||
"dev",
|
||||
&dev_name,
|
||||
])
|
||||
.output()
|
||||
.unwrap();
|
||||
assert!(
|
||||
output.status.success(),
|
||||
"{}",
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
let err = first
|
||||
.add_ip("10.233.96.1".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap_err();
|
||||
assert!(matches!(
|
||||
err,
|
||||
crate::common::error::Error::IOError(err)
|
||||
if err.kind() == std::io::ErrorKind::AlreadyExists
|
||||
));
|
||||
assert!(!first.shared_nic().lock().await.is_valid());
|
||||
tokio::time::timeout(WAIT, first_closed.notified())
|
||||
.await
|
||||
.unwrap();
|
||||
tokio::time::timeout(WAIT, second_closed.notified())
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(registry.get(&dev_name, &config()).is_none());
|
||||
assert!(first.add_route(destination, 24).await.is_err());
|
||||
|
||||
let replacement = registry.create_member(
|
||||
dev_name.clone(),
|
||||
config(),
|
||||
first_id,
|
||||
Arc::new(Notify::new()),
|
||||
);
|
||||
let replacement_tunnel = replacement.create_dev().await.unwrap();
|
||||
replacement.link_up().await.unwrap();
|
||||
replacement
|
||||
.add_ip("10.233.97.1".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap();
|
||||
replacement.add_route(destination, 24).await.unwrap();
|
||||
assert!(ipv4_route_exists_in_ns(
|
||||
&netns.name,
|
||||
&format!("10.233.90.0/24 dev {dev_name} proto static src 10.233.97.1")
|
||||
));
|
||||
|
||||
drop((first_tunnel, second_tunnel, replacement_tunnel));
|
||||
drop((first, second, replacement));
|
||||
wait_for_condition(
|
||||
|| async { !ipv4_route_exists_in_ns(&netns.name, "10.233.90.0/24") },
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(feature = "management-rpc")]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn shared_tun_survives_creator_executor_shutdown() {
|
||||
use crate::instance::factory::native_instance_manager_with_runtime;
|
||||
use easytier_core::instance::manager::ConfigFileControl;
|
||||
|
||||
let local_ns = TestNetnsGuard::new(short_name("sve"), "10.234.1.2/24", "fd44::2/64");
|
||||
let peer_ns = TestNetnsGuard::new(short_name("svp"), "10.234.2.2/24", "fd45::2/64");
|
||||
let dev_name = test_dev_name();
|
||||
let peer_dev = test_dev_name();
|
||||
let manager = native_instance_manager_with_runtime(tokio::runtime::Handle::current());
|
||||
let mut ids = Vec::new();
|
||||
for (name, network, ns, dev, ip, multi_thread) in [
|
||||
(
|
||||
"creator",
|
||||
"first",
|
||||
&local_ns.name,
|
||||
&dev_name,
|
||||
"10.244.1.1/24",
|
||||
false,
|
||||
),
|
||||
(
|
||||
"survivor",
|
||||
"second",
|
||||
&local_ns.name,
|
||||
&dev_name,
|
||||
"10.244.2.1/24",
|
||||
true,
|
||||
),
|
||||
(
|
||||
"peer",
|
||||
"second",
|
||||
&peer_ns.name,
|
||||
&peer_dev,
|
||||
"10.244.2.2/24",
|
||||
false,
|
||||
),
|
||||
] {
|
||||
let config = test_config(name, network, "secret", Some(ns), Some(dev), ip);
|
||||
let mut flags = config.get_flags();
|
||||
flags.multi_thread = multi_thread;
|
||||
flags.multi_thread_count = 2;
|
||||
flags.disable_upnp = true;
|
||||
flags.disable_p2p = true;
|
||||
flags.disable_tcp_hole_punching = true;
|
||||
flags.disable_udp_hole_punching = true;
|
||||
config.set_flags(flags);
|
||||
config.set_stun_servers(Some(Vec::new()));
|
||||
config.set_stun_servers_v6(Some(Vec::new()));
|
||||
config.set_tcp_stun_servers(Some(Vec::new()));
|
||||
let id = manager
|
||||
.run_network_instance(config, ConfigFileControl::STATIC_CONFIG)
|
||||
.unwrap();
|
||||
ids.push(id);
|
||||
// Make the first instance the actual creator, not merely the first to start.
|
||||
wait_for_condition(|| async { tun_ifindex(ns, dev).is_some() }, WAIT).await;
|
||||
wait_for_condition(|| async { manager.instance(id).unwrap().is_ready() }, WAIT).await;
|
||||
}
|
||||
let creator_runtime = manager.data_plane_runtime_handle(&ids[0]).unwrap();
|
||||
let creator_task = creator_runtime.spawn(std::future::pending::<()>());
|
||||
let survivor = manager.instance(ids[1]).unwrap();
|
||||
let peer = manager.instance(ids[2]).unwrap();
|
||||
peer.add_connector(format!("ring://{}", ids[1]).parse().unwrap())
|
||||
.unwrap();
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
survivor.connected_peers().await.contains(&peer.peer_id())
|
||||
&& peer.connected_peers().await.contains(&survivor.peer_id())
|
||||
&& ipv4_route_exists_in_ns(&local_ns.name, "10.244.2.0/24")
|
||||
&& ipv4_route_exists_in_ns(&peer_ns.name, "10.244.2.0/24")
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
|
||||
let bind = |ns: &str, address: &str| {
|
||||
let socket =
|
||||
NetNS::new(Some(ns.to_owned())).run(|| std::net::UdpSocket::bind(address).unwrap());
|
||||
socket.set_nonblocking(true).unwrap();
|
||||
tokio::net::UdpSocket::from_std(socket).unwrap()
|
||||
};
|
||||
let client = bind(&local_ns.name, "10.244.2.1:0");
|
||||
let server = bind(&peer_ns.name, "10.244.2.2:0");
|
||||
client.connect(server.local_addr().unwrap()).await.unwrap();
|
||||
let original_index = tun_ifindex(&local_ns.name, &dev_name).unwrap();
|
||||
let mut creator_task = Some(creator_task);
|
||||
for (pass, message) in [b"before shutdown".as_slice(), b"after shutdown".as_slice()]
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
{
|
||||
if pass == 1 {
|
||||
manager
|
||||
.delete_network_instances(vec![ids[0]])
|
||||
.await
|
||||
.unwrap();
|
||||
assert!(
|
||||
tokio::time::timeout(WAIT, creator_task.take().unwrap())
|
||||
.await
|
||||
.unwrap()
|
||||
.unwrap_err()
|
||||
.is_cancelled()
|
||||
);
|
||||
assert_eq!(tun_ifindex(&local_ns.name, &dev_name), Some(original_index));
|
||||
}
|
||||
tokio::time::timeout(WAIT, async {
|
||||
client.send(message).await.unwrap();
|
||||
let mut buf = [0; 64];
|
||||
let (len, from) = server.recv_from(&mut buf).await.unwrap();
|
||||
assert_eq!(&buf[..len], message);
|
||||
assert_eq!(from, client.local_addr().unwrap());
|
||||
server.send_to(&buf[..len], from).await.unwrap();
|
||||
let len = client.recv(&mut buf).await.unwrap();
|
||||
assert_eq!(&buf[..len], message);
|
||||
})
|
||||
.await
|
||||
.expect("the existing UDP socket must work across creator shutdown");
|
||||
}
|
||||
assert_eq!(tun_ifindex(&local_ns.name, &dev_name), Some(original_index));
|
||||
drop((client, server, survivor, peer));
|
||||
manager
|
||||
.delete_network_instances(ids[1..].to_vec())
|
||||
.await
|
||||
.unwrap();
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
tun_ifindex(&local_ns.name, &dev_name).is_none()
|
||||
&& tun_ifindex(&peer_ns.name, &peer_dev).is_none()
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial]
|
||||
fn shared_tun_claim_cleanup_outlives_member_runtime() {
|
||||
use crate::instance::{
|
||||
shared_virtual_nic::SharedVirtualNicRegistry, virtual_nic::VirtualNicConfig,
|
||||
};
|
||||
use tokio::{runtime::Builder, sync::Notify};
|
||||
|
||||
let netns = TestNetnsGuard::new(short_name("svc"), "10.235.1.2/24", "fd46::2/64");
|
||||
let dev_name = test_dev_name();
|
||||
let config =
|
||||
|| VirtualNicConfig::new(dev_name.clone(), 1400, NetNS::new(Some(netns.name.clone())));
|
||||
let mut registry = SharedVirtualNicRegistry::new();
|
||||
let first = registry.create_member(
|
||||
dev_name.clone(),
|
||||
config(),
|
||||
uuid::Uuid::new_v4(),
|
||||
Arc::new(Notify::new()),
|
||||
);
|
||||
let second = registry.create_member(
|
||||
dev_name.clone(),
|
||||
config(),
|
||||
uuid::Uuid::new_v4(),
|
||||
Arc::new(Notify::new()),
|
||||
);
|
||||
let member_runtime = Builder::new_current_thread().enable_all().build().unwrap();
|
||||
let observer_runtime = Builder::new_current_thread().enable_all().build().unwrap();
|
||||
let first_tunnel = member_runtime.block_on(async {
|
||||
let tunnel = first.create_dev().await.unwrap();
|
||||
first.link_up().await.unwrap();
|
||||
first
|
||||
.add_ip("10.245.1.1".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap();
|
||||
first
|
||||
.add_route("10.245.90.0".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap();
|
||||
tunnel
|
||||
});
|
||||
let second_tunnel = observer_runtime.block_on(async {
|
||||
let tunnel = second.create_dev().await.unwrap();
|
||||
second
|
||||
.add_ip("10.245.2.1".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap();
|
||||
tunnel
|
||||
});
|
||||
assert!(ipv4_route_exists_in_ns(&netns.name, "10.245.90.0/24"));
|
||||
let original_index = tun_ifindex(&netns.name, &dev_name).unwrap();
|
||||
let shared_nic = first.shared_nic();
|
||||
let guard = observer_runtime.block_on(shared_nic.lock());
|
||||
member_runtime.block_on(async {
|
||||
drop((first_tunnel, first));
|
||||
// Claim removal cannot acquire the lock until after this runtime exits.
|
||||
tokio::task::yield_now().await;
|
||||
});
|
||||
drop(member_runtime);
|
||||
drop(guard);
|
||||
|
||||
observer_runtime.block_on(async {
|
||||
wait_for_condition(
|
||||
|| async { !ipv4_route_exists_in_ns(&netns.name, "10.245.90.0/24") },
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
assert_eq!(tun_ifindex(&netns.name, &dev_name), Some(original_index));
|
||||
// Reusing the departed member's address also proves its claims were released.
|
||||
second
|
||||
.add_ip("10.245.1.1".parse().unwrap(), 24)
|
||||
.await
|
||||
.unwrap();
|
||||
});
|
||||
// The final registration can be dropped without any entered Tokio runtime.
|
||||
drop((second_tunnel, second));
|
||||
observer_runtime.block_on(wait_for_condition(
|
||||
|| async { tun_ifindex(&netns.name, &dev_name).is_none() },
|
||||
WAIT,
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn same_namespace_members_share_tun_across_independent_networks() {
|
||||
let dev_name = test_dev_name();
|
||||
let first_peer_ns = TestNetnsGuard::new(short_name("sva"), "10.231.1.2/24", "fd31::2/64");
|
||||
let second_peer_ns = TestNetnsGuard::new(short_name("svb"), "10.231.2.2/24", "fd32::2/64");
|
||||
let runtime = SharedTestRuntime::new();
|
||||
|
||||
let mut first = runtime.instance(test_config(
|
||||
"shared_tun_first",
|
||||
"shared_tun_network_a",
|
||||
"shared_tun_secret_a",
|
||||
None,
|
||||
Some(&dev_name),
|
||||
"10.144.250.1/24",
|
||||
));
|
||||
let mut second = runtime.instance(test_config(
|
||||
"shared_tun_second",
|
||||
"shared_tun_network_b",
|
||||
"shared_tun_secret_b",
|
||||
None,
|
||||
Some(&dev_name),
|
||||
"10.144.251.1/24",
|
||||
));
|
||||
let mut first_peer = runtime.instance(test_config(
|
||||
"shared_tun_first_peer",
|
||||
"shared_tun_network_a",
|
||||
"shared_tun_secret_a",
|
||||
Some(&first_peer_ns.name),
|
||||
None,
|
||||
"10.144.250.2/24",
|
||||
));
|
||||
let mut second_peer = runtime.instance(test_config(
|
||||
"shared_tun_second_peer",
|
||||
"shared_tun_network_b",
|
||||
"shared_tun_secret_b",
|
||||
Some(&second_peer_ns.name),
|
||||
None,
|
||||
"10.144.251.2/24",
|
||||
));
|
||||
|
||||
first.run().await.unwrap();
|
||||
second.run().await.unwrap();
|
||||
first_peer.run().await.unwrap();
|
||||
second_peer.run().await.unwrap();
|
||||
|
||||
wait_tun_ready(&first, &dev_name).await;
|
||||
wait_tun_ready(&second, &dev_name).await;
|
||||
assert_eq!(
|
||||
first.get_global_ctx().get_tun_device_name(),
|
||||
second.get_global_ctx().get_tun_device_name()
|
||||
);
|
||||
|
||||
first_peer.add_connector_url(first.ring_listener_url());
|
||||
second_peer.add_connector_url(second.ring_listener_url());
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
first
|
||||
.get_core_instance()
|
||||
.route_snapshots()
|
||||
.await
|
||||
.iter()
|
||||
.any(|route| route.peer_id == first_peer.peer_id())
|
||||
&& second
|
||||
.get_core_instance()
|
||||
.route_snapshots()
|
||||
.await
|
||||
.iter()
|
||||
.any(|route| route.peer_id == second_peer.peer_id())
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
wait_for_condition(
|
||||
|| async { ping_test(&first_peer_ns.name, "10.144.250.1", None).await },
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
wait_for_condition(
|
||||
|| async { ping_test(&second_peer_ns.name, "10.144.251.1", None).await },
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
|
||||
drop_insts(vec![first, second, first_peer, second_peer]).await;
|
||||
}
|
||||
|
||||
#[cfg(feature = "proxy-cidr-monitor")]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn runtime_proxy_patch_adds_and_removes_os_route() {
|
||||
use crate::proto::api::config::ConfigPatchAction;
|
||||
|
||||
let lab = ProxyLab::new();
|
||||
let source_dev = test_dev_name();
|
||||
let destination_dev = test_dev_name();
|
||||
let runtime = SharedTestRuntime::new();
|
||||
let mut source = runtime.instance(test_config(
|
||||
"shared_patch_source",
|
||||
"shared_patch_network",
|
||||
"shared_patch_secret",
|
||||
Some(lab.network.namespace("source")),
|
||||
Some(&source_dev),
|
||||
"10.144.244.1/24",
|
||||
));
|
||||
let mut destination = runtime.instance(test_config(
|
||||
"shared_patch_destination",
|
||||
"shared_patch_network",
|
||||
"shared_patch_secret",
|
||||
Some(lab.network.namespace("owner")),
|
||||
Some(&destination_dev),
|
||||
"10.144.244.2/24",
|
||||
));
|
||||
|
||||
source.run().await.unwrap();
|
||||
destination.run().await.unwrap();
|
||||
wait_tun_ready(&source, &source_dev).await;
|
||||
wait_tun_ready(&destination, &destination_dev).await;
|
||||
destination.add_connector_url(source.ring_listener_url());
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
source
|
||||
.get_core_instance()
|
||||
.route_snapshots()
|
||||
.await
|
||||
.iter()
|
||||
.any(|route| route.peer_id == destination.peer_id())
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
assert!(!ipv4_route_exists_in_ns(
|
||||
lab.network.namespace("source"),
|
||||
&format!("{PROXY_CIDR} dev {source_dev}")
|
||||
));
|
||||
|
||||
patch_proxy_cidr(&destination, ConfigPatchAction::Add).await;
|
||||
wait_proxy_route(&source, destination.peer_id(), PROXY_CIDR).await;
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
ipv4_route_exists_in_ns(
|
||||
lab.network.namespace("source"),
|
||||
&format!("{PROXY_CIDR} dev {source_dev}"),
|
||||
)
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
|
||||
patch_proxy_cidr(&destination, ConfigPatchAction::Remove).await;
|
||||
wait_proxy_route_absent(&source, destination.peer_id(), PROXY_CIDR).await;
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
!ipv4_route_exists_in_ns(
|
||||
lab.network.namespace("source"),
|
||||
&format!("{PROXY_CIDR} dev {source_dev}"),
|
||||
)
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
|
||||
drop_insts(vec![source, destination]).await;
|
||||
}
|
||||
|
||||
#[cfg(feature = "magic-dns")]
|
||||
async fn wait_shared_magic_dns_record(
|
||||
ns: &str,
|
||||
domain: &str,
|
||||
expected_source: &str,
|
||||
expected_ip: &str,
|
||||
) {
|
||||
use hickory_proto::{
|
||||
op::{Message, MessageType, Query, ResponseCode},
|
||||
rr::{RData, RecordType},
|
||||
};
|
||||
|
||||
use crate::instance::dns_server::MAGIC_DNS_FAKE_IP;
|
||||
|
||||
let expected_source = expected_source.parse::<Ipv4Addr>().unwrap();
|
||||
let expected_ip = expected_ip.parse::<Ipv4Addr>().unwrap();
|
||||
let mut query = Message::new();
|
||||
query
|
||||
.set_id(rand::random())
|
||||
.add_query(Query::query(domain.parse().unwrap(), RecordType::A));
|
||||
let payload = query.to_vec().unwrap();
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
// Let the kernel select the source from the shared route, rather
|
||||
// than binding to the server and hiding a client-owned route.
|
||||
let Ok(socket) = NetNS::new(Some(ns.to_owned())).run(|| {
|
||||
let socket = std::net::UdpSocket::bind("0.0.0.0:0")?;
|
||||
socket.connect((MAGIC_DNS_FAKE_IP, 53))?;
|
||||
socket.set_nonblocking(true)?;
|
||||
tokio::net::UdpSocket::from_std(socket)
|
||||
}) else {
|
||||
return false;
|
||||
};
|
||||
let source = socket.local_addr().unwrap().ip();
|
||||
if socket.send(&payload).await.is_err() {
|
||||
return false;
|
||||
}
|
||||
let mut buffer = [0; 512];
|
||||
let Ok(Ok(len)) =
|
||||
tokio::time::timeout(Duration::from_millis(500), socket.recv(&mut buffer)).await
|
||||
else {
|
||||
return false;
|
||||
};
|
||||
let response = Message::from_vec(&buffer[..len]).unwrap();
|
||||
source == std::net::IpAddr::V4(expected_source)
|
||||
&& response.id() == query.id()
|
||||
&& response.message_type() == MessageType::Response
|
||||
&& response.response_code() == ResponseCode::NoError
|
||||
&& response.answers().len() == 1
|
||||
&& matches!(response.answers()[0].data(), RData::A(ip) if ip.0 == expected_ip)
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[cfg(feature = "magic-dns")]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn magic_dns_route_follows_serving_member_and_takeover() {
|
||||
use crate::instance::dns_server::MAGIC_DNS_FAKE_IP;
|
||||
|
||||
let netns = TestNetnsGuard::new(short_name("svd"), "10.232.1.2/24", "fd42::2/64");
|
||||
let dev_name = test_dev_name();
|
||||
let runtime = SharedTestRuntime::new();
|
||||
let first_config = test_config(
|
||||
"shared_dns_first",
|
||||
"shared_dns_network",
|
||||
"shared_dns_secret",
|
||||
Some(&netns.name),
|
||||
Some(&dev_name),
|
||||
"10.144.243.1/24",
|
||||
);
|
||||
// The first runner wins the singleton, but the client sorts first when
|
||||
// shared route source hints are chosen by UUID.
|
||||
first_config.set_id(uuid::Uuid::from_u128(2));
|
||||
first_config.set_hostname(Some("shared-dns-first".to_owned()));
|
||||
let mut flags = first_config.get_flags();
|
||||
flags.accept_dns = true;
|
||||
first_config.set_flags(flags.clone());
|
||||
let second_config = test_config(
|
||||
"shared_dns_second",
|
||||
"shared_dns_network",
|
||||
"shared_dns_secret",
|
||||
Some(&netns.name),
|
||||
Some(&dev_name),
|
||||
"10.144.242.2/24",
|
||||
);
|
||||
second_config.set_id(uuid::Uuid::from_u128(1));
|
||||
second_config.set_hostname(Some("shared-dns-second".to_owned()));
|
||||
second_config.set_flags(flags);
|
||||
let mut first = runtime.instance(first_config);
|
||||
let mut second = runtime.instance(second_config);
|
||||
|
||||
first.run().await.unwrap();
|
||||
wait_tun_ready(&first, &dev_name).await;
|
||||
wait_shared_magic_dns_record(
|
||||
&netns.name,
|
||||
"shared-dns-first.et.net.",
|
||||
"10.144.243.1",
|
||||
"10.144.243.1",
|
||||
)
|
||||
.await;
|
||||
|
||||
second.run().await.unwrap();
|
||||
wait_tun_ready(&second, &dev_name).await;
|
||||
// Resolving the second member also waits for its client to connect and
|
||||
// publish, ensuring its failed server election has already completed.
|
||||
wait_shared_magic_dns_record(
|
||||
&netns.name,
|
||||
"shared-dns-second.et.net.",
|
||||
"10.144.243.1",
|
||||
"10.144.242.2",
|
||||
)
|
||||
.await;
|
||||
let route = SharedIpv4Route::new(MAGIC_DNS_FAKE_IP.parse::<Ipv4Addr>().unwrap(), 32, None);
|
||||
assert_eq!(
|
||||
shared_route_owner_count(&runtime.registry, &dev_name, &route).await,
|
||||
1
|
||||
);
|
||||
|
||||
drop_insts(vec![first]).await;
|
||||
wait_shared_magic_dns_record(
|
||||
&netns.name,
|
||||
"shared-dns-second.et.net.",
|
||||
"10.144.242.2",
|
||||
"10.144.242.2",
|
||||
)
|
||||
.await;
|
||||
assert_eq!(
|
||||
shared_route_owner_count(&runtime.registry, &dev_name, &route).await,
|
||||
1
|
||||
);
|
||||
|
||||
// Keep the TUN alive after the final DNS runner stops so device teardown
|
||||
// cannot hide a leaked fake-IP route.
|
||||
let mut observer = runtime.instance(test_config(
|
||||
"shared_dns_observer",
|
||||
"shared_dns_network",
|
||||
"shared_dns_secret",
|
||||
Some(&netns.name),
|
||||
Some(&dev_name),
|
||||
"10.144.244.3/24",
|
||||
));
|
||||
observer.run().await.unwrap();
|
||||
wait_tun_ready(&observer, &dev_name).await;
|
||||
|
||||
drop_insts(vec![second]).await;
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
shared_route_owner_count(&runtime.registry, &dev_name, &route).await == 0
|
||||
&& !ipv4_route_exists_in_ns(
|
||||
&netns.name,
|
||||
&format!("{MAGIC_DNS_FAKE_IP} dev {dev_name}"),
|
||||
)
|
||||
},
|
||||
WAIT,
|
||||
)
|
||||
.await;
|
||||
drop_insts(vec![observer]).await;
|
||||
}
|
||||
@@ -1016,6 +1016,166 @@ pub async fn public_ipv6_auto_addr_reconnect_reuses_same_address() {
|
||||
drop_insts(vec![provider, client]).await;
|
||||
}
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
pub async fn shared_tun_public_ipv6_auto_addr_end_to_end() {
|
||||
let lab = PublicIpv6Lab::setup_with_topology(PublicIpv6LabTopology::DelegatedPrefix);
|
||||
let provider_dev = format!("st{:08x}", rand::random::<u32>());
|
||||
let client_dev = format!("st{:08x}", rand::random::<u32>());
|
||||
let process_runtime = CoreProcessRuntime::new();
|
||||
let shared_virtual_nic_registry = Instance::new_shared_virtual_nic_registry();
|
||||
|
||||
let provider_cfg = get_public_ipv6_config(
|
||||
"provider_shared_public_ipv6",
|
||||
lab.network.namespace(PublicIpv6Lab::PROVIDER_NS),
|
||||
"10.144.144.1",
|
||||
&provider_dev,
|
||||
uuid::Uuid::parse_str("44444444-4444-4444-4444-444444444444").unwrap(),
|
||||
);
|
||||
provider_cfg.set_ipv6_public_addr_provider(true);
|
||||
|
||||
let client_cfg = get_public_ipv6_config(
|
||||
"client_shared_public_ipv6",
|
||||
lab.network.namespace(PublicIpv6Lab::CLIENT_NS),
|
||||
"10.144.144.2",
|
||||
&client_dev,
|
||||
uuid::Uuid::parse_str("55555555-5555-5555-5555-555555555555").unwrap(),
|
||||
);
|
||||
client_cfg.set_ipv6_public_addr_auto(true);
|
||||
|
||||
let client_peer_cfg = get_public_ipv6_config(
|
||||
"client_shared_public_ipv6_peer",
|
||||
lab.network.namespace(PublicIpv6Lab::CLIENT_NS),
|
||||
"10.144.145.3",
|
||||
&client_dev,
|
||||
uuid::Uuid::parse_str("66666666-6666-6666-6666-666666666666").unwrap(),
|
||||
);
|
||||
client_peer_cfg.set_listeners(vec![]);
|
||||
|
||||
let mut provider = Instance::new_with_process_runtime_and_shared_virtual_nic_registry(
|
||||
provider_cfg,
|
||||
process_runtime.clone(),
|
||||
shared_virtual_nic_registry.clone(),
|
||||
);
|
||||
let mut client = Instance::new_with_process_runtime_and_shared_virtual_nic_registry(
|
||||
client_cfg,
|
||||
process_runtime.clone(),
|
||||
shared_virtual_nic_registry.clone(),
|
||||
);
|
||||
let mut client_peer = Instance::new_with_process_runtime_and_shared_virtual_nic_registry(
|
||||
client_peer_cfg,
|
||||
process_runtime,
|
||||
shared_virtual_nic_registry,
|
||||
);
|
||||
let mut client_events = client.get_global_ctx().subscribe();
|
||||
let mut client_peer_events = client_peer.get_global_ctx().subscribe();
|
||||
|
||||
provider.run().await.unwrap();
|
||||
client.run().await.unwrap();
|
||||
client_peer.run().await.unwrap();
|
||||
|
||||
let shared_ifname = wait_for_tun_ready(&mut client_events).await;
|
||||
assert_eq!(
|
||||
shared_ifname,
|
||||
wait_for_tun_ready(&mut client_peer_events).await
|
||||
);
|
||||
assert_eq!(shared_ifname, client_dev);
|
||||
|
||||
provider.add_connector_url("tcp://10.1.1.2:11010".parse().unwrap());
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
provider.get_core_instance().route_snapshots().await.len() == 1
|
||||
&& client.get_core_instance().route_snapshots().await.len() == 1
|
||||
},
|
||||
Duration::from_secs(8),
|
||||
)
|
||||
.await;
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
provider
|
||||
.get_core_instance()
|
||||
.node_snapshot()
|
||||
.await
|
||||
.ipv6_public_addr_prefix
|
||||
== Some(PublicIpv6Lab::PROVIDER_PREFIX.parse().unwrap())
|
||||
},
|
||||
Duration::from_secs(10),
|
||||
)
|
||||
.await;
|
||||
|
||||
let leased = wait_for_public_ipv6_addr(&client).await;
|
||||
wait_for_public_ipv6_route(&provider, leased).await;
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
addr_exists_in_ns(
|
||||
lab.network.namespace(PublicIpv6Lab::CLIENT_NS),
|
||||
&client_dev,
|
||||
&leased.to_string(),
|
||||
) && route_exists_in_ns(
|
||||
lab.network.namespace(PublicIpv6Lab::CLIENT_NS),
|
||||
&format!("default dev {client_dev}"),
|
||||
) && route_exists_in_ns(
|
||||
lab.network.namespace(PublicIpv6Lab::PROVIDER_NS),
|
||||
&format!("{} dev {provider_dev}", leased.address()),
|
||||
)
|
||||
},
|
||||
Duration::from_secs(10),
|
||||
)
|
||||
.await;
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
ping6_test(
|
||||
lab.network.namespace(PublicIpv6Lab::CLIENT_NS),
|
||||
PublicIpv6Lab::SERVER_IP,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
},
|
||||
Duration::from_secs(10),
|
||||
)
|
||||
.await;
|
||||
|
||||
wait_for_condition(
|
||||
|| async {
|
||||
ping6_test(
|
||||
lab.network.namespace(PublicIpv6Lab::SERVER_NS),
|
||||
leased.address().to_string().as_str(),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
},
|
||||
Duration::from_secs(10),
|
||||
)
|
||||
.await;
|
||||
|
||||
drop_insts(vec![provider, client, client_peer]).await;
|
||||
drop(lab);
|
||||
}
|
||||
|
||||
#[cfg(feature = "tun")]
|
||||
async fn wait_for_tun_ready(
|
||||
receiver: &mut tokio::sync::broadcast::Receiver<crate::common::global_ctx::GlobalCtxEvent>,
|
||||
) -> String {
|
||||
tokio::time::timeout(Duration::from_secs(5), async {
|
||||
loop {
|
||||
match receiver.recv().await.unwrap() {
|
||||
crate::common::global_ctx::GlobalCtxEvent::TunDeviceReady(ifname) => return ifname,
|
||||
crate::common::global_ctx::GlobalCtxEvent::TunDeviceError(error) => {
|
||||
panic!("tun device error: {error}")
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("timed out waiting for tun ready")
|
||||
}
|
||||
|
||||
#[rstest::rstest]
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
|
||||
Reference in new issue
Block a user