EasyTier/easytier-go/PERFORMANCE.md

Performance log

This file records reproducible easytier-go data-plane optimizations. Each round must describe the change, the benchmark conditions, the before and after result, and any profile evidence used to choose the next change.

Benchmark setup

• Host CPU: Intel Core i7-14700KF.
• Go host CPUs: P-core logical CPUs 8,10,12,14.
• Native peer and iperf CPUs: P-core logical CPUs 0,2,4,6.
• Underlay: UDP over the local Docker bridge.
• Native peer: 10.15.15.1, listening on underlay UDP port 12012.
• Go host: 10.15.15.2, forwarding local port 5202 through Instance.Dial to 10.15.15.1:5201.
• Go forwarder: go run ./cmd/dial-forward-bench; it intentionally keeps forwarding outside the Core so measurements include the public data-plane ABI.
• MTU: 1380; encryption enabled with the default algorithm.
• TCP tests use one stream unless noted otherwise.
• UDP tests use 1200-byte datagrams.
• Reported results exclude the first two seconds of each iperf run.

The native comparison uses EasyTier's built-in port forward on the same machine, CPU groups, peer, network, and iperf server. It isolates the overhead of the Go/WASM data-plane interface from EasyTier's virtual TCP/UDP stack and transport.

Round 0: Dial baseline

Go host commit: a73f2bac

EasyTier commit: 30764897

Throughput

Path	Direction	Offered load	Result
Go DialTCP	Go to native	unlimited	14.0 Mbit/s
Go DialTCP	native to Go	unlimited	1.56 Gbit/s
Go DialTCP, 4 streams	Go to native	unlimited	56.6 Mbit/s
Native TCP port forward	forward	unlimited	1.35 Gbit/s
Native TCP port forward	reverse	unlimited	3.65 Gbit/s
Go DialUDP	Go to native	10 Mbit/s	10.0 Mbit/s, 0% loss
Go DialUDP	Go to native	100 Mbit/s	tail stalled after 4 seconds
Go DialUDP	native to Go	100 Mbit/s	61.6 Mbit/s, 38% loss
Go DialUDP	native to Go	1 Gbit/s	86.5 Mbit/s, 91% loss
Native UDP port forward	forward	1 Gbit/s	999 Mbit/s

At a 100 Mbit/s forward UDP load, the receiver averaged 40.9 Mbit/s over the full test because traffic stopped after four measured seconds. This is not a stable throughput result.

TCP forward profile

perf stat attached to the Go host during a 10-second single-stream test:

Counter	Value
elapsed time	12.014 s
task clock	1.186 s
context switches	89,072
CPU migrations	12,015
core cycles	5.98 billion
core instructions	4.09 billion
core cache misses	23.0 million

The Go process consumed only 0.099 CPU equivalents while forwarding at 14 Mbit/s. Four TCP streams scaled almost exactly linearly, showing a per-stream serialization limit rather than CPU or transport saturation.

The Go streamConn.Write waits for every guest TCP-write operation. The guest operation currently calls AsyncWriteExt::write, which may complete after a partial write. Each partial result therefore incurs another complete submit/drive/complete/take host-to-WASM cycle. The native port-forward result shows that EasyTier's underlying data plane is not the limiting component.

Round 1: finish each guest TCP write

Go host commit: a73f2bac

EasyTier commit: 73612470

The guest TCP operation now uses write_all semantics instead of completing after the first partial virtual-socket write.

Path	Before	After	Change
TCP forward	14.0 Mbit/s	15.9 Mbit/s	+13.6%

This removed redundant operation completion cycles, but CPU utilization remained low. The result showed that partial completion was real overhead but was not the main serialization point.

Round 2: repoll immediately when smoltcp has ready egress

EasyTier commit: 824e01a3

The smoltcp reactor reported a zero poll delay when more egress was ready, but the portable runtime crossed the host timer boundary before polling again. The reactor now polls again in the same drive turn for a zero delay.

Path	Before	After	Change
TCP forward	15.9 Mbit/s	1.17 Gbit/s	73.6x
Median packet gap	891 us	8.41 us	-99.1%

The packet-gap distribution was captured in /tmp/easytier-dial-forward-immediate-repoll.pcap. Extending every WASI drive turn by a fixed 50 us did not improve throughput and was later reverted; the zero-delay signal was the correct condition for immediate work.

Round 3: preserve pending UDP operations

Go host commit: cf53f92

The example refreshed SetDeadline before every datagram. Each refresh cancelled the currently pending read operation, causing forwarding to stall under sustained load. Deadline refreshes are now limited to half of the idle timeout, retaining idle-session cleanup without repeatedly cancelling active operations.

UDP forward offered load	Before	After
300 Mbit/s	45.3 Mbit/s average, tail stalled	298 Mbit/s, 0.8% loss

Logging added while diagnosing this behavior showed successful writes until the receive operation was cancelled; the stall was not a transport write failure.

Round 4: reduce per-datagram host/guest overhead

The following measurements use a 1 Gbit/s offered UDP load. Each row is measured against the immediately preceding retained state.

Change	Commit	Before	After	Change
Reuse operation deadline contexts	3d7b622	525 Mbit/s	558 Mbit/s	+6.3%
Reuse separate UDP address and payload inputs	f9092c9	558 Mbit/s	587 Mbit/s	+5.2%
Reuse data-plane output storage	f469a34	587 Mbit/s	652 Mbit/s	+11.1%
Restore a zero-duration WASI drive budget	a5ea9b6	652 Mbit/s	672 Mbit/s	+3.1%
Cache hot guest function handles	47f0809	672 Mbit/s	677 Mbit/s	+0.7%
Reuse the instance deadline timer	657022b	677 Mbit/s	677 Mbit/s	neutral
Coalesce immediately completed outcomes	9d1ec76	677 Mbit/s	688 Mbit/s	+1.6%

The reusable input experiment initially used an interior pointer into one larger guest allocation. The ABI only accepts allocation base pointers, so the final implementation keeps the remote address and payload in distinct reusable allocations.

Restoring the zero-duration drive budget also raised TCP forward throughput from 1.17 to approximately 1.30 Gbit/s. Reusing the deadline timer was kept despite neutral throughput because it removes a timer allocation from every operation without adding synchronization.

The final rebuilt artifact and host measured 693 Mbit/s UDP forward, within the variation of the 688 Mbit/s retained baseline.

Profile evidence

The final-stage profile is stored at /tmp/easytier-dial-udp-zero-budget.perf.data. Approximately 49% of samples were in the WASM JIT and 31% in the Go executable, with the remainder primarily in the kernel. Before function caching, NextDeadline accounted for 2.25% and Go map hashing for 1.48%. After removing these lookup costs, no remaining individual host-side helper accounted for enough time to explain the gap to TCP; guest drive execution and one-operation-per-datagram boundary crossings dominate.

Round 5: enlarge data-plane UDP receive queues

EasyTier commit: 1df439d

The data-plane TCP buffers were already 128 KiB, while UDP inherited the smoltcp default receive queue of about 8 KiB, enough for only a few 1200-byte datagrams. The data-plane-only UDP receive buffer is now 128 KiB with 128 packet metadata entries.

UDP reverse offered load	Before	After
100 Mbit/s	61.6 Mbit/s, 38% loss	100 Mbit/s, 0% loss
300 Mbit/s	not stable	300 Mbit/s, 0.15% loss
500 Mbit/s	not stable	495 Mbit/s, 0.95% loss
800 Mbit/s	558 Mbit/s saturated	about 570 Mbit/s saturated

Removing the redundant receive-result-size ABI query in 104223c raised the 800 Mbit/s saturated result from 558 to 563 Mbit/s. Subsequent retained changes brought it to about 570 Mbit/s.

Rejected experiments

Rejected changes were reverted and are not part of the final data path.

Experiment	Before	After	Reason rejected
Fixed 50 us WASI drive budget	1.17 Gbit/s TCP	no gain	Added unconditional work instead of following readiness
Pool queued UDP payloads	652 Mbit/s	651 Mbit/s	sync.Pool overhead offset allocation savings
Lock guest driver to one OS thread	652 Mbit/s	436 Mbit/s	Prevented the Go scheduler from placing other work effectively
Return tickets before the first drive	677 Mbit/s	564 Mbit/s	Added an extra wakeup to the normal completion path
Pool one-shot response channels	688 Mbit/s	654 Mbit/s	Pool bookkeeping cost more than direct allocation

Pinning the complete process to one P-core produced 680 Mbit/s on the earlier 652 Mbit/s build, showing that CPU migration and core selection affect the result. Thread pinning inside the library was nevertheless rejected because it regressed throughput and imposed scheduling policy on applications.

Round 6: preserve TCP write progress across cancellation

EasyTier commits: 6b55489, 4d270f3

write_all kept the TCP fast path inside the guest, but it could lose the already-written prefix when a deadline update cancelled the future. The host would then replay the complete chunk. The replacement still completes writes inside the guest, but uses cancellation-safe underlying writes and reports a completed prefix before observing cancellation or deadline expiry.

WebClient retry, feature timeout, heartbeat, and UDP port-mapping renewal timers now use the tracked portable time abstraction. This lets an externally driven host advertise and advance their deadlines without adding a periodic timer-driving fallback.

CPU	Direction	Before	After	Change
i7-14700KF	TCP forward	1.30 Gbit/s	1.31 Gbit/s	within variation
i7-14700KF	TCP reverse	1.60 Gbit/s	1.61 Gbit/s	within variation
Intel N100	TCP forward	442 Mbit/s	436 Mbit/s	-1.4%
Intel N100	TCP reverse	524 Mbit/s	517 Mbit/s	-1.3%

The N100 results varied by more than these deltas between one-second intervals, so the change has no measurable systematic throughput cost.

Round 7: store deadlines on guest resources

EasyTier commits: 05ba813, 66252ed

Go host commits: 200f32c, 59d6e5a

The ABI previously passed a remaining timeout with every TCP and UDP data operation. Once a Go caller configured a connection deadline, the WASI guest therefore registered a new timer for every read and write. Deadline changes also required the Go host to cancel and resubmit an active operation.

Data-plane ABI v3 stores independent read and write deadlines on each TCP or UDP resource. Hosts update them through one resource setter, while data operations reuse the active expiration signal without creating timers. Connect, bind, and accept retain their per-operation timeouts.

The dedicated A/B benchmark compared the pre-change fd1be5c Go host with the updated host. The helper connected directly to the native peer through udp://172.17.0.3:12012, set a one-hour deadline once when requested, and excluded the first two seconds. It used the CPU groups described above.

TCP forward	Before	After	Change
No deadline	1.30 Gbit/s	1.30 Gbit/s	neutral
One-hour deadline	1.29 Gbit/s	1.32 Gbit/s	within variation

An earlier run reported approximately 750 Mbit/s, but its helper used a loopback peer URI across different network namespaces while the machine was also under compiler load. Those TCP and UDP figures were discarded because the traffic path was not controlled. The direct A/B result shows that resource deadlines retain the established TCP throughput and remove the per-operation timer construction without a measurable deadline penalty.

Round 8: batch host TCP underlay writes

EasyTier commit: cf26d50

Go host commits: 8d010e0, 4197f0d

The TUN example connected directly to the long-running native peer through tcp://172.17.0.2:11010. The overlay addresses were 10.12.12.2 and 10.12.12.1. Tests used one iperf TCP stream for 15 seconds and excluded the first two seconds. The application and native peer remained unpinned to match the reported command; the iperf client used CPUs 0,2,4,6.

The TCP host imports still used wazero's reflection-based WithFunc path. Changing the four stream read and write imports to typed module functions removed reflection and signature decoding from every host operation.

EasyTier's FramedWriter already queues up to 64 frames and exposes them as vectored slices. HostTcpStream did not advertise vectored-write support, so poll_write_buf submitted only the first roughly 1.4 KB frame. Each VPN packet therefore required a separate asynchronous host operation and guest wakeup. HostTcpStream now combines the queued slices and submits the batch through the existing ordered write operation. The ABI and concurrency model are unchanged.

Underlay and implementation	Forward throughput	Go host CPU
TCP, baseline	302-305 Mbit/s	about 1.44 cores
TCP, typed host imports	394 Mbit/s	about 1.58 cores
TCP, typed imports and vectored writes	1.88-1.93 Gbit/s	about 2.12 cores
UDP reference	914-924 Mbit/s	about 2.01 cores

With probes attached, baseline TCP issued about 25,000 host writes and 36,000 guest drive calls per second while forwarding at 234 Mbit/s. The vectored build issued about 6,800 host writes and 14,000 drive calls per second while forwarding at 1.85 Gbit/s. Average payload per host write grew from approximately 1.2 KB to 34 KB.

TCP reverse throughput reached 969 Mbit/s, compared with 1.05 Gbit/s over the UDP underlay in the same unpinned setup. The remaining reverse difference was 7.7%, rather than the original threefold forward gap.

Round 9: batch host TCP underlay reads

EasyTier commit: 398b699

The forward write batching left the opposite direction asymmetric. FramedReader normally supplied only 2-4 KiB of spare capacity, so each host TCP read transferred at most a few frames before completing the Go operation, copying into guest memory, and waking the guest executor.

HostTcpStream now requests a bounded 64 KiB read even when its caller provides a smaller buffer. It returns the requested prefix immediately and retains the remainder in its existing read buffer. The stream still permits only one pending host read, so the ABI and concurrency model are unchanged and read-ahead remains bounded per active TCP stream.

The direct A/B used the Round 8 topology and CPU placement. System variation put the old artifact at 850 Mbit/s reverse and 1.79 Gbit/s forward during this comparison.

Artifact and direction	Throughput	Go host CPU
Old artifact, reverse	850 Mbit/s	not sampled
64 KiB read-ahead, reverse, first run	1.80 Gbit/s	about 2 cores
64 KiB read-ahead, reverse, repeat	1.84 Gbit/s	about 2 cores
64 KiB read-ahead, reverse, 60 seconds	1.90 Gbit/s	not sampled
Old artifact, forward	1.79 Gbit/s	not sampled
64 KiB read-ahead, forward	1.75-1.76 Gbit/s	not sampled

The repeat reverse run improved by 116%, and the 60-second run showed no tail stall. Forward changed by about 2%, within the observed run-to-run variation. Reverse and forward are therefore symmetric under the same test conditions without enlarging the public ABI or adding concurrent reads.

Intel N100 comparison

The N100 has four physical E-cores with no SMT. The Go host was pinned to CPUs 0-2, while iperf used CPU 3. The native EasyTier peer and iperf server remained on the i7-14700KF host.

Raw TCP between the machines reached 2.30 Gbit/s forward and 2.35 Gbit/s reverse, excluding the network as the limiting component.

Path	Direction/load	Result
Go DialTCP	Go to native	436 Mbit/s
Go DialTCP	native to Go	517 Mbit/s
Go DialUDP	forward, 1 Gbit/s offered	134 Mbit/s, saturated
Go DialUDP	reverse, 300 Mbit/s offered	120 Mbit/s, saturated
Go DialUDP	forward, 100 Mbit/s offered	99.3 Mbit/s, 0.79% loss
Go DialUDP	reverse, 100 Mbit/s offered	99.7 Mbit/s, 0.33% loss

At saturation the Go host used approximately 1.24-1.41 CPU equivalents. UDP loses proportionally more performance than TCP on the weaker cores, consistent with the one-operation-per-datagram ABI and scheduling cost.

Final result

Final Go host commit under test: 47949b5

Embedded EasyTier commit: 4d270f3

Path	Direction	Offered load	Final result
Go DialTCP	Go to native	unlimited	1.31 Gbit/s
Go DialTCP	native to Go	unlimited	1.61 Gbit/s
Go DialUDP	Go to native	1 Gbit/s	684 Mbit/s
Go DialUDP	native to Go	500 Mbit/s	494 Mbit/s, 1.2% loss
Go DialUDP	native to Go	800 Mbit/s	578 Mbit/s, 28% loss

Final TCP and UDP runs lasted 15 seconds after a three-second warm-up. The UDP API remains one operation per datagram. A batch API was deliberately not introduced because it would add a new public operation model and buffering policy for a path whose current performance is acceptable.