aboutsummaryrefslogtreecommitdiffstats
path: root/device/send.go (follow)
Commit message (Collapse)AuthorAgeFilesLines
* device: move Queue{In,Out}boundElement Mutex to container typeJordan Whited2023-10-101-46/+49
| | | | | | | | | | | Queue{In,Out}boundElement locking can contribute to significant overhead via sync.Mutex.lockSlow() in some environments. These types are passed throughout the device package as elements in a slice, so move the per-element Mutex to a container around the slice. Reviewed-by: Maisem Ali <maisem@tailscale.com> Signed-off-by: Jordan Whited <jordan@tailscale.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: distribute crypto work as slice of elementsJordan Whited2023-10-101-29/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | After reducing UDP stack traversal overhead via GSO and GRO, runtime.chanrecv() began to account for a high percentage (20% in one environment) of perf samples during a throughput benchmark. The individual packet channel ops with the crypto goroutines was the primary contributor to this overhead. Updating these channels to pass vectors, which the device package already handles at its ends, reduced this overhead substantially, and improved throughput. The iperf3 results below demonstrate the effect of this commit between two Linux computers with i5-12400 CPUs. There is roughly ~13us of round trip latency between them. The first result is with UDP GSO and GRO, and with single element channels. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.00 sec 12.3 GBytes 10.6 Gbits/sec 232 3.15 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.3 GBytes 10.6 Gbits/sec 232 sender [ 5] 0.00-10.04 sec 12.3 GBytes 10.6 Gbits/sec receiver The second result is with channels updated to pass a slice of elements. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.00 sec 13.2 GBytes 11.3 Gbits/sec 182 3.15 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 13.2 GBytes 11.3 Gbits/sec 182 sender [ 5] 0.00-10.04 sec 13.2 GBytes 11.3 Gbits/sec receiver Reviewed-by: Adrian Dewhurst <adrian@tailscale.com> Signed-off-by: Jordan Whited <jordan@tailscale.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* conn, device: use UDP GSO and GRO on LinuxJordan Whited2023-10-101-0/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | StdNetBind probes for UDP GSO and GRO support at runtime. UDP GSO is dependent on checksum offload support on the egress netdev. UDP GSO will be disabled in the event sendmmsg() returns EIO, which is a strong signal that the egress netdev does not support checksum offload. The iperf3 results below demonstrate the effect of this commit between two Linux computers with i5-12400 CPUs. There is roughly ~13us of round trip latency between them. The first result is from commit 052af4a without UDP GSO or GRO. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.00 sec 9.85 GBytes 8.46 Gbits/sec 1139 3.01 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 9.85 GBytes 8.46 Gbits/sec 1139 sender [ 5] 0.00-10.04 sec 9.85 GBytes 8.42 Gbits/sec receiver The second result is with UDP GSO and GRO. Starting Test: protocol: TCP, 1 streams, 131072 byte blocks [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-10.00 sec 12.3 GBytes 10.6 Gbits/sec 232 3.15 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - Test Complete. Summary Results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 12.3 GBytes 10.6 Gbits/sec 232 sender [ 5] 0.00-10.04 sec 12.3 GBytes 10.6 Gbits/sec receiver Reviewed-by: Adrian Dewhurst <adrian@tailscale.com> Signed-off-by: Jordan Whited <jordan@tailscale.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* global: buff -> bufJason A. Donenfeld2023-03-131-15/+15
| | | | | | This always struck me as kind of weird and non-standard. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* conn, device, tun: implement vectorized I/O plumbingJordan Whited2023-03-101-93/+177
| | | | | | | | | | | | | | | | | | | | Accept packet vectors for reading and writing in the tun.Device and conn.Bind interfaces, so that the internal plumbing between these interfaces now passes a vector of packets. Vectors move untouched between these interfaces, i.e. if 128 packets are received from conn.Bind.Read(), 128 packets are passed to tun.Device.Write(). There is no internal buffering. Currently, existing implementations are only adjusted to have vectors of length one. Subsequent patches will improve that. Also, as a related fixup, use the unix and windows packages rather than the syscall package when possible. Co-authored-by: James Tucker <james@tailscale.com> Signed-off-by: James Tucker <james@tailscale.com> Signed-off-by: Jordan Whited <jordan@tailscale.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* global: bump copyright yearJason A. Donenfeld2023-02-071-1/+1
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* global: bump copyright yearJason A. Donenfeld2022-09-201-1/+1
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* all: use Go 1.19 and its atomic typesBrad Fitzpatrick2022-09-041-11/+10
| | | | | Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* global: apply gofumptJason A. Donenfeld2021-12-091-1/+0
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: simplify allowedips lookup signatureJason A. Donenfeld2021-06-031-2/+2
| | | | | | The inliner should handle this for us. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* tun: linux: account for interface removal from outsideJason A. Donenfeld2021-05-201-1/+5
| | | | | | | | On Linux we can run `ip link del wg0`, in which case the fd becomes stale, and we should exit. Since this is an intentional action, don't treat it as an error. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: add ID to repeated routinesJason A. Donenfeld2021-05-071-3/+3
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: signal to close device in separate routineJason A. Donenfeld2021-03-111-1/+1
| | | | | | Otherwise we wind up deadlocking. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* global: remove TODO name graffitiJason A. Donenfeld2021-02-231-1/+1
| | | | | | | | | | Googlers have a habit of graffiting their name in TODO items that then are never addressed, and other people won't go near those because they're marked territory of another animal. I've been gradually cleaning these up as I see them, but this commit just goes all the way and removes the remaining stragglers. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: make RoutineReadFromTUN keep encryption queue aliveJosh Bleecher Snyder2021-02-091-0/+1
| | | | | | | | | | | | | | | | | | | | | | RoutineReadFromTUN can trigger a call to SendStagedPackets. SendStagedPackets attempts to protect against sending on the encryption queue by checking peer.isRunning and device.isClosed. However, those are subject to TOCTOU bugs. If that happens, we get this: goroutine 1254 [running]: golang.zx2c4.com/wireguard/device.(*Peer).SendStagedPackets(0xc000798300) .../wireguard-go/device/send.go:321 +0x125 golang.zx2c4.com/wireguard/device.(*Device).RoutineReadFromTUN(0xc000014780) .../wireguard-go/device/send.go:271 +0x21c created by golang.zx2c4.com/wireguard/device.NewDevice .../wireguard-go/device/device.go:315 +0x298 Fix this with a simple, big hammer: Keep the encryption queue alive as long as it might be written to. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: do not attach finalizer to non-returned objectJason A. Donenfeld2021-02-091-2/+2
| | | | | | | | Before, the code attached a finalizer to an object that wasn't returned, resulting in immediate garbage collection. Instead return the actual pointer. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: remove mutex from Peer send/receiveJosh Bleecher Snyder2021-02-081-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The immediate motivation for this change is an observed deadlock. 1. A goroutine calls peer.Stop. That calls peer.queue.Lock(). 2. Another goroutine is in RoutineSequentialReceiver. It receives an elem from peer.queue.inbound. 3. The peer.Stop goroutine calls close(peer.queue.inbound), close(peer.queue.outbound), and peer.stopping.Wait(). It blocks waiting for RoutineSequentialReceiver and RoutineSequentialSender to exit. 4. The RoutineSequentialReceiver goroutine calls peer.SendStagedPackets(). SendStagedPackets attempts peer.queue.RLock(). That blocks forever because the peer.Stop goroutine holds a write lock on that mutex. A background motivation for this change is that it can be expensive to have a mutex in the hot code path of RoutineSequential*. The mutex was necessary to avoid attempting to send elems on a closed channel. This commit removes that danger by never closing the channel. Instead, we send a sentinel nil value on the channel to indicate to the receiver that it should exit. The only problem with this is that if the receiver exits, we could write an elem into the channel which would never get received. If it never gets received, it cannot get returned to the device pools. To work around this, we use a finalizer. When the channel can be GC'd, the finalizer drains any remaining elements from the channel and restores them to the device pool. After that change, peer.queue.RWMutex no longer makes sense where it is. It is only used to prevent concurrent calls to Start and Stop. Move it to a more sensible location and make it a plain sync.Mutex. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: overhaul device state managementJosh Bleecher Snyder2021-02-081-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | This commit simplifies device state management. It creates a single unified state variable and documents its semantics. It also makes state changes more atomic. As an example of the sort of bug that occurred due to non-atomic state changes, the following sequence of events used to occur approximately every 2.5 million test runs: * RoutineTUNEventReader received an EventDown event. * It called device.Down, which called device.setUpDown. * That set device.state.changing, but did not yet attempt to lock device.state.Mutex. * Test completion called device.Close. * device.Close locked device.state.Mutex. * device.Close blocked on a call to device.state.stopping.Wait. * device.setUpDown then attempted to lock device.state.Mutex and blocked. Deadlock results. setUpDown cannot progress because device.state.Mutex is locked. Until setUpDown returns, RoutineTUNEventReader cannot call device.state.stopping.Done. Until device.state.stopping.Done gets called, device.state.stopping.Wait is blocked. As long as device.state.stopping.Wait is blocked, device.state.Mutex cannot be unlocked. This commit fixes that deadlock by holding device.state.mu when checking that the device is not closed. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: remove unnecessary zeroing in peer.SendKeepaliveJosh Bleecher Snyder2021-02-081-1/+0
| | | | | | elem.packet is always already nil. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: tie encryption queue lifetime to the peers that write to itJosh Bleecher Snyder2021-02-031-2/+0
| | | | | Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: reduce number of append calls when paddingJason A. Donenfeld2021-01-291-5/+2
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: simplify peer queue lockingJason A. Donenfeld2021-01-291-23/+25
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: reduce nesting when staging packetJason A. Donenfeld2021-01-281-6/+6
| | | | | Suggested-by: Josh Bleecher Snyder <josh@tailscale.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* global: bump copyrightJason A. Donenfeld2021-01-281-1/+1
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: get rid of nonce routineJason A. Donenfeld2021-01-271-136/+52
| | | | | | | | | | | | | This moves to a simple queue with no routine processing it, to reduce scheduler pressure. This splits latency in half! benchmark old ns/op new ns/op delta BenchmarkThroughput-16 2394 2364 -1.25% BenchmarkLatency-16 259652 120810 -53.47% Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: combine debug and info log levels into 'verbose'Jason A. Donenfeld2021-01-261-23/+23
| | | | | | | | | | | | There are very few cases, if any, in which a user only wants one of these levels, so combine it into a single level. While we're at it, reduce indirection on the loggers by using an empty function rather than a nil function pointer. It's not like we have retpolines anyway, and we were always calling through a function with a branch prior, so this seems like a net gain. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: change logging interface to use functionsJosh Bleecher Snyder2021-01-261-34/+23
| | | | | | | | | | | | | | | | | | | | | This commit overhauls wireguard-go's logging. The primary, motivating change is to use a function instead of a *log.Logger as the basic unit of logging. Using functions provides a lot more flexibility for people to bring their own logging system. It also introduces logging helper methods on Device. These reduce line noise at the call site. They also allow for log functions to be nil; when nil, instead of generating a log line and throwing it away, we don't bother generating it at all. This spares allocation and pointless work. This is a breaking change, although the fix required of clients is fairly straightforward. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: remove QueueOutboundElement.droppedJosh Bleecher Snyder2021-01-201-23/+0
| | | | | | | If we block when enqueuing encryption elements to the queue, then we never drop them. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: remove selects from encrypt/decrypt/inbound/outbound enqueuingJosh Bleecher Snyder2021-01-201-18/+2
| | | | | | | Block instead. Backpressure here is fine, probably preferable. This reduces code complexity. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: call wg.Add outside the goroutineJosh Bleecher Snyder2021-01-071-3/+0
| | | | | | | | | | | One of the first rules of WaitGroups is that you call wg.Add outside of a goroutine, not inside it. Fix this embarrassing mistake. This prevents an extremely rare race condition (2 per 100,000 runs) which could occur when attempting to start a new peer concurrently with shutting down a device. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: add missing colon to error lineJason A. Donenfeld2021-01-071-2/+2
| | | | | | | People are actually hitting this condition, so make it uniform. Also, change a printf into a println, to match the other conventions. Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: fix races from changing private_keyJosh Bleecher Snyder2021-01-071-1/+1
| | | | | | | | | | Access keypair.sendNonce atomically. Eliminate one unnecessary initialization to zero. Mutate handshake.lastSentHandshake with the mutex held. Co-authored-by: David Anderson <danderson@tailscale.com> Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: always name *Queue*Element variables elemJosh Bleecher Snyder2021-01-071-10/+10
| | | | | | | | They're called elem in most places. Rename a few local variables to make it consistent. This makes it easier to grep the code for things like elem.Drop. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: use channel close to shut down and drain outbound channelJosh Bleecher Snyder2021-01-071-53/+33
| | | | | | | | | This is a similar treatment to the handling of the encryption channel found a few commits ago: Use the closing of the channel to manage goroutine lifetime and shutdown. It is considerably simpler because there is only a single writer. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: use channel close to shut down and drain encryption channelJosh Bleecher Snyder2021-01-071-59/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The new test introduced in this commit used to deadlock about 1% of the time. I believe that the deadlock occurs as follows: * The test completes, calling device.Close. * device.Close closes device.signals.stop. * RoutineEncryption stops. * The deferred function in RoutineEncryption drains device.queue.encryption. * RoutineEncryption exits. * A peer's RoutineNonce processes an element queued in peer.queue.nonce. * RoutineNonce puts that element into the outbound and encryption queues. * RoutineSequentialSender reads that elements from the outbound queue. * It waits for that element to get Unlocked by RoutineEncryption. * RoutineEncryption has already exited, so RoutineSequentialSender blocks forever. * device.RemoveAllPeers calls peer.Stop on all peers. * peer.Stop waits for peer.routines.stopping, which blocks forever. Rather than attempt to add even more ordering to the already complex centralized shutdown orchestration, this commit moves towards a data-flow-oriented shutdown. The device.queue.encryption gets closed when there will be no more writes to it. All device.queue.encryption readers always read until the channel is closed and then exit. We thus guarantee that any element that enters the encryption queue also exits it. This removes the need for central control of the lifetime of RoutineEncryption, removes the need to drain the encryption queue on shutdown, and simplifies RoutineEncryption. This commit also fixes a data race. When RoutineSequentialSender drains its queue on shutdown, it needs to lock the elem before operating on it, just as the main body does. The new test in this commit passed 50k iterations with the race detector enabled and 150k iterations with the race detector disabled, with no failures. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: remove starting waitgroupsJosh Bleecher Snyder2021-01-071-5/+0
| | | | | | | | | | | | | | | In each case, the starting waitgroup did nothing but ensure that the goroutine has launched. Nothing downstream depends on the order in which goroutines launch, and if the Go runtime scheduler is so broken that goroutines don't get launched reasonably promptly, we have much deeper problems. Given all that, simplify the code. Passed a race-enabled stress test 25,000 times without failure. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: clear pointers when returning elems to poolsJosh Bleecher Snyder2020-12-081-1/+11
| | | | Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: use labeled for loop instead of gotoJosh Bleecher Snyder2020-12-081-4/+4
| | | | | | Minor code cleanup; no functional changes. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
* device: add write queue mutex for peerHaichao Liu2020-11-181-0/+4
| | | | | | | fix panic: send on closed channel when remove peer Signed-off-by: Haichao Liu <liuhaichao@bytedance.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* device: rework padding calculation and don't shadow paddedSizeJason A. Donenfeld2020-05-181-15/+17
| | | | | Reported-by: Jayakumar S <jayakumar82.s@gmail.com> Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* global: update header comments and modulesJason A. Donenfeld2020-05-021-1/+1
| | | | Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>
* send: account for zero mtuJason A. Donenfeld2020-02-141-4/+12
| | | | Don't divide by zero.
* device: remove dead error reporting codeJason A. Donenfeld2019-10-211-4/+1
|
* device: update transfer counters correctlyJason A. Donenfeld2019-06-111-2/+0
| | | | | The rule is to always update them to the full packet size minus UDP/IP encapsulation for all authenticated packet types.
* device, ratelimiter: replace uses of time.Now().Sub() with time.Since()Matt Layher2019-06-031-4/+4
| | | | | | | | | | | | | | | | | | | | | | | Simplification found by staticcheck: $ staticcheck ./... | grep S1012 device/cookie.go:90:5: should use time.Since instead of time.Now().Sub (S1012) device/cookie.go:127:5: should use time.Since instead of time.Now().Sub (S1012) device/cookie.go:242:5: should use time.Since instead of time.Now().Sub (S1012) device/noise-protocol.go:304:13: should use time.Since instead of time.Now().Sub (S1012) device/receive.go:82:46: should use time.Since instead of time.Now().Sub (S1012) device/send.go:132:5: should use time.Since instead of time.Now().Sub (S1012) device/send.go:139:5: should use time.Since instead of time.Now().Sub (S1012) device/send.go:235:59: should use time.Since instead of time.Now().Sub (S1012) device/send.go:393:9: should use time.Since instead of time.Now().Sub (S1012) ratelimiter/ratelimiter.go:79:10: should use time.Since instead of time.Now().Sub (S1012) ratelimiter/ratelimiter.go:87:10: should use time.Since instead of time.Now().Sub (S1012) Change applied using: $ find . -type f -name "*.go" -exec sed -i "s/Now().Sub(/Since(/g" {} \; Signed-off-by: Matt Layher <mdlayher@gmail.com>
* global: regroup all importsJason A. Donenfeld2019-05-141-3/+4
|
* device: send: check packet length before freeing elementJason A. Donenfeld2019-04-181-3/+3
|
* global: begin modularizationJason A. Donenfeld2019-03-031-0/+618
Copyright © 1996 – 2023 Jason A. Donenfeld. All Rights Reverse Engineered.