path: root/device/send.go
diff options
authorJosh Bleecher Snyder <josh@tailscale.com>2021-02-08 13:02:52 -0800
committerJosh Bleecher Snyder <josh@tailscale.com>2021-02-08 13:02:52 -0800
commitd8dd1f254fc42878970d764b046ec5789ab30259 (patch)
tree5921b86dc7d81bce2efa1a221e52f0e880e957a0 /device/send.go
parentdevice: create channels.go (diff)
device: remove mutex from Peer send/receive
The immediate motivation for this change is an observed deadlock. 1. A goroutine calls peer.Stop. That calls peer.queue.Lock(). 2. Another goroutine is in RoutineSequentialReceiver. It receives an elem from peer.queue.inbound. 3. The peer.Stop goroutine calls close(peer.queue.inbound), close(peer.queue.outbound), and peer.stopping.Wait(). It blocks waiting for RoutineSequentialReceiver and RoutineSequentialSender to exit. 4. The RoutineSequentialReceiver goroutine calls peer.SendStagedPackets(). SendStagedPackets attempts peer.queue.RLock(). That blocks forever because the peer.Stop goroutine holds a write lock on that mutex. A background motivation for this change is that it can be expensive to have a mutex in the hot code path of RoutineSequential*. The mutex was necessary to avoid attempting to send elems on a closed channel. This commit removes that danger by never closing the channel. Instead, we send a sentinel nil value on the channel to indicate to the receiver that it should exit. The only problem with this is that if the receiver exits, we could write an elem into the channel which would never get received. If it never gets received, it cannot get returned to the device pools. To work around this, we use a finalizer. When the channel can be GC'd, the finalizer drains any remaining elements from the channel and restores them to the device pool. After that change, peer.queue.RWMutex no longer makes sense where it is. It is only used to prevent concurrent calls to Start and Stop. Move it to a more sensible location and make it a plain sync.Mutex. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
Diffstat (limited to 'device/send.go')
1 files changed, 3 insertions, 2 deletions
diff --git a/device/send.go b/device/send.go
index 982fec0..911ee5c 100644
--- a/device/send.go
+++ b/device/send.go
@@ -316,7 +316,6 @@ top:
// add to parallel and sequential queue
- peer.queue.RLock()
if peer.isRunning.Get() {
peer.queue.outbound <- elem
peer.device.queue.encryption.c <- elem
@@ -324,7 +323,6 @@ top:
- peer.queue.RUnlock()
@@ -413,6 +411,9 @@ func (peer *Peer) RoutineSequentialSender() {
device.log.Verbosef("%v - Routine: sequential sender - started", peer)
for elem := range peer.queue.outbound {
+ if elem == nil {
+ return
+ }
if !peer.isRunning.Get() {
// peer has been stopped; return re-usable elems to the shared pool.