diff options
| author |   Pavel Begunkov <asml.silence@gmail.com> | 2023-08-24 23:53:29 +0100 |
|---|---|---|
| committer |   Jens Axboe <axboe@kernel.dk> | 2023-08-24 17:16:19 -0600 |
| commit | ec26c225f06f5993f8891fa6c79fab3c92981181 (patch) | |
| tree | ac8f3e43760f70381360a290840a743955e8278a /include/linux/io_uring_types.h | |
| parent | io_uring: reorder cqring_flush and wakeups (diff) | |
| download | wireguard-linux-ec26c225f06f5993f8891fa6c79fab3c92981181.tar.xz wireguard-linux-ec26c225f06f5993f8891fa6c79fab3c92981181.zip | |
io_uring: merge iopoll and normal completion paths
io_do_iopoll() and io_submit_flush_completions() are pretty similar,
both filling CQEs and then free a list of requests. Don't duplicate it
and make iopoll use __io_submit_flush_completions(), which also helps
with inlining and other optimisations.
For that, we need to first find all completed iopoll requests and splice
them from the iopoll list and then pass it down. This adds one extra
list traversal, which should be fine as requests will stay hot in cache.
CQ locking is already conditional, introduce ->lockless_cq and skip
locking for IOPOLL as it's protected by ->uring_lock.
We also add a wakeup optimisation for IOPOLL to __io_cq_unlock_post(),
so it works just like io_cqring_ev_posted_iopoll().
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
Link: https://lore.kernel.org/r/3840473f5e8a960de35b77292026691880f6bdbc.1692916914.git.asml.silence@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Diffstat (limited to 'include/linux/io_uring_types.h')
| -rw-r--r-- | include/linux/io_uring_types.h | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 9795eda529f7..c0c03d8059df 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -205,6 +205,7 @@ struct io_ring_ctx { unsigned int has_evfd: 1; /* all CQEs should be posted only by the submitter task */ unsigned int task_complete: 1; + unsigned int lockless_cq: 1; unsigned int syscall_iopoll: 1; unsigned int poll_activated: 1; unsigned int drain_disabled: 1; |