linux-dev/arch/x86/kernel/jump_label.c, branch linus/master

jump_labels: Mark __jump_label_transform() as __always_inlined to work around aggressive compiler un-inlining

2021-07-13T04:32:05Z

In randconfig testing, certain UBSAN and CC Kconfig combinations with GCC 10.3.0: CONFIG_X86_32=y CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_UBSAN=y # CONFIG_UBSAN_TRAP is not set # CONFIG_UBSAN_BOUNDS is not set CONFIG_UBSAN_SHIFT=y # CONFIG_UBSAN_DIV_ZERO is not set CONFIG_UBSAN_UNREACHABLE=y CONFIG_UBSAN_BOOL=y # CONFIG_UBSAN_ENUM is not set # CONFIG_UBSAN_ALIGNMENT is not set # CONFIG_UBSAN_SANITIZE_ALL is not set ... produce this build warning (and build error if CONFIG_SECTION_MISMATCH_WARN_ONLY=y is set): WARNING: modpost: vmlinux.o(.text+0x4c1cc): Section mismatch in reference from the function __jump_label_transform() to the function .init.text:text_poke_early() The function __jump_label_transform() references the function __init text_poke_early(). This is often because __jump_label_transform lacks a __init annotation or the annotation of text_poke_early is wrong. ERROR: modpost: Section mismatches detected. The problem is that __jump_label_transform() gets uninlined by GCC, despite there being only a single local scope user of the 'static inline' function. Mark the function __always_inline instead, to work around this compiler bug/artifact. Signed-off-by: Ingo Molnar

jump_label/x86: Remove unused JUMP_LABEL_NOP_SIZE

2021-05-14T07:00:09Z

JUMP_LABEL_NOP_SIZE is now unused, remove it. Fixes: 001951bea748 ("jump_label, x86: Add variable length patching support") Reported-by: Miroslav Benes Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/YJ00zxsvocDV5vLU@hirez.programming.kicks-ass.net

jump_label, x86: Emit short JMP

2021-05-12T12:54:55Z

Now that we can patch short JMP/NOP, allow the compiler/assembler to emit short JMP instructions. There is no way to have the assembler emit short NOPs based on the potential displacement, so leave those long for now. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/20210506194157.967034497@infradead.org

jump_label, x86: Add variable length patching support

2021-05-12T12:54:55Z

This allows the patching to to emit 2 byte JMP/NOP instruction in addition to the 5 byte JMP/NOP we already did. This allows for more compact code. This code is not yet used, as we don't emit shorter code at compile time yet. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/20210506194157.846870383@infradead.org

jump_label, x86: Introduce jump_entry_size()

2021-05-12T12:54:55Z

This allows architectures to have variable sized jumps. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/20210506194157.786777050@infradead.org

jump_label, x86: Improve error when we fail expected text

2021-05-12T12:54:55Z

There is only a single usage site left, remove the function and extend the print to include more information, like the expected text and the patch type. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Ingo Molnar Link: https://lore.kernel.org/r/20210506194157.726939027@infradead.org

x86: Remove dynamic NOP selection

2021-03-15T15:24:59Z

This ensures that a NOP is a NOP and not a random other instruction that is also a NOP. It allows simplification of dynamic code patching that wants to verify existing code before writing new instructions (ftrace, jump_label, static_call, etc..). Differentiating on NOPs is not a feature. This pessimises 32bit (DONTCARE) and 32bit on 64bit CPUs (CARELESS). 32bit is not a performance target. Everything x86_64 since AMD K10 (2007) and Intel IvyBridge (2012) is fine with using NOPL (as opposed to prefix NOP). And per FEATURE_NOPL being required for x86_64, all x86_64 CPUs can use NOPL. So stop caring about NOPs, simplify things and get on with life. [ The problem seems to be that some uarchs can only decode NOPL on a single front-end port while others have severe decode penalties for excessive prefixes. All modern uarchs can handle both, except Atom, which has prefix penalties. ] [ Also, much doubt you can actually measure any of this on normal workloads. ] After this, FEATURE_NOPL is unused except for required-features for x86_64. FEATURE_K8 is only used for PTI. [ bp: Kernel build measurements showed ~0.3s slowdown on Sandybridge which is hardly a slowdown. Get rid of X86_FEATURE_K7, while at it. ] Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Borislav Petkov Acked-by: Alexei Starovoitov # bpf Acked-by: Linus Torvalds Link: https://lkml.kernel.org/r/20210312115749.065275711@infradead.org

x86/jump_label: Move 'inline' keyword placement

2020-03-27T10:05:41Z

Fix gcc warning when -Wextra is used by moving the keyword: arch/x86/kernel/jump_label.c:61:1: warning: ‘inline’ is not at \ beginning of declaration [-Wold-style-declaration] static void inline __jump_label_transform(struct jump_entry *entry, ^~~~~~ Reported-by: Zzy Wysm Signed-off-by: Randy Dunlap Signed-off-by: Borislav Petkov Link: https://lkml.kernel.org/r/796d93d2-e73e-3447-44eb-4f89e1b636d9@infradead.org

x86/alternatives: Add and use text_gen_insn() helper

2019-11-27T06:44:24Z

Provide a simple helper function to create common instruction encodings. Tested-by: Alexei Starovoitov Tested-by: Steven Rostedt (VMware) Signed-off-by: Peter Zijlstra (Intel) Acked-by: Alexei Starovoitov Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Daniel Bristot de Oliveira Cc: H. Peter Anvin Cc: Josh Poimboeuf Cc: Linus Torvalds Cc: Masami Hiramatsu Cc: Steven Rostedt Cc: Thomas Gleixner Link: https://lkml.kernel.org/r/20191111132457.703538332@infradead.org Signed-off-by: Ingo Molnar

x86/alternatives, jump_label: Provide better text_poke() batching interface

2019-11-27T06:44:24Z

Adding another text_poke_bp_batch() user made me realize the interface is all sorts of wrong. The text poke vector should be internal to the implementation. This then results in a trivial interface: text_poke_queue() - which has the 'normal' text_poke_bp() interface text_poke_finish() - which takes no arguments and flushes any pending text_poke()s. Tested-by: Alexei Starovoitov Tested-by: Steven Rostedt (VMware) Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Masami Hiramatsu Reviewed-by: Daniel Bristot de Oliveira Acked-by: Alexei Starovoitov Cc: Andy Lutomirski Cc: Borislav Petkov Cc: H. Peter Anvin Cc: Josh Poimboeuf Cc: Linus Torvalds Cc: Steven Rostedt Cc: Thomas Gleixner Link: https://lkml.kernel.org/r/20191111132457.646280715@infradead.org Signed-off-by: Ingo Molnar