Merge branch 'jmp32-insns'

Jiong Wang says:

====================
v3 -> v4:
 - Fixed rebase issue. JMP32 checks were missing in two new functions:
    + kernel/bpf/verifier.c:insn_is_cond_jump
    + drivers/net/ethernet/netronome/nfp/bpf/main.h:is_mbpf_cond_jump
   (Daniel)
 - Further rebased on top of latest llvm-readelf change.

v2 -> v3:
 - Added missed check on JMP32 inside bpf_jit_build_body. (Sandipan)
 - Wrap ?: statements in s390 port with brace. They are used by macros
   which doesn't guard the operand with brace.
 - Fixed the ',' issues test_verifier change.
 - Reorder two selftests patches to be near each other.
 - Rebased on top of latest bpf-next.

v1 -> v2:
 - Updated encoding. Use reserved insn class 0x6 instead of packing with
   existing BPF_JMP. (Alexei)
 - Updated code comments in s390 port. (Martin)
 - Separate JIT function for jeq32_imm in NFP port. (Jakub)
 - Re-implemented auto-testing support. (Jakub)
 - Moved testcases to test_verifer.c, plus more unit tests. (Jakub)
 - Fixed JEQ/JNE range deduction. (Jakub)
 - Also supported JSET in this patch set.
 - Fixed/Improved range deduction for all the other operations. All C
   programs under bpf selftest passed verification now.
 - Improved min/max code implementation.
 - Fixed bpftool/disassembler.

Current eBPF ISA has 32-bit sub-register and has defined a set of ALU32
instructions.

However, there is no JMP32 instructions, the consequence is code-gen for
32-bit sub-registers is not efficient. For example, explicit sign-extension
from 32-bit to 64-bit is needed for signed comparison.

Adding JMP32 instruction therefore could complete eBPF ISA on 32-bit
sub-register support. This also match those JMP32 instructions in most JIT
backends, for example x64-64 and AArch64. These new eBPF JMP32 instructions
could have one-to-one map on them.

A few verifier ALU32 related bugs has been fixed recently, and JMP32
introduced by this set further improves BPF sub-register ecosystem. Once
this is landed, BPF programs using 32-bit sub-register ISA could get
reasonably good support from verifier and JIT compilers. Users then could
compare the runtime efficiency of one BPF program under both modes, and
could use the one shown better from benchmark result.

From benchmark results on some Cilium BPF programs, for 64-bit arches,
after JMP32 introduced, programs compiled with -mattr=+alu32 (meaning
enable sub-register usage) are smaller in code size and generally smaller
in verifier processed insn number.

Benchmark results
===
Text size in bytes (generated by "size")
---
LLVM code-gen option   default  alu32  alu32/jmp32  change Vs.  change Vs.
                                                    alu32       default
bpf_lb-DLB_L3.o:       6456     6280   6160         -1.91%      -4.58%
bpf_lb-DLB_L4.o:       7848     7664   7136         -6.89%      -9.07%
bpf_lb-DUNKNOWN.o:     2680     2664   2568         -3.60%      -4.18%
bpf_lxc.o:             104824   104744 97360        -7.05%      -7.12%
bpf_netdev.o:          23456    23576  21632        -8.25%      -7.78%
bpf_overlay.o:         16184    16304  14648        -10.16%     -9.49%

Processed instruction number
---
LLVM code-gen option   default  alu32  alu32/jmp32  change Vs.  change Vs.
                                                    alu32       default
bpf_lb-DLB_L3.o:       1579     1281   1295         +1.09%      -17.99%
bpf_lb-DLB_L4.o:       2045     1663   1556         -6.43%      -23.91%
bpf_lb-DUNKNOWN.o:     606      513    501          -2.34%      -17.33%
bpf_lxc.o:             85381    103218 94435        -8.51%      +10.60%
bpf_netdev.o:          5246     5809   5200         -10.48%     -0.08%
bpf_overlay.o:         2443     2705   2456         -9.02%      -0.53%

It is even better for 32-bit arches like x32, arm32 and nfp etc, as now
some conditional jump will become JMP32 which doesn't require code-gen for
high 32-bit comparison.

Encoding
===
The new JMP32 instructions are using new BPF_JMP32 class which is using
the reserved eBPF class number 0x6. And BPF_JA/CALL/EXIT only exist for
BPF_JMP, they are reserved opcode for BPF_JMP32.

LLVM support
===
A couple of unit tests has been added and included in this set. Also LLVM
code-gen for JMP32 has been added, so you could just compile any BPF C
program with both -mcpu=probe and -mattr=+alu32 specified. If you are
compiling on a machine with kernel patched by this set, LLVM will select
the ISA automatically based on host probe results. Otherwise specify
-mcpu=v3 and -mattr=+alu32 could also force use JMP32 ISA.

   LLVM support could be found at:

     https://github.com/Netronome/llvm/tree/jmp32-v2

   (clang driver also taught about the new "v3" processor, will send out
    merge request for both clang and llvm once kernel set landed.)

JIT backends support
===
A couple of JIT backends has been supported in this set except SPARC and
MIPS. It shouldn't be a big issue for these two ports as LLVM default won't
generate JMP32 insns, it will only generate them when host machine is
probed to be with the support.

Thanks.
====================

Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

2 files changed, 832 insertions, 49 deletions

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 1cca5f2567c2..d61b023c8ddc 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -10,15 +10,14 @@ ifneq ($(wildcard $(GENHDR)),)
   GENFLAGS := -DHAVE_GENHDR
 endif
 
+CLANG		?= clang
+LLC		?= llc
+LLVM_OBJCOPY	?= llvm-objcopy
+LLVM_READELF	?= llvm-readelf
+BTF_PAHOLE	?= pahole
 CFLAGS += -Wall -O2 -I$(APIDIR) -I$(LIBDIR) -I$(BPFDIR) -I$(GENDIR) $(GENFLAGS) -I../../../include
 LDLIBS += -lcap -lelf -lrt -lpthread
 
-TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read
-all: $(TEST_CUSTOM_PROGS)
-
-$(TEST_CUSTOM_PROGS): $(OUTPUT)/%: %.c
-	$(CC) -o $(TEST_CUSTOM_PROGS) -static $< -Wl,--build-id
-
 # Order correspond to 'make run_tests' order
 TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test_progs \
 	test_align test_verifier_log test_dev_cgroup test_tcpbpf_user \
@@ -26,21 +25,42 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test
 	test_socket_cookie test_cgroup_storage test_select_reuseport test_section_names \
 	test_netcnt test_tcpnotify_user
 
-TEST_GEN_FILES = test_pkt_access.o test_xdp.o test_l4lb.o test_tcp_estats.o test_obj_id.o \
-	test_pkt_md_access.o test_xdp_redirect.o test_xdp_meta.o sockmap_parse_prog.o     \
-	sockmap_verdict_prog.o dev_cgroup.o sample_ret0.o test_tracepoint.o \
-	test_l4lb_noinline.o test_xdp_noinline.o test_stacktrace_map.o \
-	test_tcpnotify_kern.o \
-	sample_map_ret0.o test_tcpbpf_kern.o test_stacktrace_build_id.o \
-	sockmap_tcp_msg_prog.o connect4_prog.o connect6_prog.o test_adjust_tail.o \
-	test_btf_haskv.o test_btf_nokv.o test_sockmap_kern.o test_tunnel_kern.o \
-	test_get_stack_rawtp.o test_sockmap_kern.o test_sockhash_kern.o \
-	test_lwt_seg6local.o sendmsg4_prog.o sendmsg6_prog.o test_lirc_mode2_kern.o \
+BPF_OBJ_FILES = \
+	test_xdp_redirect.o test_xdp_meta.o sockmap_parse_prog.o \
+	sockmap_verdict_prog.o dev_cgroup.o sample_ret0.o \
+	test_tcpnotify_kern.o sample_map_ret0.o test_tcpbpf_kern.o \
+	sockmap_tcp_msg_prog.o connect4_prog.o connect6_prog.o \
+	test_btf_haskv.o test_btf_nokv.o test_sockmap_kern.o \
+	test_tunnel_kern.o test_sockhash_kern.o test_lwt_seg6local.o \
+	sendmsg4_prog.o sendmsg6_prog.o test_lirc_mode2_kern.o \
 	get_cgroup_id_kern.o socket_cookie_prog.o test_select_reuseport_kern.o \
-	test_skb_cgroup_id_kern.o bpf_flow.o netcnt_prog.o \
-	test_sk_lookup_kern.o test_xdp_vlan.o test_queue_map.o test_stack_map.o \
+	test_skb_cgroup_id_kern.o bpf_flow.o netcnt_prog.o test_xdp_vlan.o \
 	xdp_dummy.o test_map_in_map.o
 
+# Objects are built with default compilation flags and with sub-register
+# code-gen enabled.
+BPF_OBJ_FILES_DUAL_COMPILE = \
+	test_pkt_access.o test_pkt_access.o test_xdp.o test_adjust_tail.o \
+	test_l4lb.o test_l4lb_noinline.o test_xdp_noinline.o test_tcp_estats.o \
+	test_obj_id.o test_pkt_md_access.o test_tracepoint.o \
+	test_stacktrace_map.o test_stacktrace_map.o test_stacktrace_build_id.o \
+	test_stacktrace_build_id.o test_get_stack_rawtp.o \
+	test_get_stack_rawtp.o test_tracepoint.o test_sk_lookup_kern.o \
+	test_queue_map.o test_stack_map.o
+
+TEST_GEN_FILES = $(BPF_OBJ_FILES) $(BPF_OBJ_FILES_DUAL_COMPILE)
+
+# Also test sub-register code-gen if LLVM + kernel both has eBPF v3 processor
+# support which is the first version to contain both ALU32 and JMP32
+# instructions.
+SUBREG_CODEGEN := $(shell echo "int cal(int a) { return a > 0; }" | \
+			$(CLANG) -target bpf -O2 -emit-llvm -S -x c - -o - | \
+			$(LLC) -mattr=+alu32 -mcpu=probe 2>&1 | \
+			grep 'if w')
+ifneq ($(SUBREG_CODEGEN),)
+TEST_GEN_FILES += $(patsubst %.o,alu32/%.o, $(BPF_OBJ_FILES_DUAL_COMPILE))
+endif
+
 # Order correspond to 'make run_tests' order
 TEST_PROGS := test_kmod.sh \
 	test_libbpf.sh \
@@ -65,6 +85,13 @@ TEST_GEN_PROGS_EXTENDED = test_libbpf_open test_sock_addr test_skb_cgroup_id_use
 
 include ../lib.mk
 
+# NOTE: $(OUTPUT) won't get default value if used before lib.mk
+TEST_CUSTOM_PROGS = $(OUTPUT)/urandom_read
+all: $(TEST_CUSTOM_PROGS)
+
+$(OUTPUT)/urandom_read: $(OUTPUT)/%: %.c
+	$(CC) -o $@ -static $< -Wl,--build-id
+
 BPFOBJ := $(OUTPUT)/libbpf.a
 
 $(TEST_GEN_PROGS): $(BPFOBJ)
@@ -92,12 +119,6 @@ force:
 $(BPFOBJ): force
 	$(MAKE) -C $(BPFDIR) OUTPUT=$(OUTPUT)/
 
-CLANG ?= clang
-LLC   ?= llc
-LLVM_OBJCOPY ?= llvm-objcopy
-LLVM_READELF ?= llvm-readelf
-BTF_PAHOLE ?= pahole
-
 PROBE := $(shell $(LLC) -march=bpf -mcpu=probe -filetype=null /dev/null 2>&1)
 
 # Let newer LLVM versions transparently probe the kernel for availability
@@ -149,6 +170,30 @@ endif
 endif
 endif
 
+ifneq ($(SUBREG_CODEGEN),)
+ALU32_BUILD_DIR = $(OUTPUT)/alu32
+TEST_CUSTOM_PROGS += $(ALU32_BUILD_DIR)/test_progs_32
+$(ALU32_BUILD_DIR):
+	mkdir -p $@
+
+$(ALU32_BUILD_DIR)/urandom_read: $(OUTPUT)/urandom_read
+	cp $< $@
+
+$(ALU32_BUILD_DIR)/test_progs_32: test_progs.c $(ALU32_BUILD_DIR) \
+						$(ALU32_BUILD_DIR)/urandom_read
+	$(CC) $(CFLAGS) -o $(ALU32_BUILD_DIR)/test_progs_32 $< \
+		trace_helpers.c $(OUTPUT)/libbpf.a $(LDLIBS)
+
+$(ALU32_BUILD_DIR)/%.o: %.c $(ALU32_BUILD_DIR) $(ALU32_BUILD_DIR)/test_progs_32
+	$(CLANG) $(CLANG_FLAGS) \
+		 -O2 -target bpf -emit-llvm -c $< -o - |      \
+	$(LLC) -march=bpf -mattr=+alu32 -mcpu=$(CPU) $(LLC_FLAGS) \
+		-filetype=obj -o $@
+ifeq ($(DWARF2BTF),y)
+	$(BTF_PAHOLE) -J $@
+endif
+endif
+
 # Have one program compiled without "-target bpf" to test whether libbpf loads
 # it successfully
 $(OUTPUT)/test_xdp.o: test_xdp.c
@@ -167,4 +212,4 @@ ifeq ($(DWARF2BTF),y)
 	$(BTF_PAHOLE) -J $@
 endif
 
-EXTRA_CLEAN := $(TEST_CUSTOM_PROGS)
+EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(ALU32_BUILD_DIR)
diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c
index e4fef6ca8071..1df757645a9a 100644
--- a/tools/testing/selftests/bpf/test_verifier.c
+++ b/tools/testing/selftests/bpf/test_verifier.c
@@ -211,6 +211,42 @@ static void bpf_fill_rand_ld_dw(struct bpf_test *self)
 	BPF_MOV64_IMM(BPF_REG_5, 0),					\
 	BPF_EMIT_CALL(BPF_FUNC_sk_lookup_tcp)
 
+/* BPF_DIRECT_PKT_R2 contains 7 instructions, it initializes default return
+ * value into 0 and does necessary preparation for direct packet access
+ * through r2. The allowed access range is 8 bytes.
+ */
+#define BPF_DIRECT_PKT_R2						\
+	BPF_MOV64_IMM(BPF_REG_0, 0),					\
+	BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1,			\
+		    offsetof(struct __sk_buff, data)),			\
+	BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1,			\
+		    offsetof(struct __sk_buff, data_end)),		\
+	BPF_MOV64_REG(BPF_REG_4, BPF_REG_2),				\
+	BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 8),				\
+	BPF_JMP_REG(BPF_JLE, BPF_REG_4, BPF_REG_3, 1),			\
+	BPF_EXIT_INSN()
+
+/* BPF_RAND_UEXT_R7 contains 4 instructions, it initializes R7 into a random
+ * positive u32, and zero-extend it into 64-bit.
+ */
+#define BPF_RAND_UEXT_R7						\
+	BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,			\
+		     BPF_FUNC_get_prandom_u32),				\
+	BPF_MOV64_REG(BPF_REG_7, BPF_REG_0),				\
+	BPF_ALU64_IMM(BPF_LSH, BPF_REG_7, 33),				\
+	BPF_ALU64_IMM(BPF_RSH, BPF_REG_7, 33)
+
+/* BPF_RAND_SEXT_R7 contains 5 instructions, it initializes R7 into a random
+ * negative u32, and sign-extend it into 64-bit.
+ */
+#define BPF_RAND_SEXT_R7						\
+	BPF_RAW_INSN(BPF_JMP | BPF_CALL, 0, 0, 0,			\
+		     BPF_FUNC_get_prandom_u32),				\
+	BPF_MOV64_REG(BPF_REG_7, BPF_REG_0),				\
+	BPF_ALU64_IMM(BPF_OR, BPF_REG_7, 0x80000000),			\
+	BPF_ALU64_IMM(BPF_LSH, BPF_REG_7, 32),				\
+	BPF_ALU64_IMM(BPF_ARSH, BPF_REG_7, 32)
+
 static struct bpf_test tests[] = {
 	{
 		"add+sub+mul",
@@ -15447,18 +15483,7 @@ static struct bpf_test tests[] = {
 	{
 		"jset: functional",
 		.insns = {
-			/* r0 = 0 */
-			BPF_MOV64_IMM(BPF_REG_0, 0),
-			/* prep for direct packet access via r2 */
-			BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1,
-				    offsetof(struct __sk_buff, data)),
-			BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1,
-				    offsetof(struct __sk_buff, data_end)),
-			BPF_MOV64_REG(BPF_REG_4, BPF_REG_2),
-			BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 8),
-			BPF_JMP_REG(BPF_JLE, BPF_REG_4, BPF_REG_3, 1),
-			BPF_EXIT_INSN(),
-
+			BPF_DIRECT_PKT_R2,
 			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
 
 			/* reg, bit 63 or bit 0 set, taken */
@@ -15514,18 +15539,7 @@ static struct bpf_test tests[] = {
 	{
 		"jset: sign-extend",
 		.insns = {
-			/* r0 = 0 */
-			BPF_MOV64_IMM(BPF_REG_0, 0),
-			/* prep for direct packet access via r2 */
-			BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1,
-				    offsetof(struct __sk_buff, data)),
-			BPF_LDX_MEM(BPF_W, BPF_REG_3, BPF_REG_1,
-				    offsetof(struct __sk_buff, data_end)),
-			BPF_MOV64_REG(BPF_REG_4, BPF_REG_2),
-			BPF_ALU64_IMM(BPF_ADD, BPF_REG_4, 8),
-			BPF_JMP_REG(BPF_JLE, BPF_REG_4, BPF_REG_3, 1),
-			BPF_EXIT_INSN(),
-
+			BPF_DIRECT_PKT_R2,
 			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
 
 			BPF_JMP_IMM(BPF_JSET, BPF_REG_7, 0x80000000, 1),
@@ -15795,6 +15809,730 @@ static struct bpf_test tests[] = {
 		.result = ACCEPT,
 		.retval = 2,
 	},
+	{
+		"jset32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			/* reg, high bits shouldn't be tested */
+			BPF_JMP32_IMM(BPF_JSET, BPF_REG_7, -2, 1),
+			BPF_JMP_IMM(BPF_JA, 0, 0, 1),
+			BPF_EXIT_INSN(),
+
+			BPF_JMP32_IMM(BPF_JSET, BPF_REG_7, 1, 1),
+			BPF_EXIT_INSN(),
+			BPF_MOV64_IMM(BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { 1ULL << 63, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1ULL << 63 | 1, }
+			},
+		},
+	},
+	{
+		"jset32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_LD_IMM64(BPF_REG_8, 0x8000000000000000),
+			BPF_JMP32_REG(BPF_JSET, BPF_REG_7, BPF_REG_8, 1),
+			BPF_JMP_IMM(BPF_JA, 0, 0, 1),
+			BPF_EXIT_INSN(),
+
+			BPF_LD_IMM64(BPF_REG_8, 0x8000000000000001),
+			BPF_JMP32_REG(BPF_JSET, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_MOV64_IMM(BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { 1ULL << 63, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1ULL << 63 | 1, }
+			},
+		},
+	},
+	{
+		"jset32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_MOV64_IMM(BPF_REG_0, 0),
+			BPF_JMP32_IMM(BPF_JSET, BPF_REG_7, 0x10, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP32_IMM(BPF_JGE, BPF_REG_7, 0x10, 1),
+			BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+	},
+	{
+		"jeq32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JEQ, BPF_REG_7, -1, 1),
+			BPF_EXIT_INSN(),
+			BPF_MOV64_IMM(BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 2,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { -2, }
+			},
+			{ .retval = 2,
+			  .data64 = { -1, }
+			},
+		},
+	},
+	{
+		"jeq32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_LD_IMM64(BPF_REG_8, 0x7000000000000001),
+			BPF_JMP32_REG(BPF_JEQ, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_MOV64_IMM(BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { 2, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1ULL << 63 | 1, }
+			},
+		},
+	},
+	{
+		"jeq32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_MOV64_IMM(BPF_REG_0, 0),
+			BPF_JMP32_IMM(BPF_JEQ, BPF_REG_7, 0x10, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP32_IMM(BPF_JSGE, BPF_REG_7, 0xf, 1),
+			BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+	},
+	{
+		"jne32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JNE, BPF_REG_7, -1, 1),
+			BPF_EXIT_INSN(),
+			BPF_MOV64_IMM(BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 2,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { 1, }
+			},
+			{ .retval = 0,
+			  .data64 = { -1, }
+			},
+		},
+	},
+	{
+		"jne32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_LD_IMM64(BPF_REG_8, 0x8000000000000001),
+			BPF_JMP32_REG(BPF_JNE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_MOV64_IMM(BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { 1, }
+			},
+			{ .retval = 2,
+			  .data64 = { 2, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1ULL << 63 | 2, }
+			},
+		},
+	},
+	{
+		"jne32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_MOV64_IMM(BPF_REG_0, 0),
+			BPF_JMP32_IMM(BPF_JNE, BPF_REG_7, 0x10, 1),
+			BPF_JMP_IMM(BPF_JNE, BPF_REG_7, 0x10, 1),
+			BPF_EXIT_INSN(),
+			BPF_LDX_MEM(BPF_B, BPF_REG_8, BPF_REG_9, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+	},
+	{
+		"jge32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JGE, BPF_REG_7, UINT_MAX - 1, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { UINT_MAX, }
+			},
+			{ .retval = 2,
+			  .data64 = { UINT_MAX - 1, }
+			},
+			{ .retval = 0,
+			  .data64 = { 0, }
+			},
+		},
+	},
+	{
+		"jge32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, UINT_MAX | 1ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JGE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { UINT_MAX, }
+			},
+			{ .retval = 0,
+			  .data64 = { INT_MAX, }
+			},
+			{ .retval = 0,
+			  .data64 = { (UINT_MAX - 1) | 2ULL << 32, }
+			},
+		},
+	},
+	{
+		"jge32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffff0 | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JGE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP32_IMM(BPF_JGE, BPF_REG_7, 0x7ffffff0, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
+	{
+		"jgt32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JGT, BPF_REG_7, UINT_MAX - 1, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { UINT_MAX, }
+			},
+			{ .retval = 0,
+			  .data64 = { UINT_MAX - 1, }
+			},
+			{ .retval = 0,
+			  .data64 = { 0, }
+			},
+		},
+	},
+	{
+		"jgt32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, (UINT_MAX - 1) | 1ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JGT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { UINT_MAX, }
+			},
+			{ .retval = 0,
+			  .data64 = { UINT_MAX - 1, }
+			},
+			{ .retval = 0,
+			  .data64 = { (UINT_MAX - 1) | 2ULL << 32, }
+			},
+		},
+	},
+	{
+		"jgt32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffff0 | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JGT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP_IMM(BPF_JGT, BPF_REG_7, 0x7ffffff0, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
+	{
+		"jle32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JLE, BPF_REG_7, INT_MAX, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { INT_MAX - 1, }
+			},
+			{ .retval = 0,
+			  .data64 = { UINT_MAX, }
+			},
+			{ .retval = 2,
+			  .data64 = { INT_MAX, }
+			},
+		},
+	},
+	{
+		"jle32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, (INT_MAX - 1) | 2ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JLE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { INT_MAX | 1ULL << 32, }
+			},
+			{ .retval = 2,
+			  .data64 = { INT_MAX - 2, }
+			},
+			{ .retval = 0,
+			  .data64 = { UINT_MAX, }
+			},
+		},
+	},
+	{
+		"jle32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffff0 | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JLE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP32_IMM(BPF_JLE, BPF_REG_7, 0x7ffffff0, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
+	{
+		"jlt32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JLT, BPF_REG_7, INT_MAX, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { INT_MAX, }
+			},
+			{ .retval = 0,
+			  .data64 = { UINT_MAX, }
+			},
+			{ .retval = 2,
+			  .data64 = { INT_MAX - 1, }
+			},
+		},
+	},
+	{
+		"jlt32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, INT_MAX | 2ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JLT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { INT_MAX | 1ULL << 32, }
+			},
+			{ .retval = 0,
+			  .data64 = { UINT_MAX, }
+			},
+			{ .retval = 2,
+			  .data64 = { (INT_MAX - 1) | 3ULL << 32, }
+			},
+		},
+	},
+	{
+		"jlt32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffff0 | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JLT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0x7ffffff0, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
+	{
+		"jsge32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JSGE, BPF_REG_7, -1, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { 0, }
+			},
+			{ .retval = 2,
+			  .data64 = { -1, }
+			},
+			{ .retval = 0,
+			  .data64 = { -2, }
+			},
+		},
+	},
+	{
+		"jsge32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, (__u32)-1 | 2ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JSGE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { -1, }
+			},
+			{ .retval = 2,
+			  .data64 = { 0x7fffffff | 1ULL << 32, }
+			},
+			{ .retval = 0,
+			  .data64 = { -2, }
+			},
+		},
+	},
+	{
+		"jsge32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffff0 | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JSGE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP_IMM(BPF_JSGE, BPF_REG_7, 0x7ffffff0, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
+	{
+		"jsgt32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JSGT, BPF_REG_7, -1, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { (__u32)-2, }
+			},
+			{ .retval = 0,
+			  .data64 = { -1, }
+			},
+			{ .retval = 2,
+			  .data64 = { 1, }
+			},
+		},
+	},
+	{
+		"jsgt32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffffe | 1ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JSGT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 0,
+			  .data64 = { 0x7ffffffe, }
+			},
+			{ .retval = 0,
+			  .data64 = { 0x1ffffffffULL, }
+			},
+			{ .retval = 2,
+			  .data64 = { 0x7fffffff, }
+			},
+		},
+	},
+	{
+		"jsgt32: min/max deduction",
+		.insns = {
+			BPF_RAND_SEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, (__u32)(-2) | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JSGT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP_IMM(BPF_JSGT, BPF_REG_7, -2, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
+	{
+		"jsle32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JSLE, BPF_REG_7, -1, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { (__u32)-2, }
+			},
+			{ .retval = 2,
+			  .data64 = { -1, }
+			},
+			{ .retval = 0,
+			  .data64 = { 1, }
+			},
+		},
+	},
+	{
+		"jsle32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffffe | 1ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JSLE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { 0x7ffffffe, }
+			},
+			{ .retval = 2,
+			  .data64 = { (__u32)-1, }
+			},
+			{ .retval = 0,
+			  .data64 = { 0x7fffffff | 2ULL << 32, }
+			},
+		},
+	},
+	{
+		"jsle32: min/max deduction",
+		.insns = {
+			BPF_RAND_UEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, 0x7ffffff0 | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JSLE, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP_IMM(BPF_JSLE, BPF_REG_7, 0x7ffffff0, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
+	{
+		"jslt32: BPF_K",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_IMM(BPF_JSLT, BPF_REG_7, -1, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { (__u32)-2, }
+			},
+			{ .retval = 0,
+			  .data64 = { -1, }
+			},
+			{ .retval = 0,
+			  .data64 = { 1, }
+			},
+		},
+	},
+	{
+		"jslt32: BPF_X",
+		.insns = {
+			BPF_DIRECT_PKT_R2,
+			BPF_LD_IMM64(BPF_REG_8, 0x7fffffff | 1ULL << 32),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_7, BPF_REG_2, 0),
+			BPF_JMP32_REG(BPF_JSLT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_EXIT_INSN(),
+		},
+		.prog_type = BPF_PROG_TYPE_SCHED_CLS,
+		.result = ACCEPT,
+		.runs = 3,
+		.retvals = {
+			{ .retval = 2,
+			  .data64 = { 0x7ffffffe, }
+			},
+			{ .retval = 2,
+			  .data64 = { 0xffffffff, }
+			},
+			{ .retval = 0,
+			  .data64 = { 0x7fffffff | 2ULL << 32, }
+			},
+		},
+	},
+	{
+		"jslt32: min/max deduction",
+		.insns = {
+			BPF_RAND_SEXT_R7,
+			BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 2),
+			BPF_LD_IMM64(BPF_REG_8, (__u32)(-1) | 1ULL << 32),
+			BPF_JMP32_REG(BPF_JSLT, BPF_REG_7, BPF_REG_8, 1),
+			BPF_EXIT_INSN(),
+			BPF_JMP32_IMM(BPF_JSLT, BPF_REG_7, -1, 1),
+			BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0),
+			BPF_EXIT_INSN(),
+		},
+		.result = ACCEPT,
+		.retval = 2,
+	},
 };
 
 static int probe_filter_length(const struct bpf_insn *fp)