aboutsummaryrefslogtreecommitdiffstats
path: root/arch/mips/net (follow)
AgeCommit message (Collapse)AuthorFilesLines
2021-11-30mips, bpf: Fix reference to non-existing Kconfig symbolJohan Almbladh1-1/+1
The Kconfig symbol for R10000 ll/sc errata workaround in the MIPS JIT was misspelled, causing the workaround to not take effect when enabled. Fixes: 72570224bb8f ("mips, bpf: Add JIT workarounds for CPU errata") Reported-by: Lukas Bulwahn <lukas.bulwahn@gmail.com> Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20211130160824.3781635-1-johan.almbladh@anyfinetworks.com
2021-10-11bpf, mips: Fix comment on tail call count limitingTiezhu Yang1-1/+1
In emit_tail_call() of bpf_jit_comp32.c, "blez t2" (t2 <= 0) is not consistent with the comment "t2 < 0", update the comment to keep consistency. Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Link: https://lore.kernel.org/bpf/1633915150-13220-3-git-send-email-yangtiezhu@loongson.cn
2021-10-11bpf, mips: Clean up config options about JITTiezhu Yang1-3/+3
The config options MIPS_CBPF_JIT and MIPS_EBPF_JIT are useless, remove them in arch/mips/Kconfig, and then modify arch/mips/net/Makefile. Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Link: https://lore.kernel.org/bpf/1633915150-13220-2-git-send-email-yangtiezhu@loongson.cn
2021-10-07mips, bpf: Optimize loading of 64-bit constantsJohan Almbladh1-1/+7
This patch shaves off a few instructions when loading sparse 64-bit constants to register. The change is covered by additional tests in lib/test_bpf.c. Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20211007142828.634182-1-johan.almbladh@anyfinetworks.com
2021-10-07mips, bpf: Fix Makefile that referenced a removed fileJohan Almbladh1-1/+0
This patch removes a stale Makefile reference to the cBPF JIT that was removed. Fixes: ebcbacfa50ec ("mips, bpf: Remove old BPF JIT implementations") Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20211007142339.633899-1-johan.almbladh@anyfinetworks.com
2021-10-06mips, bpf: Remove old BPF JIT implementationsJohan Almbladh4-3603/+0
This patch removes the old 32-bit cBPF and 64-bit eBPF JIT implementations. They are replaced by a new eBPF implementation that supports both 32-bit and 64-bit MIPS CPUs. Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20211005165408.2305108-8-johan.almbladh@anyfinetworks.com
2021-10-06mips, bpf: Enable eBPF JITsJohan Almbladh1-2/+3
This patch enables the new eBPF JITs for 32-bit and 64-bit MIPS. It also disables the old cBPF JIT to so cBPF programs are converted to use the new JIT. Workarounds for R4000 CPU errata are not implemented by the JIT, so the JIT is disabled if any of those workarounds are configured. Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20211005165408.2305108-7-johan.almbladh@anyfinetworks.com
2021-10-06mips, bpf: Add JIT workarounds for CPU errataJohan Almbladh3-7/+35
This patch adds workarounds for the following CPU errata to the MIPS eBPF JIT, if enabled in the kernel configuration. - R10000 ll/sc weak ordering - Loongson-3 ll/sc weak ordering - Loongson-2F jump hang The Loongson-2F nop errata is implemented in uasm, which the JIT uses, so no additional mitigations are needed for that. Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com> Link: https://lore.kernel.org/bpf/20211005165408.2305108-6-johan.almbladh@anyfinetworks.com
2021-10-06mips, bpf: Add new eBPF JIT for 64-bit MIPSJohan Almbladh1-0/+1052
This is an implementation on of an eBPF JIT for 64-bit MIPS III-V and MIPS64r1-r6. It uses the same framework introduced by the 32-bit JIT. Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20211005165408.2305108-5-johan.almbladh@anyfinetworks.com
2021-10-06mips, bpf: Add eBPF JIT for 32-bit MIPSJohan Almbladh4-1/+3148
This is an implementation of an eBPF JIT for 32-bit MIPS I-V and MIPS32. The implementation supports all 32-bit and 64-bit ALU and JMP operations, including the recently-added atomics. 64-bit div/mod and 64-bit atomics are implemented using function calls to math64 and atomic64 functions, respectively. All 32-bit operations are implemented natively by the JIT, except if the CPU lacks ll/sc instructions. Register mapping ================ All 64-bit eBPF registers are mapped to native 32-bit MIPS register pairs, and does not use any stack scratch space for register swapping. This means that all eBPF register data is kept in CPU registers all the time, and this simplifies the register management a lot. It also reduces the JIT's pressure on temporary registers since we do not have to move data around. Native register pairs are ordered according to CPU endiannes, following the O32 calling convention for passing 64-bit arguments and return values. The eBPF return value, arguments and callee-saved registers are mapped to their native MIPS equivalents. Since the 32 highest bits in the eBPF FP (frame pointer) register are always zero, only one general-purpose register is actually needed for the mapping. The MIPS fp register is used for this purpose. The high bits are mapped to MIPS register r0. This saves us one CPU register, which is much needed for temporaries, while still allowing us to treat the R10 (FP) register just like any other eBPF register in the JIT. The MIPS gp (global pointer) and at (assembler temporary) registers are used as internal temporary registers for constant blinding. CPU registers t6-t9 are used internally by the JIT when constructing more complex 64-bit operations. This is precisely what is needed - two registers to store an operand value, and two more as scratch registers when performing the operation. The register mapping is shown below. R0 - $v1, $v0 return value R1 - $a1, $a0 argument 1, passed in registers R2 - $a3, $a2 argument 2, passed in registers R3 - $t1, $t0 argument 3, passed on stack R4 - $t3, $t2 argument 4, passed on stack R5 - $t4, $t3 argument 5, passed on stack R6 - $s1, $s0 callee-saved R7 - $s3, $s2 callee-saved R8 - $s5, $s4 callee-saved R9 - $s7, $s6 callee-saved FP - $r0, $fp 32-bit frame pointer AX - $gp, $at constant-blinding $t6 - $t9 unallocated, JIT temporaries Jump offsets ============ The JIT tries to map all conditional JMP operations to MIPS conditional PC-relative branches. The MIPS branch offset field is 18 bits, in bytes, which is equivalent to the eBPF 16-bit instruction offset. However, since the JIT may emit more than one CPU instruction per eBPF instruction, the field width may overflow. If that happens, the JIT converts the long conditional jump to a short PC-relative branch with the condition inverted, jumping over a long unconditional absolute jmp (j). This conversion will change the instruction offset mapping used for jumps, and may in turn result in more branch offset overflows. The JIT therefore dry-runs the translation until no more branches are converted and the offsets do not change anymore. There is an upper bound on this of course, and if the JIT hits that limit, the last two iterations are run with all branches being converted. Tail call count =============== The current tail call count is stored in the 16-byte area of the caller's stack frame that is reserved for the callee in the o32 ABI. The value is initialized in the prologue, and propagated to the tail-callee by skipping the initialization instructions when emitting the tail call. Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Andrii Nakryiko <andrii@kernel.org> Link: https://lore.kernel.org/bpf/20211005165408.2305108-4-johan.almbladh@anyfinetworks.com
2021-09-15bpf, mips: Validate conditional branch offsetsPiotr Krysiuk1-14/+43
The conditional branch instructions on MIPS use 18-bit signed offsets allowing for a branch range of 128 KBytes (backward and forward). However, this limit is not observed by the cBPF JIT compiler, and so the JIT compiler emits out-of-range branches when translating certain cBPF programs. A specific example of such a cBPF program is included in the "BPF_MAXINSNS: exec all MSH" test from lib/test_bpf.c that executes anomalous machine code containing incorrect branch offsets under JIT. Furthermore, this issue can be abused to craft undesirable machine code, where the control flow is hijacked to execute arbitrary Kernel code. The following steps can be used to reproduce the issue: # echo 1 > /proc/sys/net/core/bpf_jit_enable # modprobe test_bpf test_name="BPF_MAXINSNS: exec all MSH" This should produce multiple warnings from build_bimm() similar to: ------------[ cut here ]------------ WARNING: CPU: 0 PID: 209 at arch/mips/mm/uasm-mips.c:210 build_insn+0x558/0x590 Micro-assembler field overflow Modules linked in: test_bpf(+) CPU: 0 PID: 209 Comm: modprobe Not tainted 5.14.3 #1 Stack : 00000000 807bb824 82b33c9c 801843c0 00000000 00000004 00000000 63c9b5ee 82b33af4 80999898 80910000 80900000 82fd6030 00000001 82b33a98 82087180 00000000 00000000 80873b28 00000000 000000fc 82b3394c 00000000 2e34312e 6d6d6f43 809a180f 809a1836 6f6d203a 80900000 00000001 82b33bac 80900000 00027f80 00000000 00000000 807bb824 00000000 804ed790 001cc317 00000001 [...] Call Trace: [<80108f44>] show_stack+0x38/0x118 [<807a7aac>] dump_stack_lvl+0x5c/0x7c [<807a4b3c>] __warn+0xcc/0x140 [<807a4c3c>] warn_slowpath_fmt+0x8c/0xb8 [<8011e198>] build_insn+0x558/0x590 [<8011e358>] uasm_i_bne+0x20/0x2c [<80127b48>] build_body+0xa58/0x2a94 [<80129c98>] bpf_jit_compile+0x114/0x1e4 [<80613fc4>] bpf_prepare_filter+0x2ec/0x4e4 [<8061423c>] bpf_prog_create+0x80/0xc4 [<c0a006e4>] test_bpf_init+0x300/0xba8 [test_bpf] [<8010051c>] do_one_initcall+0x50/0x1d4 [<801c5e54>] do_init_module+0x60/0x220 [<801c8b20>] sys_finit_module+0xc4/0xfc [<801144d0>] syscall_common+0x34/0x58 [...] ---[ end trace a287d9742503c645 ]--- Then the anomalous machine code executes: => 0xc0a18000: addiu sp,sp,-16 0xc0a18004: sw s3,0(sp) 0xc0a18008: sw s4,4(sp) 0xc0a1800c: sw s5,8(sp) 0xc0a18010: sw ra,12(sp) 0xc0a18014: move s5,a0 0xc0a18018: move s4,zero 0xc0a1801c: move s3,zero # __BPF_STMT(BPF_LDX | BPF_B | BPF_MSH, 0) 0xc0a18020: lui t6,0x8012 0xc0a18024: ori t4,t6,0x9e14 0xc0a18028: li a1,0 0xc0a1802c: jalr t4 0xc0a18030: move a0,s5 0xc0a18034: bnez v0,0xc0a1ffb8 # incorrect branch offset 0xc0a18038: move v0,zero 0xc0a1803c: andi s4,s3,0xf 0xc0a18040: b 0xc0a18048 0xc0a18044: sll s4,s4,0x2 [...] # __BPF_STMT(BPF_LDX | BPF_B | BPF_MSH, 0) 0xc0a1ffa0: lui t6,0x8012 0xc0a1ffa4: ori t4,t6,0x9e14 0xc0a1ffa8: li a1,0 0xc0a1ffac: jalr t4 0xc0a1ffb0: move a0,s5 0xc0a1ffb4: bnez v0,0xc0a1ffb8 # incorrect branch offset 0xc0a1ffb8: move v0,zero 0xc0a1ffbc: andi s4,s3,0xf 0xc0a1ffc0: b 0xc0a1ffc8 0xc0a1ffc4: sll s4,s4,0x2 # __BPF_STMT(BPF_LDX | BPF_B | BPF_MSH, 0) 0xc0a1ffc8: lui t6,0x8012 0xc0a1ffcc: ori t4,t6,0x9e14 0xc0a1ffd0: li a1,0 0xc0a1ffd4: jalr t4 0xc0a1ffd8: move a0,s5 0xc0a1ffdc: bnez v0,0xc0a3ffb8 # correct branch offset 0xc0a1ffe0: move v0,zero 0xc0a1ffe4: andi s4,s3,0xf 0xc0a1ffe8: b 0xc0a1fff0 0xc0a1ffec: sll s4,s4,0x2 [...] # epilogue 0xc0a3ffb8: lw s3,0(sp) 0xc0a3ffbc: lw s4,4(sp) 0xc0a3ffc0: lw s5,8(sp) 0xc0a3ffc4: lw ra,12(sp) 0xc0a3ffc8: addiu sp,sp,16 0xc0a3ffcc: jr ra 0xc0a3ffd0: nop To mitigate this issue, we assert the branch ranges for each emit call that could generate an out-of-range branch. Fixes: 36366e367ee9 ("MIPS: BPF: Restore MIPS32 cBPF JIT") Fixes: c6610de353da ("MIPS: net: Add BPF JIT") Signed-off-by: Piotr Krysiuk <piotras@gmail.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Tested-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Acked-by: Johan Almbladh <johan.almbladh@anyfinetworks.com> Cc: Paul Burton <paulburton@kernel.org> Cc: Thomas Bogendoerfer <tsbogend@alpha.franken.de> Link: https://lore.kernel.org/bpf/20210915160437.4080-1-piotras@gmail.com
2021-07-29bpf: Introduce BPF nospec instruction for mitigating Spectre v4Daniel Borkmann1-0/+3
In case of JITs, each of the JIT backends compiles the BPF nospec instruction /either/ to a machine instruction which emits a speculation barrier /or/ to /no/ machine instruction in case the underlying architecture is not affected by Speculative Store Bypass or has different mitigations in place already. This covers both x86 and (implicitly) arm64: In case of x86, we use 'lfence' instruction for mitigation. In case of arm64, we rely on the firmware mitigation as controlled via the ssbd kernel parameter. Whenever the mitigation is enabled, it works for all of the kernel code with no need to provide any additional instructions here (hence only comment in arm64 JIT). Other archs can follow as needed. The BPF nospec instruction is specifically targeting Spectre v4 since i) we don't use a serialization barrier for the Spectre v1 case, and ii) mitigation instructions for v1 and v4 might be different on some archs. The BPF nospec is required for a future commit, where the BPF verifier does annotate intermediate BPF programs with speculation barriers. Co-developed-by: Piotr Krysiuk <piotras@gmail.com> Co-developed-by: Benedict Schlueter <benedict.schlueter@rub.de> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Piotr Krysiuk <piotras@gmail.com> Signed-off-by: Benedict Schlueter <benedict.schlueter@rub.de> Acked-by: Alexei Starovoitov <ast@kernel.org>
2021-01-14bpf: Rename BPF_XADD and prepare to encode other atomics in .immBrendan Jackman1-3/+8
A subsequent patch will add additional atomic operations. These new operations will use the same opcode field as the existing XADD, with the immediate discriminating different operations. In preparation, rename the instruction mode BPF_ATOMIC and start calling the zero immediate BPF_ADD. This is possible (doesn't break existing valid BPF progs) because the immediate field is currently reserved MBZ and BPF_ADD is zero. All uses are removed from the tree but the BPF_XADD definition is kept around to avoid breaking builds for people including kernel headers. Signed-off-by: Brendan Jackman <jackmanb@google.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Björn Töpel <bjorn.topel@gmail.com> Link: https://lore.kernel.org/bpf/20210114181751.768687-5-jackmanb@google.com
2020-01-10MIPS: BPF: Use sizeof_field() instead of FIELD_SIZEOF()Kees Cook1-9/+9
The FIELD_SIZEOF() macro was redundant, and is being removed from the kernel. Since commit c593642c8be0 ("treewide: Use sizeof_field() macro") this is one of the last users of the old macro, so replace it. Signed-off-by: Kees Cook <keescook@chromium.org> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: linux-kernel@vger.kernel.org Cc: linux-mips@vger.kernel.org
2020-01-09MIPS: BPF: Restore MIPS32 cBPF JITPaul Burton3-0/+1556
Commit 716850ab104d ("MIPS: eBPF: Initial eBPF support for MIPS32 architecture.") enabled our eBPF JIT for MIPS32 kernels, whereas it has previously only been availailable for MIPS64. It was my understanding at the time that the BPF test suite was passing & JITing a comparable number of tests to our cBPF JIT [1], but it turns out that was not the case. The eBPF JIT has a number of problems on MIPS32: - Most notably various code paths still result in emission of MIPS64 instructions which will cause reserved instruction exceptions & kernel panics when run on MIPS32 CPUs. - The eBPF JIT doesn't account for differences between the O32 ABI used by MIPS32 kernels versus the N64 ABI used by MIPS64 kernels. Notably arguments beyond the first 4 are passed on the stack in O32, and this is entirely unhandled when JITing a BPF_CALL instruction. Stack space must be reserved for arguments even if they all fit in registers, and the callee is free to assume that stack space has been reserved for its use - with the eBPF JIT this is not the case, so calling any function can result in clobbering values on the stack & unpredictable behaviour. Function arguments in eBPF are always 64-bit values which is also entirely unhandled - the JIT still uses a single (32-bit) register per argument. As a result all function arguments are always passed incorrectly when JITing a BPF_CALL instruction, leading to kernel crashes or strange behavior. - The JIT attempts to bail our on use of ALU64 instructions or 64-bit memory access instructions. The code doing this at the start of build_one_insn() incorrectly checks whether BPF_OP() equals BPF_DW, when it should really be checking BPF_SIZE() & only doing so when BPF_CLASS() is one of BPF_{LD,LDX,ST,STX}. This results in false positives that cause more bailouts than intended, and that in turns hides some of the problems described above. - The kernel's cBPF->eBPF translation makes heavy use of 64-bit eBPF instructions that the MIPS32 eBPF JIT bails out on, leading to most cBPF programs not being JITed at all. Until these problems are resolved, revert the removal of the cBPF JIT performed by commit 716850ab104d ("MIPS: eBPF: Initial eBPF support for MIPS32 architecture."). Together with commit f8fffebdea75 ("MIPS: BPF: Disable MIPS32 eBPF JIT") this restores MIPS32 BPF JIT behavior back to the same state it was prior to the introduction of the broken eBPF JIT support. [1] https://lore.kernel.org/linux-mips/MWHPR2201MB13583388481F01A422CE7D66D4410@MWHPR2201MB1358.namprd22.prod.outlook.com/ Signed-off-by: Paul Burton <paulburton@kernel.org> Fixes: 716850ab104d ("MIPS: eBPF: Initial eBPF support for MIPS32 architecture.") Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Hassan Naveed <hnaveed@wavecomp.com> Cc: Tony Ambardar <itugrok@yahoo.com> Cc: bpf@vger.kernel.org Cc: netdev@vger.kernel.org Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org
2020-01-04Merge tag 'mips_fixes_5.5_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linuxLinus Torvalds1-1/+1
Pull MIPS fixes from Paul Burton: "A collection of MIPS fixes: - Fill the struct cacheinfo shared_cpu_map field with sensible values, notably avoiding issues with perf which was unhappy in the absence of these values. - A boot fix for Loongson 2E & 2F machines which was fallout from some refactoring performed this cycle. - A Kconfig dependency fix for the Loongson CPU HWMon driver. - A couple of VDSO fixes, ensuring gettimeofday() behaves appropriately for kernel configurations that don't include support for a clocksource the VDSO can use & fixing the calling convention for the n32 & n64 VDSOs which would previously clobber the $gp/$28 register. - A build fix for vmlinuz compressed images which were inappropriately building with -fsanitize-coverage despite not being part of the kernel proper, then failing to link due to the missing __sanitizer_cov_trace_pc() function. - A couple of eBPF JIT fixes, including disabling it for MIPS32 due to a large number of issues with the code generated there & reflecting ISA dependencies in Kconfig to enforce that systems which don't support the JIT must include the interpreter" * tag 'mips_fixes_5.5_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux: MIPS: Avoid VDSO ABI breakage due to global register variable MIPS: BPF: eBPF JIT: check for MIPS ISA compliance in Kconfig MIPS: BPF: Disable MIPS32 eBPF JIT MIPS: Prevent link failure with kcov instrumentation MIPS: Kconfig: Use correct form for 'depends on' mips: Fix gettimeofday() in the vdso library MIPS: Fix boot on Fuloong2 systems mips: cacheinfo: report shared CPU map
2019-12-18MIPS: BPF: eBPF JIT: check for MIPS ISA compliance in KconfigAlexander Lobakin1-1/+1
It is completely wrong to check for compile-time MIPS ISA revision in the body of bpf_int_jit_compile() as it may lead to get MIPS JIT fully omitted by the CC while the rest system will think that the JIT is actually present and works [1]. We can check if the selected CPU really supports MIPS eBPF JIT at configure time and avoid such situations when kernel can be built without both JIT and interpreter, but with CONFIG_BPF_SYSCALL=y. [1] https://lore.kernel.org/linux-mips/09d713a59665d745e21d021deeaebe0a@dlink.ru/ Fixes: 716850ab104d ("MIPS: eBPF: Initial eBPF support for MIPS32 architecture.") Cc: <stable@vger.kernel.org> # v5.2+ Signed-off-by: Alexander Lobakin <alobakin@dlink.ru> Signed-off-by: Paul Burton <paulburton@kernel.org> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: James Hogan <jhogan@kernel.org> Cc: Hassan Naveed <hnaveed@wavecomp.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Martin KaFai Lau <kafai@fb.com> Cc: Song Liu <songliubraving@fb.com> Cc: Yonghong Song <yhs@fb.com> Cc: Andrii Nakryiko <andriin@fb.com> Cc: linux-mips@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: netdev@vger.kernel.org Cc: bpf@vger.kernel.org
2019-12-11bpf, mips: Limit to 33 tail callsPaul Chaignon1-4/+5
All BPF JIT compilers except RISC-V's and MIPS' enforce a 33-tail calls limit at runtime. In addition, a test was recently added, in tailcalls2, to check this limit. This patch updates the tail call limit in MIPS' JIT compiler to allow 33 tail calls. Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Reported-by: Mahshid Khezri <khezri.mahshid@gmail.com> Signed-off-by: Paul Chaignon <paul.chaignon@orange.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Martin KaFai Lau <kafai@fb.com> Link: https://lore.kernel.org/bpf/b8eb2caac1c25453c539248e56ca22f74b5316af.1575916815.git.paul.chaignon@gmail.com
2019-06-05treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 441Thomas Gleixner2-8/+2
Based on 1 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license as published by the free software foundation version 2 of the license extracted by the scancode license scanner the SPDX license identifier GPL-2.0-only has been chosen to replace the boilerplate/reference in 315 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Allison Randal <allison@lohutok.net> Reviewed-by: Armijn Hemel <armijn@tjaldur.nl> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190531190115.503150771@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-05-21treewide: Add SPDX license identifier - Makefile/KconfigThomas Gleixner1-0/+1
Add SPDX license identifiers to all Make/Kconfig files which: - Have no license information of any form These files fall under the project license, GPL v2 only. The resulting SPDX license identifier is: GPL-2.0-only Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-05-08Merge tag 'mips_5.2' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linuxLinus Torvalds4-1609/+138
Pull MIPS updates from Paul Burton: - A set of memblock initialization improvements thanks to Serge Semin, tidying up after our conversion from bootmem to memblock back in v4.20. - Our eBPF JIT the previously supported only MIPS64r2 through MIPS64r5 is improved to also support MIPS64r6. Support for MIPS32 systems is introduced, with the caveat that it only works for programs that don't use 64 bit registers or operations - those will bail out & need to be interpreted. - Improvements to the allocation & configuration of our exception vector that should fix issues seen on some platforms using recent versions of U-Boot. - Some minor improvements to code generated for jump labels, along with enabling them by default for generic kernels. * tag 'mips_5.2' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux: (27 commits) mips: Manually call fdt_init_reserved_mem() method mips: Make sure dt memory regions are valid mips: Perform early low memory test mips: Dump memblock regions for debugging mips: Add reserve-nomap memory type support mips: Use memblock to reserve the __nosave memory range mips: Discard post-CMA-init foreach loop mips: Reserve memory for the kernel image resources MIPS: Remove duplicate EBase configuration MIPS: Sync icache for whole exception vector MIPS: Always allocate exception vector for MIPSr2+ MIPS: Use memblock_phys_alloc() for exception vector mips: Combine memblock init and memory reservation loops mips: Discard rudiments from bootmem_init mips: Make sure kernel .bss exists in boot mem pool mips: vdso: drop unnecessary cc-ldoption Revert "MIPS: ralink: fix cpu clock of mt7621 and add dt clk devices" MIPS: generic: Enable CONFIG_JUMP_LABEL MIPS: jump_label: Use compact branches for >= r6 MIPS: jump_label: Remove redundant nops ...
2019-04-25MIPS: eBPF: Make ebpf_to_mips_reg() staticYueHaibing1-2/+3
Fix sparse warning: arch/mips/net/ebpf_jit.c:196:5: warning: symbol 'ebpf_to_mips_reg' was not declared. Should it be static? Reported-by: Hulk Robot <hulkci@huawei.com> Signed-off-by: YueHaibing <yuehaibing@huawei.com> Acked-by: Yonghong Song <yhs@fb.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2019-03-19MIPS: eBPF: Initial eBPF support for MIPS32 architecture.Hassan Naveed4-1600/+69
Currently MIPS32 supports a JIT for classic BPF only, not extended BPF. This patch adds JIT support for extended BPF on MIPS32, so code is actually JIT'ed instead of being only interpreted. Instructions with 64-bit operands are not supported at this point. We can delete classic BPF because the kernel will translate classic BPF programs into extended BPF and JIT them, eliminating the need for classic BPF. Signed-off-by: Hassan Naveed <hnaveed@wavecomp.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: kafai@fb.com Cc: songliubraving@fb.com Cc: yhs@fb.com Cc: netdev@vger.kernel.org Cc: bpf@vger.kernel.org Cc: linux-mips@vger.kernel.org Cc: Ralf Baechle <ralf@linux-mips.org> Cc: James Hogan <jhogan@kernel.org> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: open list:MIPS <linux-mips@linux-mips.org> Cc: open list <linux-kernel@vger.kernel.org>
2019-03-19MIPS: eBPF: Provide eBPF support for MIPS64R6Hassan Naveed1-9/+69
Currently eBPF support is available on MIPS64R2 only. Use MIPS64R6 variants of instructions like multiply, divide, movn, movz so eBPF can run on the newer ISA. Also, we only need to check ISA revision before JIT'ing code, because we know the CPU is running a 64-bit kernel because eBPF JIT is only included in kernels with CONFIG_64BIT=y due to Kconfig dependencies. Signed-off-by: Hassan Naveed <hnaveed@wavecomp.com> Signed-off-by: Paul Burton <paul.burton@mips.com> Cc: kafai@fb.com Cc: songliubraving@fb.com Cc: yhs@fb.com Cc: netdev@vger.kernel.org Cc: bpf@vger.kernel.org Cc: linux-mips@vger.kernel.org Cc: Ralf Baechle <ralf@linux-mips.org> Cc: James Hogan <jhogan@kernel.org> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: open list:MIPS <linux-mips@linux-mips.org> Cc: open list <linux-kernel@vger.kernel.org>
2019-03-02MIPS: eBPF: Fix icache flush end addressPaul Burton1-1/+1
The MIPS eBPF JIT calls flush_icache_range() in order to ensure the icache observes the code that we just wrote. Unfortunately it gets the end address calculation wrong due to some bad pointer arithmetic. The struct jit_ctx target field is of type pointer to u32, and as such adding one to it will increment the address being pointed to by 4 bytes. Therefore in order to find the address of the end of the code we simply need to add the number of 4 byte instructions emitted, but we mistakenly add the number of instructions multiplied by 4. This results in the call to flush_icache_range() operating on a memory region 4x larger than intended, which is always wasteful and can cause crashes if we overrun into an unmapped page. Fix this by correcting the pointer arithmetic to remove the bogus multiplication, and use braces to remove the need for a set of brackets whilst also making it obvious that the target field is a pointer. Signed-off-by: Paul Burton <paul.burton@mips.com> Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Martin KaFai Lau <kafai@fb.com> Cc: Song Liu <songliubraving@fb.com> Cc: Yonghong Song <yhs@fb.com> Cc: netdev@vger.kernel.org Cc: bpf@vger.kernel.org Cc: linux-mips@vger.kernel.org Cc: stable@vger.kernel.org # v4.13+ Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2019-02-16MIPS: eBPF: Remove REG_32BIT_ZERO_EXPaul Burton1-10/+7
REG_32BIT_ZERO_EX and REG_64BIT are always handled in exactly the same way, and reg_val_propagate_range() never actually sets any register to type REG_32BIT_ZERO_EX. Remove the redundant & unused REG_32BIT_ZERO_EX. Signed-off-by: Paul Burton <paul.burton@mips.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2019-02-16MIPS: eBPF: Always return sign extended 32b valuesPaul Burton1-3/+6
The function prototype used to call JITed eBPF code (ie. the type of the struct bpf_prog bpf_func field) returns an unsigned int. The MIPS n64 ABI that MIPS64 kernels target defines that 32 bit integers should always be sign extended when passed in registers as either arguments or return values. This means that when returning any value which may not already be sign extended (ie. of type REG_64BIT or REG_32BIT_ZERO_EX) we need to perform that sign extension in order to comply with the n64 ABI. Without this we see strange looking test failures from test_bpf.ko, such as: test_bpf: #65 ALU64_MOV_X: dst = 4294967295 jited:1 ret -1 != -1 FAIL (1 times) Although the return value printed matches the expected value, this is only because printf is only examining the least significant 32 bits of the 64 bit register value we returned. The register holding the expected value is sign extended whilst the v0 register was set to a zero extended value by our JITed code, so when compared by a conditional branch instruction the values are not equal. We already handle this when the return value register is of type REG_32BIT_ZERO_EX, so simply extend this to also cover REG_64BIT. Signed-off-by: Paul Burton <paul.burton@mips.com> Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Cc: stable@vger.kernel.org # v4.13+ Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2018-12-07mips: bpf: implement jitting of BPF_ALU | BPF_ARSH | BPF_XJiong Wang1-0/+4
Jitting of BPF_K is supported already, but not BPF_X. This patch complete the support for the latter on both MIPS and microMIPS. Cc: Paul Burton <paul.burton@mips.com> Cc: linux-mips@vger.kernel.org Acked-by: Paul Burton <paul.burton@mips.com> Signed-off-by: Jiong Wang <jiong.wang@netronome.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-11-16net: remove VLAN_TAG_PRESENTMichał Mirosław1-3/+0
Replace VLAN_TAG_PRESENT with single bit flag and free up VLAN.CFI overload. Now VLAN.CFI is visible in networking stack and can be passed around intact. Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl> Signed-off-by: David S. Miller <davem@davemloft.net>
2018-11-16net/bpf_jit: MIPS: split VLAN_PRESENT bit handling from VLAN_TCIMichał Mirosław1-9/+12
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl> Signed-off-by: David S. Miller <davem@davemloft.net>
2018-05-14bpf, mips: remove unused functionDaniel Borkmann1-26/+0
The ool_skb_header_pointer() and size_to_len() is unused same as tmp_offset, therefore remove all of them. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-05-03bpf, mips64: remove ld_abs/ld_indDaniel Borkmann1-104/+0
Since LD_ABS/LD_IND instructions are now removed from the core and reimplemented through a combination of inlined BPF instructions and a slow-path helper, we can get rid of the complexity from mips64 JIT. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-03-09MIPS: BPF: Replace __mips_isa_rev with MIPS_ISA_REVMatt Redfearn1-4/+5
Remove the need to check that __mips_isa_rev is defined by using the newly added MIPS_ISA_REV. Signed-off-by: Matt Redfearn <matt.redfearn@mips.com> Cc: Ralf Baechle <ralf@linux-mips.org> Cc: Paul Burton <paul.burton@mips.com> Cc: David Daney <david.daney@cavium.com> Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/18677/ Signed-off-by: James Hogan <jhogan@kernel.org>
2018-01-26bpf, mips64: remove unneeded zero check from div/mod with kDaniel Borkmann1-15/+4
The verifier in both cBPF and eBPF reject div/mod by 0 imm, so this can never load. Remove emitting such test and reject it from being JITed instead (the latter is actually also not needed, but given practice in sparc64, ppc64 today, so doesn't hurt to add it here either). Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Cc: David Daney <david.daney@cavium.com> Reviewed-by: David Daney <david.daney@cavium.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-01-26bpf, mips64: remove obsolete exception handling from div/modDaniel Borkmann1-10/+0
Since we've changed div/mod exception handling for src_reg in eBPF verifier itself, remove the leftovers from mips64 JIT. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Cc: David Daney <david.daney@cavium.com> Reviewed-by: David Daney <david.daney@cavium.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2018-01-19bpf: get rid of pure_initcall dependency to enable jitsDaniel Borkmann2-4/+0
Having a pure_initcall() callback just to permanently enable BPF JITs under CONFIG_BPF_JIT_ALWAYS_ON is unnecessary and could leave a small race window in future where JIT is still disabled on boot. Since we know about the setting at compilation time anyway, just initialize it properly there. Also consolidate all the individual bpf_jit_enable variables into a single one and move them under one location. Moreover, don't allow for setting unspecified garbage values on them. Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Acked-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Alexei Starovoitov <ast@kernel.org>
2017-12-17bpf: fix net.core.bpf_jit_enable raceAlexei Starovoitov1-1/+1
global bpf_jit_enable variable is tested multiple times in JITs, blinding and verifier core. The malicious root can try to toggle it while loading the programs. This race condition was accounted for and there should be no issues, but it's safer to avoid this race condition. Signed-off-by: Alexei Starovoitov <ast@kernel.org> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
2017-11-01MIPS: bpf: Fix a typo in build_one_insn()Wei Yongjun1-1/+1
Fix a typo in build_one_insn(). Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Signed-off-by: Wei Yongjun <weiyongjun1@huawei.com> Cc: <stable@vger.kernel.org> # 4.13+ Patchwork: https://patchwork.linux-mips.org/patch/17491/ Signed-off-by: James Hogan <jhogan@kernel.org>
2017-10-09MIPS: bpf: Fix uninitialised target compiler errorMatt Redfearn1-1/+1
Compiling ebpf_jit.c with gcc 4.9 results in a (likely spurious) compiler warning, as gcc has detected that the variable "target" may be used uninitialised. Since -Werror is active, this is treated as an error and causes a kernel build failure whenever CONFIG_MIPS_EBPF_JIT is enabled. arch/mips/net/ebpf_jit.c: In function 'build_one_insn': arch/mips/net/ebpf_jit.c:1118:80: error: 'target' may be used uninitialized in this function [-Werror=maybe-uninitialized] emit_instr(ctx, j, target); ^ cc1: all warnings being treated as errors Fix this by initialising "target" to 0. If it really is used uninitialised this would result in a jump to 0 and a detectable run time failure. Signed-off-by: Matt Redfearn <matt.redfearn@imgtec.com> Fixes: b6bd53f9c4e8 ("MIPS: Add missing file for eBPF JIT.") Cc: James Hogan <james.hogan@imgtec.com> Cc: David Daney <david.daney@cavium.com> Cc: David S. Miller <davem@davemloft.net> Cc: Colin Ian King <colin.king@canonical.com> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: linux-mips@linux-mips.org Cc: linux-kernel@vger.kernel.org Cc: <stable@vger.kernel.org> # v4.13+ Patchwork: https://patchwork.linux-mips.org/patch/17375/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2017-08-22MIPS,bpf: fix missing break in switch statementColin Ian King1-0/+1
There is a missing break causing a fall-through and setting ctx.use_bbit_insns to the wrong value. Fix this by adding the missing break. Detected with cppcheck: "Variable 'ctx.use_bbit_insns' is reassigned a value before the old one has been used. 'break;' missing?" Fixes: 8d8d18c3283f ("MIPS,bpf: Fix using smp_processor_id() in preemptible splat.") Signed-off-by: Colin Ian King <colin.king@canonical.com> Acked-by: David Daney <david.daney@cavium.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-21MIPS,bpf: Cache value of BPF_OP(insn->code) in eBPF JIT.David Daney1-33/+34
The code looks a little cleaner if we replace BPF_OP(insn->code) with the local variable bpf_op. Caching the value this way also saves 300 bytes (about 1%) in the code size of the JIT. Signed-off-by: David Daney <david.daney@cavium.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-21MIPS, bpf: Implement JLT, JLE, JSLT and JSLE ops in the eBPF JIT.David Daney1-29/+72
Signed-off-by: David Daney <david.daney@cavium.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-21MIPS,bpf: Fix using smp_processor_id() in preemptible splat.David Daney1-14/+14
If the kernel is configured with preemption enabled we were getting warning stack traces for use of current_cpu_type(). Fix by moving the test between preempt_disable()/preempt_enable() and caching the results of the CPU type tests for use during code generation. Signed-off-by: David Daney <david.daney@cavium.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-04MIPS: Add missing file for eBPF JIT.David Daney1-0/+1950
Inexplicably, commit f381bf6d82f0 ("MIPS: Add support for eBPF JIT.") lost a file somewhere on its path to Linus' tree. Add back the missing ebpf_jit.c so that we can build with CONFIG_BPF_JIT selected. This version of ebpf_jit.c is identical to the original except for two minor change need to resolve conflicts with changes merged from the BPF branch: A) Set prog->jited_len = image_size; B) Use BPF_TAIL_CALL instead of BPF_CALL | BPF_X Fixes: f381bf6d82f0 ("MIPS: Add support for eBPF JIT.") Signed-off-by: David Daney <david.daney@cavium.com> Acked-by: Daniel Borkmann <daniel@iogearbox.net> Signed-off-by: David S. Miller <davem@davemloft.net>
2017-06-28MIPS: Add support for eBPF JIT.David Daney1-1/+2
Since the eBPF machine has 64-bit registers, we only support this in 64-bit kernels. As of the writing of this commit log test-bpf is showing: test_bpf: Summary: 316 PASSED, 0 FAILED, [308/308 JIT'ed] All current test cases are successfully compiled. Many examples in samples/bpf are usable, specifically tracex5 which uses tail calls works. Signed-off-by: David Daney <david.daney@cavium.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Daniel Borkmann <daniel@iogearbox.net> Cc: Markos Chandras <markos.chandras@imgtec.com> Cc: Matt Redfearn <matt.redfearn@imgtec.com> Cc: netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: linux-mips@linux-mips.org Patchwork: https://patchwork.linux-mips.org/patch/16369/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2017-04-10MIPS: BPF: Fix multiple problems in JIT skb access helpers.David Daney1-11/+12
o Socket data is unsigned, so use unsigned accessors instructions. o Fix path result pointer generation arithmetic. o Fix half-word byte swapping code for unsigned semantics. Signed-off-by: David Daney <david.daney@cavium.com> Cc: James Hogan <james.hogan@imgtec.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Steven J. Hill <steven.hill@cavium.com> Cc: linux-mips@linux-mips.org Cc: netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/15747/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2017-04-10MIPS: BPF: Quit clobbering callee saved registers in JIT code.David Daney1-4/+12
If bpf_needs_clear_a() returns true, only actually clear it if it is ever used. If it is not used, we don't save and restore it, so the clearing has the nasty side effect of clobbering caller state. Also, don't emit stack pointer adjustment instructions if the adjustment amount is zero. Signed-off-by: David Daney <david.daney@cavium.com> Cc: James Hogan <james.hogan@imgtec.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Steven J. Hill <steven.hill@cavium.com> Cc: linux-mips@linux-mips.org Cc: netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/15745/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2017-04-10MIPS: BPF: Use unsigned access for unsigned SKB fields.David Daney1-2/+2
The SKB vlan_tci and queue_mapping fields are unsigned, don't sign extend these in the BPF JIT. In the vlan_tci case, the value gets masked so the change is not needed for correctness, but do it anyway for agreement with the types defined in struct sk_buff. Signed-off-by: David Daney <david.daney@cavium.com> Cc: James Hogan <james.hogan@imgtec.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Steven J. Hill <steven.hill@cavium.com> Cc: linux-mips@linux-mips.org Cc: netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/15746/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2017-04-10MIPS: BPF: Add JIT support for SKF_AD_HATYPE.David Daney1-4/+17
This let's us pass some additional "modprobe test-bpf" tests with JIT enabled. Reuse the code for SKF_AD_IFINDEX, but substitute the offset and size of the "type" field. Signed-off-by: David Daney <david.daney@cavium.com> Cc: James Hogan <james.hogan@imgtec.com> Cc: Alexei Starovoitov <ast@kernel.org> Cc: Steven J. Hill <steven.hill@cavium.com> Cc: linux-mips@linux-mips.org Cc: netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org Patchwork: https://patchwork.linux-mips.org/patch/15744/ Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
2016-10-11treewide: remove redundant #include <linux/kconfig.h>Masahiro Yamada1-1/+0
Kernel source files need not include <linux/kconfig.h> explicitly because the top Makefile forces to include it with: -include $(srctree)/include/linux/kconfig.h This commit removes explicit includes except the following: * arch/s390/include/asm/facilities_src.h * tools/testing/radix-tree/linux/kernel.h These two are used for host programs. Link: http://lkml.kernel.org/r/1473656164-11929-1-git-send-email-yamada.masahiro@socionext.com Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Copyright © 1996 – 2023 Jason A. Donenfeld. All Rights Reverse Engineered.