aboutsummaryrefslogtreecommitdiffstats
path: root/arch/arm64/include
AgeCommit message (Collapse)AuthorFilesLines
2015-10-21arm64: Refactor check_cpu_capabilitiesSuzuki K. Poulose1-1/+1
check_cpu_capabilities runs through a given list of caps and checks if the system has the cap, updates the system capability bitmap and also runs any enable() methods associated with them. All of this is not quite obvious from the name 'check'. This patch splits the check_cpu_capabilities into two parts : 1) update_cpu_capabilities => Runs through the given list and updates the system wide capability map. 2) enable_cpu_capabilities => Runs through the given list and invokes enable() (if any) for the caps enabled on the system. Cc: Andre Przywara <andre.przywara@arm.com> Cc: Will Deacon <will.deacon@arm.com> Suggested-by: Catalin Marinas <catalin.marinsa@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-21arm64: Cleanup mixed endian support detectionSuzuki K. Poulose1-2/+10
Make use of the system wide safe register to decide the support for mixed endian. Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-21arm64: Read system wide CPUID valueSuzuki K. Poulose1-0/+2
Add an API for reading the safe CPUID value across the system from the new infrastructure. Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-21arm64: Consolidate CPU Sanity check to CPU Feature infrastructureSuzuki K. Poulose1-1/+2
This patch consolidates the CPU Sanity check to the new infrastructure. Cc: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-21arm64: Keep track of CPU feature registersSuzuki K. Poulose3-5/+169
This patch adds an infrastructure to keep track of the CPU feature registers on the system. For each register, the infrastructure keeps track of the system wide safe value of the feature bits. Also, tracks the which fields of a register should be matched strictly across all the CPUs on the system for the SANITY check infrastructure. The feature bits are classified into following 3 types depending on the implication of the possible values. This information is used to decide the safe value for a feature. LOWER_SAFE - The smaller value is safer HIGHER_SAFE - The bigger value is safer EXACT - We can't decide between the two, so a predefined safe_value is used. This infrastructure will be later used to make better decisions for: - Kernel features (e.g, KVM, Debug) - SANITY Check - CPU capability - ELF HWCAP - Exposing CPU Feature register to userspace. Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> [catalin.marinas@arm.com: whitespace fix] Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-21arm64: Handle width of a cpuid featureSuzuki K. Poulose1-3/+9
Introduce a helper to extract cpuid feature for any given width. Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-21arm64: Move mixed endian support detectionSuzuki K. Poulose4-15/+11
Move the mixed endian support detection code to cpufeature.c from cpuinfo.c. This also moves the update_cpu_features() used by mixed endian detection code, which will get more functionality. Also moves the ID register field shifts to asm/sysreg.h, where all the useful definitions will end up in later patches. Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-21arm64: Delay ELF HWCAP initialisation until all CPUs are upSuzuki K. Poulose1-0/+1
Delay the ELF HWCAP initialisation until all the (enabled) CPUs are up, i.e, smp_cpus_done(). This is in preparation for detecting the common features across the CPUS and creating a consistent ELF HWCAP for the system. Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Tested-by: Dave Martin <Dave.Martin@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-20Merge tag 'v4.3-rc6' into locking/core, to pick up fixes before applying new changesIngo Molnar3-1/+13
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-10-19arm64: Add 16K page size supportSuzuki K. Poulose3-0/+8
This patch turns on the 16K page support in the kernel. We support 48bit VA (4 level page tables) and 47bit VA (3 level page tables). With 16K we can map 128 entries using contiguous bit hint at level 3 to map 2M using single TLB entry. TODO: 16K supports 32 contiguous entries at level 2 to get us 1G(which is not yet supported by the infrastructure). That should be a separate patch altogether. Cc: Will Deacon <will.deacon@arm.com> Cc: Jeremy Linton <jeremy.linton@arm.com> Cc: Marc Zyngier <marc.zyngier@arm.com> Cc: Christoffer Dall <christoffer.dall@linaro.org> Cc: Steve Capper <steve.capper@linaro.org> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-19arm64: Check for selected granule supportSuzuki K. Poulose1-0/+20
Ensure that the selected page size is supported by the CPU(s). If it doesn't park it. Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-19arm64: Simplify NR_FIX_BTMAPS calculationMark Rutland1-5/+2
We choose NR_FIX_BTMAPS such that each slot (NR_FIX_BTMAPS * PAGE_SIZE) can address 256K. Use division to derive NR_FIX_BTMAPS rather than defining it for each page size. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-19arm64: Clean config usages for page sizeSuzuki K. Poulose1-1/+1
We use !CONFIG_ARM64_64K_PAGES for CONFIG_ARM64_4K_PAGES (and vice versa) in code. It all worked well, so far since we only had two options. Now, with the introduction of 16K, these cases will break. This patch cleans up the code to use the required CONFIG symbol expression without the assumption that !64K => 4K (and vice versa) Cc: Will Deacon <will.deacon@arm.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-19arm64: Calculate size for idmap_pg_dir at compile timeSuzuki K. Poulose1-2/+5
Now that we can calculate the number of levels required for mapping a va width, reserve exact number of pages that would be required to cover the idmap. The idmap should be able to handle the maximum physical address size supported. Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-19arm64: Introduce helpers for page table levelsSuzuki K. Poulose1-3/+36
Introduce helpers for finding the number of page table levels required for a given VA width, shift for a particular page table level. Convert the existing users to the new helpers. More users to follow. Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> Cc: Will Deacon <will.deacon@arm.com> Cc: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Acked-by: Christoffer Dall <christoffer.dall@linaro.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-19arm64: Handle section maps for swapper/idmapSuzuki K. Poulose1-11/+26
We use section maps with 4K page size to create the swapper/idmaps. So far we have used !64K or 4K checks to handle the case where we use the section maps. This patch adds a new symbol, ARM64_SWAPPER_USES_SECTION_MAPS, to handle cases where we use section maps, instead of using the page size symbols. Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-19arm64: Move swapper pagetable definitionsSuzuki K. Poulose2-18/+65
Move the kernel pagetable (both swapper and idmap) definitions from the generic asm/page.h to a new file, asm/kernel-pgtable.h. This is mostly a cosmetic change, to clean up the asm/page.h to get rid of the arch specific details which are not needed by the generic code. Also renames the symbols to prevent conflicts. e.g, BLOCK_SHIFT => SWAPPER_BLOCK_SHIFT Cc: Will Deacon <will.deacon@arm.com> Signed-off-by: Suzuki K. Poulose <suzuki.poulose@arm.com> Reviewed-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Tested-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Acked-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-16arm64: Minor coding style fixes for kc_offset_to_vaddr and kc_vaddr_to_offsetCatalin Marinas1-2/+3
These were introduced by commit 03875ad52fdd (arm64: add kc_offset_to_vaddr and kc_vaddr_to_offset macro). Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-15arm64: Hook up IOMMU dma_opsRobin Murphy1-8/+7
With iommu_dma_ops in place, hook them up to the configuration code, so IOMMU-fronted devices will get them automatically. Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Robin Murphy <robin.murphy@arm.com> Signed-off-by: Joerg Roedel <jroedel@suse.de>
2015-10-14Merge branch 'x86/urgent' into core/efi, to pick up a pending EFI fixIngo Molnar49-694/+1540
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2015-10-14arm64: compat: wire up new syscallsWill Deacon2-1/+10
Commit 208473c1f3ac ("ARM: wire up new syscalls") hooked up the new userfaultfd and membarrier syscalls for ARM, so do the same for our compat syscall table in arm64. Signed-off-by: Will Deacon <will.deacon@arm.com>
2015-10-13Merge branch 'linus' into irq/coreThomas Gleixner1-2/+2
Bring in upstream updates for patches which depend on them
2015-10-13Revert "arm64: ioremap: add ioremap_cache macro"Catalin Marinas1-1/+0
This reverts commit 1b6d7f8742d5d46c478f10c9e57da18d049b116d. This patch would conflict with Dan Williams' "tree-wide convert to memremap()" series (ioremap_cache replaced by arch_memremap) Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-13arm64: add kc_offset_to_vaddr and kc_vaddr_to_offset macroyalin wang1-0/+2
This patch add kc_offset_to_vaddr() and kc_vaddr_to_offset(), the default version doesn't work on arm64, because arm64 kernel address is below the PAGE_OFFSET, like module address and vmemmap address are all below PAGE_OFFSET address. Signed-off-by: yalin wang <yalin.wang2010@gmail.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-13arm64: ioremap: add ioremap_cache macroyalin wang1-0/+1
Add ioremap_cache macro, because some code will test if this macro is defined or not, and will generate a generric version if not defined, for example, memremap.c do like this. Signed-off-by: yalin wang <yalin.wang2010@gmail.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-13arm64: kasan: fix issues reported by sparseWill Deacon1-0/+2
Sparse reports some new issues introduced by the kasan patches: arch/arm64/mm/kasan_init.c:91:13: warning: no previous prototype for 'kasan_early_init' [-Wmissing-prototypes] void __init kasan_early_init(void) ^ arch/arm64/mm/kasan_init.c:91:13: warning: symbol 'kasan_early_init' was not declared. Should it be static? [sparse] This patch resolves the problem by adding a prototype for kasan_early_init and marking the function as asmlinkage, since it's only called from head.S. Signed-off-by: Will Deacon <will.deacon@arm.com> Acked-by: Andrey Ryabinin <ryabinin.a.a@gmail.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-12Merge 4.3-rc5 into tty-nextGreg Kroah-Hartman1-2/+2
We want the tty fixes and reverts in here as well so that people can properly test and use it. Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2015-10-12arm64: add KASAN supportAndrey Ryabinin3-0/+59
This patch adds arch specific code for kernel address sanitizer (see Documentation/kasan.txt). 1/8 of kernel addresses reserved for shadow memory. There was no big enough hole for this, so virtual addresses for shadow were stolen from vmalloc area. At early boot stage the whole shadow region populated with just one physical page (kasan_zero_page). Later, this page reused as readonly zero shadow for some memory that KASan currently don't track (vmalloc). After mapping the physical memory, pages for shadow memory are allocated and mapped. Functions like memset/memmove/memcpy do a lot of memory accesses. If bad pointer passed to one of these function it is important to catch this. Compiler's instrumentation cannot do this since these functions are written in assembly. KASan replaces memory functions with manually instrumented variants. Original functions declared as weak symbols so strong definitions in mm/kasan/kasan.c could replace them. Original functions have aliases with '__' prefix in name, so we could call non-instrumented variant if needed. Some files built without kasan instrumentation (e.g. mm/slub.c). Original mem* function replaced (via #define) with prefixed variants to disable memory access checks for such files. Signed-off-by: Andrey Ryabinin <ryabinin.a.a@gmail.com> Tested-by: Linus Walleij <linus.walleij@linaro.org> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-12arm64: move PGD_SIZE definition to pgalloc.hAndrey Ryabinin1-0/+1
This will be used by KASAN latter. Signed-off-by: Andrey Ryabinin <ryabinin.a.a@gmail.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-12arm64: Fix MINSIGSTKSZ and SIGSTKSZManjeet Pawar1-0/+3
MINSIGSTKSZ and SIGSTKSZ for ARM64 are not correctly set in latest kernel. This patch fixes this issue. This issue is reported in LTP (testcase: sigaltstack02.c). Testcase failed when sigaltstack() called with stack size "MINSIGSTKSZ - 1" Since in Glibc-2.22, MINSIGSTKSZ is set to 5120 but in kernel it is set to 2048 so testcase gets failed. Testcase Output: sigaltstack02 1 TPASS : stgaltstack() fails, Invalid Flag value,errno:22 sigaltstack02 2 TFAIL : sigaltstack() returned 0, expected -1,errno:12 Reported Issue in Glibc Bugzilla: Bugfix in Glibc-2.22: [Bug 16850] https://sourceware.org/bugzilla/show_bug.cgi?id=16850 Acked-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Akhilesh Kumar <akhilesh.k@samsung.com> Signed-off-by: Manjeet Pawar <manjeet.p@samsung.com> Signed-off-by: Rohit Thapliyal <r.thapliyal@samsung.com> Signed-off-by: Will Deacon <will.deacon@arm.com>
2015-10-12arm64: atomics: implement native {relaxed, acquire, release} atomicsWill Deacon4-258/+367
Commit 654672d4ba1a ("locking/atomics: Add _{acquire|release|relaxed}() variants of some atomic operation") introduced a relaxed atomic API to Linux that maps nicely onto the arm64 memory model, including the new ARMv8.1 atomic instructions. This patch hooks up the API to our relaxed atomic instructions, rather than have them all expand to the full-barrier variants as they do currently. Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-12arm64: use ENDPIPROC() to annotate position independent assembler routinesArd Biesheuvel1-0/+11
For more control over which functions are called with the MMU off or with the UEFI 1:1 mapping active, annotate some assembler routines as position independent. This is done by introducing ENDPIPROC(), which replaces the ENDPROC() declaration of those routines. Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-09irqchip/gic-v3: Specialize readq and writeq accessesJean-Philippe Brucker1-0/+3
On 32bit platforms, we cannot assure that an I/O ldrd or strd will be done atomically. Besides, an hypervisor would be unable to emulate such accesses. In order to allow the AArch32 version of the driver to split them into two 32bit accesses while keeping the requirement for atomic writes, this patch specializes the IROUTER and TYPER accesses. Since the latter is an ID register, it won't need to be read atomically, but we still avoid future confusion by using gic_read_typer instead of a generic gic_readq. Reviewed-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Jean-Philippe Brucker <jean-philippe.brucker@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2015-10-09irqchip/gic-v3: Change unsigned types for AArch32 compatibilityJean-Philippe Brucker1-14/+19
This patch does a few simple compatibility-related changes: - change the system register access prototypes to their actual size, - homogenise mpidr accesses with unsigned long, - force the 64bit register values to unsigned long long. Note: the list registers are 64bit on GICv3, but the AArch32 vGIC driver will need to split their values into two 32bit registers: LRn and LRCn. Reviewed-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Jean-Philippe Brucker <jean-philippe.brucker@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2015-10-09irqchip/gic-v3: Refactor the arm64 specific partsJean-Philippe Brucker1-0/+162
This patch moves the GICv3 system register access helpers to arch/arm64/. Their 32bit counterparts will need to use mrc/mcr accesses instead of mrs_s/msr_s. [maz: fixed conflict with Cavium erratum handling] Reviewed-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Jean-Philippe Brucker <jean-philippe.brucker@arm.com> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
2015-10-09arm64: fix a migrating irq bug when hotplug cpuYang Yingliang1-1/+0
When cpu is disabled, all irqs will be migratged to another cpu. In some cases, a new affinity is different, the old affinity need to be updated and if irq_set_affinity's return value is IRQ_SET_MASK_OK_DONE, the old affinity can not be updated. Fix it by using irq_do_set_affinity. And migrating interrupts is a core code matter, so use the generic function irq_migrate_all_off_this_cpu() to migrate interrupts in kernel/irq/migration.c. Cc: Jiang Liu <jiang.liu@linux.intel.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Russell King - ARM Linux <linux@arm.linux.org.uk> Cc: Hanjun Guo <hanjun.guo@linaro.org> Acked-by: Marc Zyngier <marc.zyngier@arm.com> Signed-off-by: Yang Yingliang <yangyingliang@huawei.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-08arm64: Default kernel pages should be contiguousJeremy Linton1-0/+1
The default page attributes for a PMD being broken should have the CONT bit set. Create a new definition for an early boot range of PTE's that are contiguous. Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-08arm64: Macros to check/set/unset the contiguous bitJeremy Linton1-0/+11
Add the supporting macros to check if the contiguous bit is set, set the bit, or clear it in a PTE entry. Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-08arm64: PTE/PMD contiguous bit definitionJeremy Linton1-0/+9
Define the bit positions in the PTE and PMD for the contiguous bit. Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-08arm64: Add contiguous page flag shifts and constantsJeremy Linton1-1/+7
Add the number of pages required to form a contiguous range, as well as some supporting constants. Signed-off-by: Jeremy Linton <jeremy.linton@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: perf: move to shared arm_pmu frameworkMark Rutland1-83/+0
Now that the arm_pmu framework has been factored out to drivers/perf we can make use of it for arm64, gaining support for heterogeneous PMUs and unifying the two codebases before they diverge further. The as yet unused PMU name for PMUv3 is changed to armv8_pmuv3, matching the style previously applied to the 32-bit PMUs. Signed-off-by: Mark Rutland <mark.rutland@arm.com> Acked-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: mm: remove dsb from update_mmu_cacheWill Deacon1-3/+3
update_mmu_cache() consists of a dsb(ishst) instruction so that new user mappings are guaranteed to be visible to the page table walker on exception return. In reality this can be a very expensive operation which is rarely needed. Removing this barrier shows a modest improvement in hackbench scores and , in the worst case, we re-take the user fault and establish that there was nothing to do. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: tlb: remove redundant barrier from __flush_tlb_pgtableWill Deacon1-1/+0
__flush_tlb_pgtable is used to invalidate intermediate page table entries after they have been cleared and are about to be freed. Since pXd_clear imply memory barriers, we don't need the extra one here. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: switch_mm: simplify mm and CPU checksWill Deacon1-2/+4
switch_mm performs some checks to try and avoid entering the ASID allocator: (1) If we're switching to the init_mm (no user mappings), then simply set a reserved TTBR0 value with no page table (the zero page) (2) If prev == next *and* the mm_cpumask indicates that we've run on this CPU before, then we can skip the allocator. However, there is plenty of redundancy here. With the new ASID allocator, if prev == next, then we know that our ASID is valid and do not need to worry about re-allocation. Consequently, we can drop the mm_cpumask check in (2) and move the prev == next check before the init_mm check, since if prev == next == init_mm then there's nothing to do. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: tlbflush: avoid flushing when fullmm == 1Will Deacon1-11/+15
The TLB gather code sets fullmm=1 when tearing down the entire address space for an mm_struct on exit or execve. Given that the ASID allocator will never re-allocate a dirty ASID, this flushing is not needed and can simply be avoided in the flushing code. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: tlbflush: remove redundant ASID casts to (unsigned long)Will Deacon1-5/+4
The ASID macro returns a 64-bit (long long) value, so there is no need to cast to (unsigned long) before shifting prior to a TLBI operation. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: mm: rewrite ASID allocator and MM context-switching codeWill Deacon3-73/+19
Our current switch_mm implementation suffers from a number of problems: (1) The ASID allocator relies on IPIs to synchronise the CPUs on a rollover event (2) Because of (1), we cannot allocate ASIDs with interrupts disabled and therefore make use of a TIF_SWITCH_MM flag to postpone the actual switch to finish_arch_post_lock_switch (3) We run context switch with a reserved (invalid) TTBR0 value, even though the ASID and pgd are updated atomically (4) We take a global spinlock (cpu_asid_lock) during context-switch (5) We use h/w broadcast TLB operations when they are not required (e.g. in flush_context) This patch addresses these problems by rewriting the ASID algorithm to match the bitmap-based arch/arm/ implementation more closely. This in turn allows us to remove much of the complications surrounding switch_mm, including the ugly thread flag. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: flush: use local TLB and I-cache invalidationWill Deacon2-0/+15
There are a number of places where a single CPU is running with a private page-table and we need to perform maintenance on the TLB and I-cache in order to ensure correctness, but do not require the operation to be broadcast to other CPUs. This patch adds local variants of tlb_flush_all and __flush_icache_all to support these use-cases and updates the callers respectively. __local_flush_icache_all also implies an isb, since it is intended to be used synchronously. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: David Daney <david.daney@cavium.com> Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: mm: remove unused cpu_set_idmap_tcr_t0sz functionWill Deacon1-23/+12
With commit b08d4640a3dc ("arm64: remove dead code"), cpu_set_idmap_tcr_t0sz is no longer called and can therefore be removed from the kernel. This patch removes the function and effectively inlines the helper function __cpu_set_tcr_t0sz into cpu_set_default_tcr_t0sz. Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Acked-by: Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by: Will Deacon <will.deacon@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2015-10-07arm64: introduce VA_START macro - the first kernel virtual address.Andrey Ryabinin2-1/+3
In order to not use lengthy (UL(0xffffffffffffffff) << VA_BITS) everywhere, replace it with VA_START. Signed-off-by: Andrey Ryabinin <ryabinin.a.a@gmail.com> Reviewed-by: Catalin Marinas <catalin.marinas@arm.com> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>