| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
On gfx9 ASICs with mqd_on_vram(), a compute queue MQD lives in a pinned
VRAM buffer object. Pinned BOs are skipped by the VRAM eviction done at S4
suspend, so the MQD contents are lost across hibernation and the first
submission after resume page-faults on a stale MQD.
Unpin the MQD BO at suspend so the eviction migrates it into the
hibernation image, and pin it back to VRAM on resume. The BO may return at
a different VRAM address, so refresh the kernel mapping and cached GPU
addresses and patch the MQD self-address via a new update_mqd_gpu_addr()
mqd_manager op; skip eviction with a warning if that op is not implemented.
v3: use unpin/repin instead of shadowing the MQD into a separate buffer.
v4: drop the explicit VRAM->GTT placement at evict (a bare unpin is enough
for the eviction pass to move the BO out of VRAM), and also repin at queue
destroy. KFD queue restore runs late - user processes thaw before it, and
under SR-IOV it is deferred until the VF exits full access - so once the
VM has resumed an application can destroy a queue before its MQD BO is
repinned, which would otherwise unpin an already-unpinned BO and touch a
stale q->mqd.
v5: drop support for no-HWS mode, and set q->mqd to NULL at eviction.
Signed-off-by: Shikang Fan <shikang.fan@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Fix a spelling mistake in a comment.
Signed-off-by: Kenji Takahashi <dken4546@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Enabled RS64mem for KFD queues by integrating
process and gang context index allocation in
the per KFD device process and queue creation.
Signed-off-by: Prike Liang <Prike.Liang@amd.com>
Reviewed-by: Michael Chen <michael.chen@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
When KFD_IOCTL_SVM_ATTR_NO_ACCESS is applied to a GPU that has an SVM
range mapped, unmap the range from that GPU. Once no GPU maps the range,
the MMU notifier can skip queue eviction on CPU page faults.
Replace the mapped_to_gpu boolean with bitmap_mapped to track which
GPUs currently have the range mapped. Set bits in svm_range_map_to_gpus()
and clear them in svm_range_unmap_from_gpus(). This is separate from
bitmap_access/bitmap_aip which track user-requested attributes and must
not be used to determine mapping state.
Add bitmap_needs_unmap to svm_range, set for each GPU given no-access.
Add svm_range_needs_unmap() to unmap the range from those GPUs when the
app sets the no-access attribute.
Bump the checkpoint timestamp on unmap so retry faults queued before the
no-access unmap are dropped instead of restoring the mapping.
v4:
- Rename and set prange->mapping_done to false if validate and map not
complete successfully (Felix)
v3:
- Correct error handling, support app retry update mapping (Felix)
v2:
- Add bitmap_mapped to not break get_attr (Felix)
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Extract svm_range_update_checkpoint_timestamp() from
svm_range_unmap_from_cpu(). The next patch calls it when the app sets
the no-access attribute.
Change checkpoint_ts in svm_range_list from uint64_t to atomic64_t so
svm_range_restore_pages() can read it from the page fault handler
without holding the svms lock.
No functional change, preparation for the next patch.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
SVM BOs are now migrated back to system memory synchronously from the TTM
eviction path (svm_range_evict_svm_bo), so the per-svm_bo eviction fence
is no longer used.
Remove the eviction fence from svm_range_bo, drop the
amdgpu_amdkfd_fence->svm_bo back pointer and the
amdgpu_amdkfd_evict_svm_bo() helper, and stop special-casing svm_bo
fences in the KFD fence enable_signaling and check_mm paths. Embed struct
amdgpu_bo directly in svm_range_bo with a dedicated svm_range_bo_destroy()
callback, and keep the owning mm via mmgrab()/mmdrop() instead of through
the fence.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
svm_range_evict_svm_bo_worker() migrated an SVM BO's pages back to system
memory from a work item that took mmap_read_lock. When an mmap writer was
pending, that read lock blocked behind the writer while the thread
allocating a new migration VRAM BO waited on this BO's eviction fence - a
circular wait that hung the SVM workers.
Evict the SVM BO synchronously from the TTM eviction path
(amdgpu_ttm_bo_eviction_valuable) instead of deferring to a work item.
The BO is already reserved and the lock order is mmap_lock -> BO
reservation, so only trylock the owning process's mmap lock; on
contention return -EBUSY so TTM skips this BO. This removes the eviction
work item and the enable_signaling path, so no worker can block on
mmap_read_lock.
The SVM BO uses AMDGPU_GEM_CREATE_DISCARDABLE, so ttm_bo_evict takes the
pipeline_gutting path and skips allocating a system memory placement.
That would be wasted work, since svm_migrate_vram_to_ram allocates the
system pages and copies the data back itself.
Eviction now migrates ranges directly, so it must serialize with the
owning process: it trylocks migrate_mutex under svm_bo->list_lock before
unlinking the range, and svm_range_free() unlinks the range then waits on
migrate_mutex, so a concurrent eviction cannot free a range under it.
Drop the mm reference with mmput_async so exit_mmap() does not run under
the BO reservation.
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
When a mode reset happens, driver needs to notify the process on that
GPU a reset event is happening. The existing code assumes the process is
using the GPU that is getting mode reset, which is not always true. For
example, on a 8G system, the process may be only using GPU 0~4 but a
mode 2 reset is resetting the all 8 GPUs connected by XGMI. Trying to
find a process on GPU 5~7 will fail, which is fine and should skip the
event signal.
Signed-off-by: Amber Lin <amber.lin@amd.com>
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The debugger sets the CWSR grace period via AMDKFD_IOC_DBG_TRAP and may
request a value of 0. An earlier change clamped 0 to 1 unconditionally
because most firmware revisions locked up when given an infinite (0)
grace period.
Firmware has since been fixed on most ASICs, so gate the clamp on ASIC
type and MEC firmware version and allow 0 where the running firmware is
known to handle it. Navi3x and MI350 support 0 in every firmware
revision and need no version check. MI100 never received the firmware
fix and is kept clamped. Any unlisted or future ASIC defaults to the
safe (clamped) behaviour.
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
kfd_criu_checkpoint_events() counts the entries in p->event_idr via
kfd_get_num_events(), allocates an array sized to that count, and then
walks the same IDR to fill it. Neither the count nor the walk holds
p->event_mutex.
The CRIU checkpoint caller holds only p->mutex. Event create and destroy
(kfd_event_create()/kfd_event_destroy()) take p->event_mutex and do not
take p->mutex, so a second thread in the same process can insert or remove
events between the count and the walk. If an event is inserted, the walk
iterates more entries than were counted and writes past the end of the
ev_privs allocation; if an event is removed, the walk dereferences an
entry that is being freed.
Hold p->event_mutex across the count and the walk so both observe a
consistent view of p->event_idr. The lock is released before
copy_to_user(), which only touches the local buffer. The caller already
holds p->mutex and the create/destroy paths never take p->mutex, so the
p->mutex -> p->event_mutex order is not inverted and no deadlock is
introduced.
Fixes: 40e8a766a761 ("drm/amdkfd: CRIU checkpoint and restore events")
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Alysa Liu <Alysa.Liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Use process-local GPU lookup in kfd_mmap() instead of kfd_device_by_id(),
so mmap path does not take topology_lock while holding mmap_lock.
Lockdep warning: possible circular locking dependency detected.
Chain:
&topology_lock -> &root->kernfs_rwsem -> &mm->mmap_lock
Backtrace #0 (&topology_lock):
down_read+0x8b/0x490
kfd_device_by_id+0x1d/0xf0 [amdgpu]
kfd_mmap+0x11e/0x750 [amdgpu]
__mmap_region+0x101f/0x25e0
mmap_region+0x228/0x2e0
do_mmap+0x9f2/0x1070
vm_mmap_pgoff+0x25d/0x430
Backtrace #1 (&root->kernfs_rwsem):
down_write+0x80/0x200
kernfs_add_one+0x30/0x6a0
kernfs_create_dir_ns+0xd2/0x160
sysfs_create_dir_ns+0x12c/0x2a0
kobject_add_internal+0x280/0x8e0
kfd_topology_update_sysfs+0xb85/0x1970 [amdgpu]
Backtrace #2 (&mm->mmap_lock):
down_read_killable+0x8e/0x510
lock_mm_and_find_vma+0x299/0xb00
do_user_addr_fault+0x3fc/0xf80
exc_page_fault+0x73/0x110
filldir64+0x1aa/0x610
kernfs_fop_readdir+0x3c2/0x810
Signed-off-by: Philip Yang <Philip.Yang@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Convert ptl_hw_supported to enum with three states:
- AMDGPU_PTL_HW_UNINIT: not yet initialized
- AMDGPU_PTL_HW_SUPPORTED: initialized and supported
- AMDGPU_PTL_HW_NOT_SUPPORTED: initialized and not supported
This allows skipping PTL initialization attempts when hardware is
known to not support it, avoiding repeated initialization failures
after GPU resets.
v2:move ptl_hw_supported_state to AMDGPU_PTL_HW_NOT_SUPPORTED regardless
of error code during first time initialization. Print init fail log when
error code is not EOPNOTSUPP.
Signed-off-by: Victor Zhao <Victor.Zhao@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
eop_ring_buffer_size in struct queue_properties is a u32. In
kfd_queue_acquire_buffers() the expected EOP buffer size is computed as
ALIGN(eop_ring_buffer_size, PAGE_SIZE); ALIGN uses typeof(x), so the
addition is done in 32-bit. A user-supplied size of 0xFFFFF001 wraps to
0, causing kfd_queue_buffer_get() to skip its exact-size check (gated on
size != 0) and accept any BO mapped at the address. On GFX8/GFX9 the MQD
cp_hqd_eop_control is then programmed for an 8KB EOP ring backed by a 4KB
BO, so CP EOP writes can land past the buffer and fault the GPU.
Cast the operand to u64 so the alignment is computed in 64-bit; the size
check in kfd_queue_buffer_get() then rejects the oversized request.
Fixes: 42ea9cf2f16b ("drm/amdkfd: Relax size checking during queue buffer get")
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Alysa Liu <Alysa.Liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Prevent unauthorized termination of active GPU debug sessions.
Previously, users with /dev/kfd access could terminate another process's
debug session without proper ownership or ptrace authorization.
Signed-off-by: Gang Ba <Gang.Ba@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
In kfd_criu_restore_event, there was no handling for
the event priv data having an invalid event type. The priv
data here is untrusted and can be invalid.
In that case, fail with EINVAL.
Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
When MES is enabled and amdgpu_amdkfd_alloc_kernel_mem() fails during
the first queue creation for a process, pqm_create_queue() returns
early via 'return retval' without going through the err_create_queue
cleanup label.
This means clear_bit(*qid, pqm->queue_slot_bitmap) is never called,
leaving the reserved QID bit permanently set in queue_slot_bitmap.
Over time this leaks QID slots, potentially exhausting all available
queue slots.
Fix this by replacing 'return retval' with 'goto err_allocate_pqn'
so that clear_bit() is always called on the error path without
touching the uninitialized pqn pointer.
AILIKFD-813
Reported-by: Deucher, Alexander <alexander.deucher@amd.com>
Signed-off-by: Vladimir Marioukhine <Vladimir.Marioukhine@amd.com>
Reviewed-by: Kent Russell <kent.russell@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The CRAT parser validates that the subtype header fits within the image,
but does not verify that the advertised subtype length fits. A malformed
CRAT table with an oversized length field causes out-of-bounds reads when
kfd_parse_subtype() casts the header to specific subtype structures.
Add validation that sub_type_hdr + length does not exceed the image
boundary before parsing the subtype contents.
Signed-off-by: William Palacek <William.Palacek@amd.com>
Reviewed-by: Alysa Liu <Alysa.Liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Having single lines with random hex codes really doesn't help a user to
know what's going on. Give it a title, and print 8 8-length hex values
per line, instead of a single 2-length hex value per printed line.
Previous output:
...
amdgpu: 0x20000010
amdgpu: 0x3000
amdgpu: 0x12E6E00
amdgpu: 0xFF
amdgpu: 0x207008
amdgpu: 0x 0
amdgpu:
New output:
amdgpu: Runlist dump:
amdgpu: 0: 0xc00ea100 0x14008008 0x0f4fffc0 0x00000000 0x20002000 0x00000338 0x00000020 0x00080017
amdgpu: 8: 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x03400000 0x00000000 0x00000000
amdgpu: 16: 0xc005a200 0x20000010 0x00003030 0x012ed000 0x000000ff 0x00252008 0x00000000 0xc005a200
amdgpu: 24: 0x20000010 0x0000302c 0x012ec800 0x000000ff 0x0024c008 0x00000000 0xc005a200 0x20000010
amdgpu: 32: 0x00003028 0x012ec000 0x000000ff 0x00246008 0x00000000 0xc005a200 0x20000010 0x00003024
amdgpu: 40: 0x012eb800 0x000000ff 0x00240008 0x00000000 0xc005a200 0x20000010 0x00003020 0x012eb000
amdgpu: 48: 0x000000ff 0x0023a008 0x00000000 0xc005a200 0x20000010 0x0000301c 0x012ea800 0x000000ff
amdgpu: 56: 0x00234008 0x00000000 0xc005a200 0x20000010 0x00003018 0x012ea000 0x000000ff 0x0022e008
amdgpu: 64: 0x00000000 0xc005a200 0x20000010 0x00003014 0x012e9800 0x000000ff 0x00228008 0x00000000
amdgpu: 72: 0xc005a200 0x20000010 0x00003010 0x012e9000 0x000000ff 0x00222008 0x00000000 0xc005a200
amdgpu: 80: 0x20000010 0x0000300c 0x012e8800 0x000000ff 0x0021c008 0x00000000 0xc005a200 0x20000010
amdgpu: 88: 0x00003008 0x012e8000 0x000000ff 0x00216008 0x00000000 0xc005a200 0x20000010 0x00003004
amdgpu: 96: 0x012e7800 0x000000ff 0x00210008 0x00000000 0xc005a200 0x20000010 0x00003000 0x012e6e00
amdgpu: 104: 0x000000ff 0x00207008 0x00000000
Signed-off-by: Kent Russell <kent.russell@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Move the job output parameter to be the last parameter in the
amdgpu_job_alloc() and amdgpu_job_alloc_with_ib() function signature. This
aligns with the common kernel coding convention where output parameters
typically come last.
Assisted-by: Claude:Sonnet 4
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Timur Kristóf <timur.kristof@gmail.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
We need to increment the reset counter, force fence completion,
and set the wedged event when a user queue is reset, but only
for the guilty queue. We don't want additional events for
collateral damage.
Only increment the reset counter and set the wedged event
in the top level userq reset callers (both KGD and KFD).
This aligns with how this is managed for kernel queue resets.
v2: fix task info refcount handling, align with
kfd error reporting.
v3: check for has_reset_queue in KFD path (Prike)
Cc: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Cc: Prike Liang <Prike.Liang@amd.com>
Cc: Sunil Khatri <sunil.khatri@amd.com>
Cc: Lazar, Lijo <lijo.lazar@amd.com>
Reviewed-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Reviewed-by: Lijo Lazar <lijo.lazar@amd.com>
Reviewed-by: Prike Liang <Prike.Liang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The change referenced by the Fixes tag releases the HIQ SDMA MQD trunk
buffer when device_queue_manager_init() fails after it has been
allocated.
However, the same failure path can also be reached after
init_mqd_managers() has succeeded. At that point dqm->mqd_mgrs[] contains
per-type MQD manager objects owned by the device queue manager. The
normal teardown path frees those objects from uninitialize(), but the
initialization error path only frees dqm itself.
Free the MQD managers from the initialization error path as well. This is
safe for earlier failures because dqm is zeroed when allocated and
init_mqd_managers() clears the entries it rolls back internally.
Fixes: b7cccc8286bb ("drm/amdkfd: fix a memory leak in device_queue_manager_init()")
Signed-off-by: Guangshuo Li <lgs201920130244@gmail.com>
Reviewed-by: Mukul Joshi <mukul.joshi@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
debugfs is intended for debugging only, and failures to create debugfs
entries should not affect normal operation.
Remove the check for debugfs_create_dir() in kfd_debugfs_add_process().
If debugfs entries cannot be created, continue without them instead of
reporting an unnecessary error.
Fixes: 325de6bf3328 ("drm/amdkfd: expose pasid of secondary contexts by debugfs")
Reported-by: Dan Carpenter <error27@gmail.com>
Cc: Zhu Lingshan <lingshan.zhu@amd.com>
Cc: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Srinivasan Shanmugam <srinivasan.shanmugam@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
total_cwsr_size was computed in 32-bit before being used as a BO/SVM
allocation size.
With large ctx_save_restore_area_size and debug_memory_size
multiplied by the XCC count, the product can wrap,
yielding an undersized CWSR save area that firmware later overruns.
Promote total_cwsr_size to u64 and use check_add_overflow()/
check_mul_overflow() in both kfd_queue_acquire_buffers() and
kfd_queue_release_buffers().
Signed-off-by: Yongqiang Sun <Yongqiang.Sun@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The valid amdkfd event ids go from 0 to KFD_SIGNAL_EVENT_LIMIT - 1.
During CRIU restore, ensure that the provided event ids are
in that range.
v2: No need for lower bound check since idr_alloc rejects negative
inputs
v3: Also change error message to reflect new error condition
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Current kfd debugfs interfaces only expose pasid
of the primary process, this commit exposes
pasid of secondary contexts by debugfs
Just like entries under sysfs,
the secondary contexts are named as
context_<id> under its primary kfd process.
The layout:
/sys/kernel/debug/kfd/proc# tree
.
└── 5802
├── context_0
│ ├── pasid_1025
│ └── pasid_63266
├── context_1
│ ├── pasid_1025
│ └── pasid_63266
├── pasid_1025
└── pasid_63266
Another fix is, kfd_debugfs_add_process may fail,
this commit change it to return a meaningful
value other than void
Signed-off-by: Zhu Lingshan <lingshan.zhu@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The valid event ids go from 0 to KFD_SIGNAL_EVENT_LIMIT
allocate_event_notification_slot has an option to specify
an event id to allocate at, used by CRIU. We weren't checking
the bounds on that value.
Check them.
v2: Lower bounds check is unecessary because of idr_alloc
already rejecting negative numbers. Upper bounds check should
be KFD_SIGNAL_EVENT_LIMIT since the signal mode mappings might
not yet exist
Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Function kfd_process_free_id() should skip over
the primary kfd process because its context id
is fixed assigned, not allocated through the ida table.
This function should only work on secondary contexts.
Fixes: fac682a1d1af ("amdkfd: identify a secondary kfd process by its id")
Signed-off-by: Zhu Lingshan <lingshan.zhu@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
kfd_criu_restore_queue's call of kfd_queue_acquire_buffers was
failing for multiple reasons
- The ctl_stack_size set by the CRIU plugin doesn't match
what is expected by acquire_buffers
- The svm buffer cannot be acquired at this point because
CRIU may not have restored it, or may have restored it
to a different address.
The only reason acquire_buffers was necessary here was to
avoid a null ptr dereference in init_user_queue.
Just put in a check for that dereference; it doesn't appear to
come up in real use cases right now. That is, there is no
usage of CRIU with shared MES.
This is a partial revert of
commit 20a5e7ffdfec ("drm/amdkfd: Properly acquire queue buffers in CRIU restore")
Fixes: 20a5e7ffdfec ("drm/amdkfd: Properly acquire queue buffers in CRIU restore")
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
We weren't checking whether the values provided in the private
data in kfd CRIU restore were within bounds.
For queue type, add a KFD_QUEUE_TYPE_MAX and ensure the provided
type is less than it.
For mqd_size, add new function mqd_size_from_queue_type and confirm
that the provided mqd_size matches expectations.
Reviewed-by: David Yat Sin <david.yatsin@amd.com>
Signed-off-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
After moving TBA/TMA from GTT to VRAM for GFX9.4.2+ in commit
5088a1ba6d6d ("drm/amdkfd: move TBA/TMA from system to device memory"),
direct pointer dereferences to CWSR buffers became unsafe because VRAM
is accessed via MMIO (PCI BAR mappings).
Direct writes like 'tma[2] = enabled' and memcpy() can fail or
produce incorrect results on non-x86 architectures because:
- MMIO requires specific accessor functions (writeq/readq)
- Compiler optimizations may generate invalid instruction sequences
- No guarantee of proper memory barriers or atomic access
This patch converts CWSR buffer access to use struct iosys_map,
which automatically handles both system memory (GTT) and MMIO
(VRAM) correctly by:
- Using writeq/writel/memcpy_toio for MMIO regions
- Using WRITE_ONCE/memcpy for system memory
- Providing proper memory barriers and access guarantees
Changes:
- Replace void *cwsr_kaddr with struct iosys_map cwsr_map
- Detect MMIO vs system memory using TTM_BO_MAP_IOMEM_MASK
- Use iosys_map_wr() for writing trap handler addresses and flags
- Use iosys_map_memcpy_to() for copying CWSR ISA code
This ensures correct operation on all architectures while maintaining
backward compatibility with older GPUs and APUs that use GTT.
Assisted-by: Claude:Claude-Opus-4.6
Signed-off-by: James Zhu <James.Zhu@amd.com>
Acked-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Felix Kuehling <felix.kuehling@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Duplicated code from the merge. Drop it.
Fixes: 0461ba9a7994 ("Merge tag 'amd-drm-next-7.3-2026-07-02' of https://gitlab.freedesktop.org/agd5f/linux into drm-next")
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
[airlied: had to reapply
drm/amdgpu: Implement "color format" DRM property by hand to amdgpu_dm_connector.c]
amd-drm-next-7.3-2026-07-02:
amdgpu:
- Queue reset updates
- Initial compute pipe reset support
- Improved boundary checking for bios parsing
- Cleaned up sysfs input parsing
- devcoredump fixes
- RAS updates and rework
- VCN secure submission fixes
- 8K panel fix
- Add display KUnit tests
- Display CRC fixes
- UserQ updates
- Backlight fixes
- Parse panel type info from DisplayID
- Align IP discovery to pci device lifetime
- IOCTL boundary check fixes
- Convert amdgpu_vm_lock_by_pasid() to drm_exec
- Ctx fixes and cleanup
- SOC15 register macro cleanups
- Memory placement fixes for UVD
- Disable KQ support for MI3xx
- GFX9 mode2 reset fix
- BO list cleanup
- Soc24 aborted suspend fix
- Gfx8 soft reset rework
- Enable soft reset on gfx8
- Drop unnecessary BUG() and BUG_ON() in error paths
- Fix power reporting unit conversion
- Improve vbios command table bounds checking
- UVD bounds checking improvements
- VCN bounds checking improvements
- PSR and replay fixes
- DCN 4.2 updates
- Colorop updates
- DC GPIO rework
- ACP fixes
- Fix aperture mapping leak
- Ignore_damage_clips fix
- Fixes for non-4K pages
- JPEG idle check fixes
- Userptr fixes
- GPUVM fixes
- GC 11.7 updates
- SMU 13 fixes
amdkfd:
- Initial compute pipe reset support
- Allow applications to opt out of sigbus on fatal errors
- Fix doorbell/mmio BO cleanup
- Improved CRIU boundary checking
- MQD handling rework
- SMI fixes
- Reset event fixes
- CRIU fixes
- Sysfs teardown fixes
- IOCTL boundary check fixes
- SVM fixes
- Soft IH ring fixes
- Move TBA/TMA from system to device memory
radeon:
- Blit fix for large BOs
- r600 dpm cleanup fix
drm:
- Extract EDID base section header processing into helper
- Parse panel type from DisplayID 2.x Display Parameters
UAPI:
- KFD interface for applications to select sigbus behavior on fatal errors
Proposed userspace: https://github.com/ROCm/rocm-systems/pull/6190
From: Alex Deucher <alexander.deucher@amd.com>
Link: https://patch.msgid.link/20260702141515.67919-1-alexander.deucher@amd.com
Signed-off-by: Dave Airlie <airlied@redhat.com>
|
|
Running RCCL unit tests on a system with a 64K PAGE_SIZE triggers
the following warning and causes the test to terminate on latest
upstream kernel:
WARNING: drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:1335 at
amdgpu_bo_release_notify+0x1bc/0x280 [amdgpu],
CPU#18: rccl-UnitTests/33151
Call trace:
amdgpu_bo_release_notify
ttm_bo_release
amdgpu_gem_object_free
drm_gem_object_free
amdgpu_bo_unref
amdgpu_bo_create
amdgpu_bo_create_user
amdgpu_gem_object_create
amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu
kfd_ioctl_alloc_memory_of_gpu
kfd_ioctl
sys_ioctl
The warning is triggered because
amdgpu_ttm_next_clear_entity() returns NULL when a clear buffer
operation is requested. This happens because the GART window
allocation for the default_entity, clear_entity and move_entity
fails during initialization.
Commit [1] introduced separate GART windows for the
default_entity, clear_entity and move_entity of each SDMA
instance. Their sizes are derived from
AMDGPU_GTT_MAX_TRANSFER_SIZE, which is currently defined as 1024
pages. This implicitly assumes a 4K PAGE_SIZE, where 1024 pages
correspond to a 4MB transfer. On a 64K PAGE_SIZE system, however,
the same value expands to 64MB.
The default_entity and clear_entity each allocate one
AMDGPU_GTT_MAX_TRANSFER_SIZE GART window, while the move_entity
allocates two such windows. This results in 16MB of GART space
per SDMA instance on a 4K PAGE_SIZE system, but 256MB per SDMA
instance on a 64K PAGE_SIZE system.
On an MI210 system with five SDMA instances and a 512MB GART
aperture, the total GART space required becomes 1.25GB,
exceeding the available GART aperture. Consequently, GART window
allocation fails, amdgpu_ttm_next_clear_entity() returns NULL,
and the above warning is triggered.
Redefine AMDGPU_GTT_MAX_TRANSFER_SIZE in bytes instead of page
units. Where a page count is required, convert it using
PAGE_SHIFT. This preserves the existing 4MB transfer size across
all PAGE_SIZE configurations while keeping GART window
allocations within the available GART aperture.
[1] https://lore.kernel.org/all/20260408100327.1372-3-pierre-eric.pelloux-prayer@amd.com/#t
Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5435
Fixes: 897ee11ec020 ("drm/amdgpu: create multiple clear/move ttm entities")
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 27213b776a666d3030de5acc3cd75278197b0494)
Cc: stable@vger.kernel.org
|
|
There were a few instances in kfd_chardev.c of kvzalloc being
used to allocate memory for an array.
Switch those to kvcalloc, which
- is the standard way of allocating a zero-initialized array
- does a check for the mul overflowing
Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 60b048c93f7a3add39757ad65fe2bb6e58eeae23)
Cc: stable@vger.kernel.org
|
|
Initialize GC IP 11_7_1
Signed-off-by: Granthali Vinodkumar Dhandar <granthali.vinodkumardhandar@amd.com>
Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit a928d8d81ec5cdb5a8944d08136720811efad0f6)
|
|
Initialize GC IP 11_7_0
Signed-off-by: Granthali Vinodkumar Dhandar <granthali.vinodkumardhandar@amd.com>
Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit cf591e67c095542a16475df293ec7bc9a118e4ee)
|
|
CRIU checkpoint copies the MQD control stack using cp_hqd_cntl_stack_size
from hardware without bounding it to the allocated BO region. If the HW
field is larger than the queue's control stack allocation, memcpy reads
past the BO into adjacent GTT memory and can leak kernel data to userspace.
Store the page-aligned control stack BO size in mqd_manager and clamp
checkpoint copies and reported checkpoint sizes to
min(cp_hqd_cntl_stack_size, mm->ctl_stack_size). Apply the same bound
for multi-XCC v9.4.3 checkpoint layout.
Signed-off-by: Yongqiang Sun <Yongqiang.Sun@amd.com>
Reviewed-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit 6c2abd0ec09e86c6323010673766f76050e28aa3)
Cc: stable@vger.kernel.org
|
|
To avoid wraparound if the value is 0.
Signed-off-by: Xiaogang Chen <xiaogang.chen@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
(cherry picked from commit c0cae35661868af207077a4306bc42c7c972947c)
Cc: stable@vger.kernel.org
|
|
Initialize GC IP 11_7_1
Signed-off-by: Granthali Vinodkumar Dhandar <granthali.vinodkumardhandar@amd.com>
Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Initialize GC IP 11_7_0
Signed-off-by: Granthali Vinodkumar Dhandar <granthali.vinodkumardhandar@amd.com>
Reviewed-by: Mario Limonciello <mario.limonciello@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
Running RCCL unit tests on a system with a 64K PAGE_SIZE triggers
the following warning and causes the test to terminate on latest
upstream kernel:
WARNING: drivers/gpu/drm/amd/amdgpu/amdgpu_object.c:1335 at
amdgpu_bo_release_notify+0x1bc/0x280 [amdgpu],
CPU#18: rccl-UnitTests/33151
Call trace:
amdgpu_bo_release_notify
ttm_bo_release
amdgpu_gem_object_free
drm_gem_object_free
amdgpu_bo_unref
amdgpu_bo_create
amdgpu_bo_create_user
amdgpu_gem_object_create
amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu
kfd_ioctl_alloc_memory_of_gpu
kfd_ioctl
sys_ioctl
The warning is triggered because
amdgpu_ttm_next_clear_entity() returns NULL when a clear buffer
operation is requested. This happens because the GART window
allocation for the default_entity, clear_entity and move_entity
fails during initialization.
Commit [1] introduced separate GART windows for the
default_entity, clear_entity and move_entity of each SDMA
instance. Their sizes are derived from
AMDGPU_GTT_MAX_TRANSFER_SIZE, which is currently defined as 1024
pages. This implicitly assumes a 4K PAGE_SIZE, where 1024 pages
correspond to a 4MB transfer. On a 64K PAGE_SIZE system, however,
the same value expands to 64MB.
The default_entity and clear_entity each allocate one
AMDGPU_GTT_MAX_TRANSFER_SIZE GART window, while the move_entity
allocates two such windows. This results in 16MB of GART space
per SDMA instance on a 4K PAGE_SIZE system, but 256MB per SDMA
instance on a 64K PAGE_SIZE system.
On an MI210 system with five SDMA instances and a 512MB GART
aperture, the total GART space required becomes 1.25GB,
exceeding the available GART aperture. Consequently, GART window
allocation fails, amdgpu_ttm_next_clear_entity() returns NULL,
and the above warning is triggered.
Redefine AMDGPU_GTT_MAX_TRANSFER_SIZE in bytes instead of page
units. Where a page count is required, convert it using
PAGE_SHIFT. This preserves the existing 4MB transfer size across
all PAGE_SIZE configurations while keeping GART window
allocations within the available GART aperture.
[1] https://lore.kernel.org/all/20260408100327.1372-3-pierre-eric.pelloux-prayer@amd.com/#t
Closes: https://gitlab.freedesktop.org/drm/amd/-/work_items/5435
Fixes: 897ee11ec020 ("drm/amdgpu: create multiple clear/move ttm entities")
Signed-off-by: Donet Tom <donettom@linux.ibm.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
There were a few instances in kfd_chardev.c of kvzalloc being
used to allocate memory for an array.
Switch those to kvcalloc, which
- is the standard way of allocating a zero-initialized array
- does a check for the mul overflowing
Signed-off-by: David Francis <David.Francis@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
The MES interface takes queue types as enum amdgpu_ring_type values.
The MES backend is responsible for converting those values to
firmware-facing MES_QUEUE_TYPE values when building MES packets.
The KFD queue manager was converting KFD queue types directly to
MES_QUEUE_TYPE values before filling the MES input structures. That is
the wrong abstraction level for the generic MES interface.
Change the KFD helper to return AMDGPU_RING_TYPE_* values and rename it
to make the expected type explicit. Use the helper for the add, remove,
and reset MES paths.
Signed-off-by: Geoffrey McRae <geoffrey.mcrae@amd.com>
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
MES ADD_QUEUE programs the firmware with the queue type from the driver
input, but MES REMOVE_QUEUE leaves queue_type at the zero-initialized
value. Zero decodes as GFX in the MES REMOVE_QUEUE packet.
That means removing a KFD compute queue can be submitted to MES as a GFX
queue. In a debug-trap suspend/remove sequence this can leave MES
looking for the doorbell in the wrong queue class and the REMOVE_QUEUE
command may never complete. The observed failing packet removed
doorbell 0x1002 with queue_type=GFX even though the corresponding
ADD_QUEUE for the same doorbell was queue_type=COMPUTE.
Populate REMOVE_QUEUE.queue_type the same way ADD_QUEUE does.
Signed-off-by: Geoffrey McRae <geoffrey.mcrae@amd.com>
Reviewed-by: Sunil Khatri <sunil.khatri@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
to prevent svm range to be overflow or underflow.
Signed-off-by: Eric Huang <jinhuieric.huang@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
set_queue_properties_from_criu() divided the checkpointed control stack
size by NUM_XCC(adev->gfx.xcc_mask) (whole GPU), while the checkpoint
size was recorded, the MQD buffer allocated, and the control stack
restored using the per-node mask NUM_XCC(mm->dev->xcc_mask). On spatially
partitioned GFX9.4.3 (CPX/QPX) these differ, so the per-XCC control stack
size used for the restore memcpy could exceed the region sized for the
MQD allocation, writing past the BO into adjacent kernel memory; it also
broke legitimate restore on partitioned parts.
Divide by the per-node XCC count so allocation and copy agree,
leaving kfd_queue_acquire_buffers() to bound the size against
the node's advertised control stack size.
Signed-off-by: Yongqiang Sun <Yongqiang.Sun@amd.com>
Reviewed-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
for GFX9.4.2 and above.
-v2: keep APU with GTT allocation
-v3: use dev->adev->apu_prefer_gtt instead
Signed-off-by: James Zhu <James.Zhu@amd.com>
Reviewed-by: Vladimir Indic <vladimir.indic@amd.com>
Reviewed-by: Harish Kasiviswanathan <Harish.Kasiviswanathan@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
This change allows amdgpu_amdkfd_gpuvm_map_bo_to_kernel() to pin buffers
in either GTT or VRAM based on caller specification, providing flexibility
for different memory placement requirements across various kernel buffers.
The domain parameter accepts AMDGPU_GEM_DOMAIN_GTT, AMDGPU_GEM_DOMAIN_VRAM,
or their combination (GTT|VRAM) to let amdgpu_bo_pin() choose the optimal
placement via amdgpu_bo_get_preferred_domain(). This flexible validation
allows callers to specify their preference while delegating final placement
decisions to the driver when appropriate.
CPU visibility is automatically enforced by amdgpu_bo_pin() regardless of
the domain parameter (see amdgpu_bo_pin() line 975-976 which sets
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED for kernel mappings).
-v3: update amdgpu_amdkfd_gpuvm_map_bo_to_kernel description
Signed-off-by: James Zhu <James.Zhu@amd.com>
Reviewed-by: Vladimir Indic <vladimir.indic@amd.com>
Reviewed-by: Philip Yang <philip.yang@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
CRIU checkpoint copies the MQD control stack using cp_hqd_cntl_stack_size
from hardware without bounding it to the allocated BO region. If the HW
field is larger than the queue's control stack allocation, memcpy reads
past the BO into adjacent GTT memory and can leak kernel data to userspace.
Store the page-aligned control stack BO size in mqd_manager and clamp
checkpoint copies and reported checkpoint sizes to
min(cp_hqd_cntl_stack_size, mm->ctl_stack_size). Apply the same bound
for multi-XCC v9.4.3 checkpoint layout.
Signed-off-by: Yongqiang Sun <Yongqiang.Sun@amd.com>
Reviewed-by: David Francis <David.Francis@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
|
This adds queue reset support in KFD topology for gfx12.0.0 and
gfx12.0.1 on non-sriov mode.
Signed-off-by: Amber Lin <amber.lin@amd.com>
Reviewed-by: Shaoyun Liu <shaoyun.liu@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
|
Shikang Fan
Kenji Takahashi
Prike Liang
Philip Yang
Amber Lin
William Palacek
Victor Zhao
Gang Ba
David Francis
Vladimir Marioukhine
Kent Russell
Christian König
Alex Deucher
Guangshuo Li
Srinivasan Shanmugam
Yongqiang Sun
Zhu Lingshan
James Zhu
Dave Airlie
Donet Tom
Granthali Vinodkumar Dhandar
Xiaogang Chen
Geoffrey McRae
Eric Huang