aboutsummaryrefslogtreecommitdiffstats
path: root/drivers/vdpa (follow)
AgeCommit message (Collapse)AuthorFilesLines
2020-10-31Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhostLinus Torvalds2-6/+32
Pull vhost fixes from Michael Tsirkin: "Fixes all over the place. A new UAPI is borderline: can also be considered a new feature but also seems to be the only way we could come up with to fix addressing for userspace - and it seems important to switch to it now before userspace making assumptions about addressing ability of devices is set in stone" * tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost: vdpasim: allow to assign a MAC address vdpasim: fix MAC address configuration vdpa: handle irq bypass register failure case vdpa_sim: Fix DMA mask Revert "vhost-vdpa: fix page pinning leakage in error path" vdpa/mlx5: Fix error return in map_direct_mr() vhost_vdpa: Return -EFAULT if copy_from_user() fails vdpa_sim: implement get_iova_range() vhost: vdpa: report iova range vdpa: introduce config op to get valid iova range
2020-10-30vdpasim: allow to assign a MAC addressLaurent Vivier1-1/+13
Add macaddr parameter to the module to set the MAC address to use Signed-off-by: Laurent Vivier <lvivier@redhat.com> Link: https://lore.kernel.org/r/20201029122050.776445-3-lvivier@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-10-30vdpasim: fix MAC address configurationLaurent Vivier1-1/+2
vdpa_sim generates a ramdom MAC address but it is never used by upper layers because the VIRTIO_NET_F_MAC bit is not set in the features list. Because of that, virtio-net always regenerates a random MAC address each time it is loaded whereas the address should only change on vdpa_sim load/unload. Fix that by adding VIRTIO_NET_F_MAC in the features list of vdpa_sim. Fixes: 2c53d0f64c06 ("vdpasim: vDPA device simulator") Cc: jasowang@redhat.com Signed-off-by: Laurent Vivier <lvivier@redhat.com> Link: https://lore.kernel.org/r/20201029122050.776445-2-lvivier@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-10-30vdpa_sim: Fix DMA maskLaurent Vivier1-1/+3
Since commit f959dcd6ddfd ("dma-direct: Fix potential NULL pointer dereference") an error is reported when we load vdpa_sim and virtio-vdpa: [ 129.351207] net eth0: Unexpected TXQ (0) queue failure: -12 It seems that dma_mask is not initialized. This patch initializes dma_mask() and calls dma_set_mask_and_coherent() to fix the problem. Full log: [ 128.548628] ------------[ cut here ]------------ [ 128.553268] WARNING: CPU: 23 PID: 1105 at kernel/dma/mapping.c:149 dma_map_page_attrs+0x14c/0x1d0 [ 128.562139] Modules linked in: virtio_net net_failover failover virtio_vdpa vdpa_sim vringh vhost_iotlb vdpa xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi rfkill intel_rapl_msr intel_rapl_common isst_if_common sunrpc skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm mgag200 i2c_algo_bit irqbypass drm_kms_helper crct10dif_pclmul crc32_pclmul syscopyarea ghash_clmulni_intel iTCO_wdt sysfillrect iTCO_vendor_support sysimgblt rapl fb_sys_fops dcdbas intel_cstate drm acpi_ipmi ipmi_si mei_me dell_smbios intel_uncore ipmi_devintf mei i2c_i801 dell_wmi_descriptor wmi_bmof pcspkr lpc_ich i2c_smbus ipmi_msghandler acpi_power_meter ip_tables xfs libcrc32c sd_mod t10_pi sg ahci libahci libata megaraid_sas tg3 crc32c_intel wmi dm_mirror dm_region_hash dm_log [ 128.562188] dm_mod [ 128.651334] CPU: 23 PID: 1105 Comm: NetworkManager Tainted: G S I 5.10.0-rc1+ #59 [ 128.659939] Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.8.1 06/30/2020 [ 128.667419] RIP: 0010:dma_map_page_attrs+0x14c/0x1d0 [ 128.672384] Code: 1c 25 28 00 00 00 0f 85 97 00 00 00 48 83 c4 10 5b 5d 41 5c 41 5d c3 4c 89 da eb d7 48 89 f2 48 2b 50 18 48 89 d0 eb 8d 0f 0b <0f> 0b 48 c7 c0 ff ff ff ff eb c3 48 89 d9 48 8b 40 40 e8 2d a0 aa [ 128.691131] RSP: 0018:ffffae0f0151f3c8 EFLAGS: 00010246 [ 128.696357] RAX: ffffffffc06b7400 RBX: 00000000000005fa RCX: 0000000000000000 [ 128.703488] RDX: 0000000000000040 RSI: ffffcee3c7861200 RDI: ffff9e2bc16cd000 [ 128.710620] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000000 [ 128.717754] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9e472cb291f8 [ 128.724886] R13: ffff9e2bc14da780 R14: ffff9e472bc20000 R15: ffff9e2bc1b14940 [ 128.732020] FS: 00007f887bae23c0(0000) GS:ffff9e4ac01c0000(0000) knlGS:0000000000000000 [ 128.740105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 128.745852] CR2: 0000562bc09de998 CR3: 00000003c156c006 CR4: 00000000007706e0 [ 128.752982] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 128.760114] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 128.767247] PKRU: 55555554 [ 128.769961] Call Trace: [ 128.772418] virtqueue_add+0x81e/0xb00 [ 128.776176] virtqueue_add_inbuf_ctx+0x26/0x30 [ 128.780625] try_fill_recv+0x3a2/0x6e0 [virtio_net] [ 128.785509] virtnet_open+0xf9/0x180 [virtio_net] [ 128.790217] __dev_open+0xe8/0x180 [ 128.793620] __dev_change_flags+0x1a7/0x210 [ 128.797808] dev_change_flags+0x21/0x60 [ 128.801646] do_setlink+0x328/0x10e0 [ 128.805227] ? __nla_validate_parse+0x121/0x180 [ 128.809757] ? __nla_parse+0x21/0x30 [ 128.813338] ? inet6_validate_link_af+0x5c/0xf0 [ 128.817871] ? cpumask_next+0x17/0x20 [ 128.821535] ? __snmp6_fill_stats64.isra.54+0x6b/0x110 [ 128.826676] ? __nla_validate_parse+0x47/0x180 [ 128.831120] __rtnl_newlink+0x541/0x8e0 [ 128.834962] ? __nla_reserve+0x38/0x50 [ 128.838713] ? security_sock_rcv_skb+0x2a/0x40 [ 128.843158] ? netlink_deliver_tap+0x2c/0x1e0 [ 128.847518] ? netlink_attachskb+0x1d8/0x220 [ 128.851793] ? skb_queue_tail+0x1b/0x50 [ 128.855641] ? fib6_clean_node+0x43/0x170 [ 128.859652] ? _cond_resched+0x15/0x30 [ 128.863406] ? kmem_cache_alloc_trace+0x3a3/0x420 [ 128.868110] rtnl_newlink+0x43/0x60 [ 128.871602] rtnetlink_rcv_msg+0x12c/0x380 [ 128.875701] ? rtnl_calcit.isra.39+0x110/0x110 [ 128.880147] netlink_rcv_skb+0x50/0x100 [ 128.883987] netlink_unicast+0x1a5/0x280 [ 128.887913] netlink_sendmsg+0x23d/0x470 [ 128.891839] sock_sendmsg+0x5b/0x60 [ 128.895331] ____sys_sendmsg+0x1ef/0x260 [ 128.899255] ? copy_msghdr_from_user+0x5c/0x90 [ 128.903702] ___sys_sendmsg+0x7c/0xc0 [ 128.907369] ? dev_forward_change+0x130/0x130 [ 128.911731] ? sysctl_head_finish.part.29+0x24/0x40 [ 128.916616] ? new_sync_write+0x11f/0x1b0 [ 128.920628] ? mntput_no_expire+0x47/0x240 [ 128.924727] __sys_sendmsg+0x57/0xa0 [ 128.928309] do_syscall_64+0x33/0x40 [ 128.931887] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 128.936937] RIP: 0033:0x7f88792e3857 [ 128.940518] Code: c3 66 90 41 54 41 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 0b ed ff ff 44 89 e2 48 89 ee 89 df 41 89 c0 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 48 89 44 24 08 e8 44 ed ff ff 48 [ 128.959263] RSP: 002b:00007ffdca60dea0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e [ 128.966827] RAX: ffffffffffffffda RBX: 000000000000000c RCX: 00007f88792e3857 [ 128.973960] RDX: 0000000000000000 RSI: 00007ffdca60def0 RDI: 000000000000000c [ 128.981095] RBP: 00007ffdca60def0 R08: 0000000000000000 R09: 0000000000000000 [ 128.988224] R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000000 [ 128.995357] R13: 0000000000000000 R14: 00007ffdca60e0a8 R15: 00007ffdca60e09c [ 129.002492] CPU: 23 PID: 1105 Comm: NetworkManager Tainted: G S I 5.10.0-rc1+ #59 [ 129.011093] Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.8.1 06/30/2020 [ 129.018571] Call Trace: [ 129.021027] dump_stack+0x57/0x6a [ 129.024346] __warn.cold.14+0xe/0x3d [ 129.027925] ? dma_map_page_attrs+0x14c/0x1d0 [ 129.032283] report_bug+0xbd/0xf0 [ 129.035602] handle_bug+0x44/0x80 [ 129.038922] exc_invalid_op+0x13/0x60 [ 129.042589] asm_exc_invalid_op+0x12/0x20 [ 129.046602] RIP: 0010:dma_map_page_attrs+0x14c/0x1d0 [ 129.051566] Code: 1c 25 28 00 00 00 0f 85 97 00 00 00 48 83 c4 10 5b 5d 41 5c 41 5d c3 4c 89 da eb d7 48 89 f2 48 2b 50 18 48 89 d0 eb 8d 0f 0b <0f> 0b 48 c7 c0 ff ff ff ff eb c3 48 89 d9 48 8b 40 40 e8 2d a0 aa [ 129.070311] RSP: 0018:ffffae0f0151f3c8 EFLAGS: 00010246 [ 129.075536] RAX: ffffffffc06b7400 RBX: 00000000000005fa RCX: 0000000000000000 [ 129.082669] RDX: 0000000000000040 RSI: ffffcee3c7861200 RDI: ffff9e2bc16cd000 [ 129.089803] RBP: 0000000000000000 R08: 0000000000000002 R09: 0000000000000000 [ 129.096936] R10: 0000000000000002 R11: 0000000000000000 R12: ffff9e472cb291f8 [ 129.104068] R13: ffff9e2bc14da780 R14: ffff9e472bc20000 R15: ffff9e2bc1b14940 [ 129.111200] virtqueue_add+0x81e/0xb00 [ 129.114952] virtqueue_add_inbuf_ctx+0x26/0x30 [ 129.119399] try_fill_recv+0x3a2/0x6e0 [virtio_net] [ 129.124280] virtnet_open+0xf9/0x180 [virtio_net] [ 129.128984] __dev_open+0xe8/0x180 [ 129.132390] __dev_change_flags+0x1a7/0x210 [ 129.136575] dev_change_flags+0x21/0x60 [ 129.140415] do_setlink+0x328/0x10e0 [ 129.143994] ? __nla_validate_parse+0x121/0x180 [ 129.148528] ? __nla_parse+0x21/0x30 [ 129.152107] ? inet6_validate_link_af+0x5c/0xf0 [ 129.156639] ? cpumask_next+0x17/0x20 [ 129.160306] ? __snmp6_fill_stats64.isra.54+0x6b/0x110 [ 129.165443] ? __nla_validate_parse+0x47/0x180 [ 129.169890] __rtnl_newlink+0x541/0x8e0 [ 129.173731] ? __nla_reserve+0x38/0x50 [ 129.177483] ? security_sock_rcv_skb+0x2a/0x40 [ 129.181928] ? netlink_deliver_tap+0x2c/0x1e0 [ 129.186286] ? netlink_attachskb+0x1d8/0x220 [ 129.190560] ? skb_queue_tail+0x1b/0x50 [ 129.194401] ? fib6_clean_node+0x43/0x170 [ 129.198411] ? _cond_resched+0x15/0x30 [ 129.202163] ? kmem_cache_alloc_trace+0x3a3/0x420 [ 129.206869] rtnl_newlink+0x43/0x60 [ 129.210361] rtnetlink_rcv_msg+0x12c/0x380 [ 129.214462] ? rtnl_calcit.isra.39+0x110/0x110 [ 129.218908] netlink_rcv_skb+0x50/0x100 [ 129.222747] netlink_unicast+0x1a5/0x280 [ 129.226672] netlink_sendmsg+0x23d/0x470 [ 129.230599] sock_sendmsg+0x5b/0x60 [ 129.234090] ____sys_sendmsg+0x1ef/0x260 [ 129.238015] ? copy_msghdr_from_user+0x5c/0x90 [ 129.242461] ___sys_sendmsg+0x7c/0xc0 [ 129.246128] ? dev_forward_change+0x130/0x130 [ 129.250487] ? sysctl_head_finish.part.29+0x24/0x40 [ 129.255368] ? new_sync_write+0x11f/0x1b0 [ 129.259381] ? mntput_no_expire+0x47/0x240 [ 129.263478] __sys_sendmsg+0x57/0xa0 [ 129.267058] do_syscall_64+0x33/0x40 [ 129.270639] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 129.275689] RIP: 0033:0x7f88792e3857 [ 129.279268] Code: c3 66 90 41 54 41 89 d4 55 48 89 f5 53 89 fb 48 83 ec 10 e8 0b ed ff ff 44 89 e2 48 89 ee 89 df 41 89 c0 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 48 89 44 24 08 e8 44 ed ff ff 48 [ 129.298015] RSP: 002b:00007ffdca60dea0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e [ 129.305581] RAX: ffffffffffffffda RBX: 000000000000000c RCX: 00007f88792e3857 [ 129.312712] RDX: 0000000000000000 RSI: 00007ffdca60def0 RDI: 000000000000000c [ 129.319846] RBP: 00007ffdca60def0 R08: 0000000000000000 R09: 0000000000000000 [ 129.326978] R10: 0000000000000001 R11: 0000000000000293 R12: 0000000000000000 [ 129.334109] R13: 0000000000000000 R14: 00007ffdca60e0a8 R15: 00007ffdca60e09c [ 129.341249] ---[ end trace c551e8028fbaf59d ]--- [ 129.351207] net eth0: Unexpected TXQ (0) queue failure: -12 [ 129.360445] net eth0: Unexpected TXQ (0) queue failure: -12 [ 129.824428] net eth0: Unexpected TXQ (0) queue failure: -12 Fixes: 2c53d0f64c06 ("vdpasim: vDPA device simulator") Signed-off-by: Laurent Vivier <lvivier@redhat.com> Link: https://lore.kernel.org/r/20201027175914.689278-1-lvivier@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Cc: stable@vger.kernel.org Acked-by: Jason Wang <jasowang@redhat.com>
2020-10-30vdpa/mlx5: Fix error return in map_direct_mr()Jing Xiangfeng1-3/+2
Fix to return the variable "err" from the error handling case instead of "ret". Fixes: 94abbccdf291 ("vdpa/mlx5: Add shared memory registration code") Signed-off-by: Jing Xiangfeng <jingxiangfeng@huawei.com> Link: https://lore.kernel.org/r/20201026070637.164321-1-jingxiangfeng@huawei.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Eli Cohen <elic@nvidia.com> Cc: stable@vger.kernel.org Acked-by: Jason Wang <jasowang@redhat.com>
2020-10-23Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhostLinus Torvalds1-2/+10
Pull virtio updates from Michael Tsirkin: "vhost, vdpa, and virtio cleanups and fixes A very quiet cycle, no new features" * tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost: MAINTAINERS: add URL for virtio-mem vhost_vdpa: remove unnecessary spin_lock in vhost_vring_call vringh: fix __vringh_iov() when riov and wiov are different vdpa/mlx5: Setup driver only if VIRTIO_CONFIG_S_DRIVER_OK s390: virtio: PV needs VIRTIO I/O device protection virtio: let arch advertise guest's memory access restrictions vhost_vdpa: Fix duplicate included kernel.h vhost: reduce stack usage in log_used virtio-mem: Constify mem_id_table virtio_input: Constify id_table virtio-balloon: Constify id_table vdpa/mlx5: Fix failure to bring link up vdpa/mlx5: Make use of a specific 16 bit endianness API
2020-10-23vdpa_sim: implement get_iova_range()Jason Wang1-0/+12
This implements a sample get_iova_range() for the simulator which advertise [0, ULLONG_MAX] as the valid range. Signed-off-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20201023090043.14430-4-jasowang@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-10-21vdpa/mlx5: Setup driver only if VIRTIO_CONFIG_S_DRIVER_OKEli Cohen1-0/+3
set_map() is used by mlx5 vdpa to create a memory region based on the address map passed by the iotlb argument. If we get successive calls, we will destroy the current memory region and build another one based on the new address mapping. We also need to setup the hardware resources since they depend on the memory region. If these calls happen before DRIVER_OK, It means that driver VQs may also not been setup and we may not create them yet. In this case we want to avoid setting up the other resources and defer this till we get DRIVER OK. Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Eli Cohen <elic@nvidia.com> Link: https://lore.kernel.org/r/20200908123346.GA169007@mtl-vdi-166.wap.labs.mlnx Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-10-21vdpa/mlx5: Fix failure to bring link upEli Cohen1-0/+1
Set VIRTIO_NET_S_LINK_UP in config status to allow the get the bring the net device's link up. Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Eli Cohen <elic@nvidia.com> Link: https://lore.kernel.org/r/20200917121540.GA98184@mtl-vdi-166.wap.labs.mlnx Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-10-21vdpa/mlx5: Make use of a specific 16 bit endianness APIEli Cohen1-2/+6
Introduce a dedicated function to be used for setting 16 bit fields per virio endianness requirements and use it to set the mtu field. Signed-off-by: Eli Cohen <elic@nvidia.com> Link: https://lore.kernel.org/r/20200917121425.GA98139@mtl-vdi-166.wap.labs.mlnx Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-10-15Merge tag 'dma-mapping-5.10' of git://git.infradead.org/users/hch/dma-mappingLinus Torvalds1-1/+1
Pull dma-mapping updates from Christoph Hellwig: - rework the non-coherent DMA allocator - move private definitions out of <linux/dma-mapping.h> - lower CMA_ALIGNMENT (Paul Cercueil) - remove the omap1 dma address translation in favor of the common code - make dma-direct aware of multiple dma offset ranges (Jim Quinlan) - support per-node DMA CMA areas (Barry Song) - increase the default seg boundary limit (Nicolin Chen) - misc fixes (Robin Murphy, Thomas Tai, Xu Wang) - various cleanups * tag 'dma-mapping-5.10' of git://git.infradead.org/users/hch/dma-mapping: (63 commits) ARM/ixp4xx: add a missing include of dma-map-ops.h dma-direct: simplify the DMA_ATTR_NO_KERNEL_MAPPING handling dma-direct: factor out a dma_direct_alloc_from_pool helper dma-direct check for highmem pages in dma_direct_alloc_pages dma-mapping: merge <linux/dma-noncoherent.h> into <linux/dma-map-ops.h> dma-mapping: move large parts of <linux/dma-direct.h> to kernel/dma dma-mapping: move dma-debug.h to kernel/dma/ dma-mapping: remove <asm/dma-contiguous.h> dma-mapping: merge <linux/dma-contiguous.h> into <linux/dma-map-ops.h> dma-contiguous: remove dma_contiguous_set_default dma-contiguous: remove dev_set_cma_area dma-contiguous: remove dma_declare_contiguous dma-mapping: split <linux/dma-mapping.h> cma: decrease CMA_ALIGNMENT lower limit to 2 firewire-ohci: use dma_alloc_pages dma-iommu: implement ->alloc_noncoherent dma-mapping: add new {alloc,free}_noncoherent dma_map_ops methods dma-mapping: add a new dma_alloc_pages API dma-mapping: remove dma_cache_sync 53c700: convert to dma_alloc_noncoherent ...
2020-10-08vdpa/mlx5: Fix dependency on MLX5_COREEli Cohen1-4/+3
Remove propmt for selecting MLX5_VDPA by the user and modify MLX5_VDPA_NET to select MLX5_VDPA. Also modify MLX5_VDPA_NET to depend on mlx5_core. This fixes an issue where configuration sets 'y' for MLX5_VDPA_NET while MLX5_CORE is compiled as a module causing link errors. Reported-by: kernel test robot <lkp@intel.com> Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 device")s Signed-off-by: Eli Cohen <elic@nvidia.com> Link: https://lore.kernel.org/r/20201007064011.GA50074@mtl-vdi-166.wap.labs.mlnx Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-10-08vdpa/mlx5: should keep avail_index despite device statusSi-Wei Liu1-6/+14
A VM with mlx5 vDPA has below warnings while being reset: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11) vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11) We should allow userspace emulating the virtio device be able to get to vq's avail_index, regardless of vDPA device status. Save the index that was last seen when virtq was stopped, so that userspace doesn't complain. Signed-off-by: Si-Wei Liu <si-wei.liu@oracle.com> Link: https://lore.kernel.org/r/1601583511-15138-1-git-send-email-si-wei.liu@oracle.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Eli Cohen <elic@nvidia.com>
2020-10-06dma-mapping: split <linux/dma-mapping.h>Christoph Hellwig1-1/+1
Split out all the bits that are purely for dma_map_ops implementations and related code into a new <linux/dma-map-ops.h> header so that they don't get pulled into all the drivers. That also means the architecture specific <asm/dma-mapping.h> is not pulled in by <linux/dma-mapping.h> any more, which leads to a missing includes that were pulled in by the x86 or arm versions in a few not overly portable drivers. Signed-off-by: Christoph Hellwig <hch@lst.de>
2020-08-26vdpa/mlx5: Avoid warnings about shifts on 32-bit platformsNathan Chancellor1-25/+25
Clang warns several times when building for 32-bit ARM along the lines of: drivers/vdpa/mlx5/net/mlx5_vnet.c:1462:31: warning: shift count >= width of type [-Wshift-count-overflow] ndev->mvdev.mlx_features |= BIT(VIRTIO_F_VERSION_1); ^~~~~~~~~~~~~~~~~~~~~~~ This is related to the BIT macro, which uses an unsigned long literal, which is 32-bit on ARM so having a shift equal to or larger than 32 will cause this warning, such as the above, where VIRTIO_F_VERSION_1 is 32. To avoid this, use BIT_ULL, which will be an unsigned long long. This matches the size of the features field throughout this driver, which is u64 so there should be no functional change. Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Link: https://github.com/ClangBuiltLinux/linux/issues/1140 Signed-off-by: Nathan Chancellor <natechancellor@gmail.com> Link: https://lore.kernel.org/r/20200821225018.940798-1-natechancellor@gmail.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Reported-by: Randy Dunlap <rdunlap@infradead.org> Acked-by: Randy Dunlap <rdunlap@infradead.org> # build-tested Acked-by: Eli Cohen <elic@nvidia.com>
2020-08-26vdpa: ifcvf: free config irq in ifcvf_free_irq()Jason Wang2-3/+4
We don't free config irq in ifcvf_free_irq() which will trigger a BUG() in pci core since we try to free the vectors that has an action. Fixing this by recording the config irq in ifcvf_hw structure and free it in ifcvf_free_irq(). Fixes: e7991f376a4d ("ifcvf: implement config interrupt in IFCVF") Cc: Zhu Lingshan <lingshan.zhu@intel.com> Signed-off-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200723091254.20617-2-jasowang@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Zhu Lingshan <lingshan.zhu@intel.com> Fixes: e7991f376a4d ("ifcvf: implement config interrupt in IFCVF") Cc: Zhu Lingshan <a class="moz-txt-link-rfc2396E" href="mailto:lingshan.zhu@intel.com">&lt;lingshan.zhu@intel.com&gt;</a> Signed-off-by: Jason Wang <a class="moz-txt-link-rfc2396E" href="mailto:jasowang@redhat.com">&lt;jasowang@redhat.com&gt;</a>
2020-08-26vdpa: ifcvf: return err when fail to request config irqJason Wang1-0/+4
We ignore the err of requesting config interrupt, fix this. Fixes: e7991f376a4d ("ifcvf: implement config interrupt in IFCVF") Cc: Zhu Lingshan <lingshan.zhu@intel.com> Signed-off-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200723091254.20617-1-jasowang@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: Zhu Lingshan <lingshan.zhu@intel.com> Fixes: e7991f376a4d ("ifcvf: implement config interrupt in IFCVF") Cc: Zhu Lingshan <a class="moz-txt-link-rfc2396E" href="mailto:lingshan.zhu@intel.com">&lt;lingshan.zhu@intel.com&gt;</a> Signed-off-by: Jason Wang <a class="moz-txt-link-rfc2396E" href="mailto:jasowang@redhat.com">&lt;jasowang@redhat.com&gt;</a> Tested-by: Maxime Coquelin <maxime.coquelin@redhat.com>
2020-08-11Merge tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhostLinus Torvalds15-32/+3264
Pull virtio updates from Michael Tsirkin: - IRQ bypass support for vdpa and IFC - MLX5 vdpa driver - Endianness fixes for virtio drivers - Misc other fixes * tag 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mst/vhost: (71 commits) vdpa/mlx5: fix up endian-ness for mtu vdpa: Fix pointer math bug in vdpasim_get_config() vdpa/mlx5: Fix pointer math in mlx5_vdpa_get_config() vdpa/mlx5: fix memory allocation failure checks vdpa/mlx5: Fix uninitialised variable in core/mr.c vdpa_sim: init iommu lock virtio_config: fix up warnings on parisc vdpa/mlx5: Add VDPA driver for supported mlx5 devices vdpa/mlx5: Add shared memory registration code vdpa/mlx5: Add support library for mlx5 VDPA implementation vdpa/mlx5: Add hardware descriptive header file vdpa: Modify get_vq_state() to return error code net/vdpa: Use struct for set/get vq state vdpa: remove hard coded virtq num vdpasim: support batch updating vhost-vdpa: support IOTLB batching hints vhost-vdpa: support get/set backend features vhost: generialize backend features setting/getting vhost-vdpa: refine ioctl pre-processing vDPA: dont change vq irq after DRIVER_OK ...
2020-08-10vdpa/mlx5: fix up endian-ness for mtuMichael S. Tsirkin1-1/+11
VDPA mlx5 accesses config space as native endian - this is wrong since it's a modern device and actually uses LE. It only supports modern guests so we could punt and just force LE, but let's use the full virtio APIs since people tend to copy/paste code, and this is not data path anyway. Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-10vdpa: Fix pointer math bug in vdpasim_get_config()Dan Carpenter1-1/+1
If "offset" is non-zero then we end up copying from beyond the end of the config because of pointer math. We can fix this by casting the struct to a u8 pointer. Fixes: 2c53d0f64c06 ("vdpasim: vDPA device simulator") Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com> Link: https://lore.kernel.org/r/20200406144552.GF68494@mwanda Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-08-10vdpa/mlx5: Fix pointer math in mlx5_vdpa_get_config()Dan Carpenter1-1/+1
There is a pointer math bug here so if "offset" is non-zero then this will copy memory from beyond the end of the array. Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com> Link: https://lore.kernel.org/r/20200808093241.GB115053@mwanda Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Eli Cohen <elic@nvidia.com> Cc: Jason Wang <jasowang@redhat.com>; Parav Pandit <parav@mellanox.com>; virtualization@lists.linux-foundation.org; linux-kernel@vger.kernel.org; kernel-janitors@vger.kernel.org Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
2020-08-10vdpa/mlx5: fix memory allocation failure checksColin Ian King1-7/+6
The memory allocation failure checking for in and out is currently checking if the pointers are valid rather than the contents of what they point to. Hence the null check on failed memory allocations is incorrect. Fix this by adding the missing indirection in the check. Also for the default case, just set the *in and *out to null as these don't have any thing allocated to kfree. Finally remove the redundant *in and *out check as these have been already done on each allocation in the case statement. Addresses-Coverity: ("Null pointer dereference") Fixes: 1a86b377aa21 ("vdpa/mlx5: Add VDPA driver for supported mlx5 devices") Signed-off-by: Colin Ian King <colin.king@canonical.com> Link: https://lore.kernel.org/r/20200806160828.90463-1-colin.king@canonical.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com> Acked-by: Eli Cohen <eli@mellanox.com>
2020-08-10vdpa/mlx5: Fix uninitialised variable in core/mr.cAlex Dewar1-1/+3
If the kernel is unable to allocate memory for the variable dmr then err will be returned without being set. Set err to -ENOMEM in this case. Fixes: 94abbccdf291 ("vdpa/mlx5: Add shared memory registration code") Addresses-Coverity: ("Uninitialized variables") Signed-off-by: Alex Dewar <alex.dewar@gmx.co.uk> Link: https://lore.kernel.org/r/20200806185625.67344-1-alex.dewar@gmx.co.uk Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com> Acked-by: Eli Cohen <eli@mellanox.com>
2020-08-10vdpa_sim: init iommu lockMichael S. Tsirkin1-0/+1
The patch adding the iommu lock did not initialize it. The struct is zero-initialized so this is mostly a problem when using lockdep. Reported-by: kernel test robot <rong.a.chen@intel.com> Cc: Max Gurtovoy <maxg@mellanox.com> Fixes: 0ea9ee430e74 ("vdpasim: protect concurrent access to iommu iotlb") Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpa/mlx5: Add VDPA driver for supported mlx5 devicesEli Cohen6-2/+2080
Add a front end VDPA driver that registers in the VDPA bus and provides networking to a guest. The VDPA driver creates the necessary resources on the VF it is driving such that data path will be offloaded. Notifications are being communicated through the driver. Currently, only VFs are supported. In subsequent patches we will have devlink support to control which VF is used for VDPA and which function is used for regular networking. Reviewed-by: Parav Pandit <parav@mellanox.com> Signed-off-by: Eli Cohen <eli@mellanox.com> Link: https://lore.kernel.org/r/20200804162048.22587-13-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-08-05vdpa/mlx5: Add shared memory registration codeEli Cohen5-2/+522
Add code to support registering address space region for the device. The virtio driver can run as either: 1. Guest virtio driver 2. Userspace virtio driver on the host 3. Kernel virtio driver on the host In any case a memory key object is required to provide access to memory for the device. This code will be shared by network or block driver implementations. Reviewed-by: Parav Pandit <parav@mellanox.com> Signed-off-by: Eli Cohen <eli@mellanox.com> Link: https://lore.kernel.org/r/20200804162048.22587-12-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpa/mlx5: Add support library for mlx5 VDPA implementationEli Cohen6-0/+350
Following patches introduce VDPA network driver for Mellanox Connectx6 devices. This patch provides functionality that will be used by those patches. Reviewed-by: Parav Pandit <parav@mellanox.com> Signed-off-by: Eli Cohen <eli@mellanox.com> Link: https://lore.kernel.org/r/20200804162048.22587-11-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpa/mlx5: Add hardware descriptive header fileEli Cohen1-0/+168
Keep all vdpa related hardware definitions in this file. Reviewed-by: Parav Pandit <parav@mellanox.com> Signed-off-by: Eli Cohen <eli@mellanox.com> Link: https://lore.kernel.org/r/20200804162048.22587-10-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpa: Modify get_vq_state() to return error codeEli Cohen2-4/+6
Modify get_vq_state() so it returns an error code. In case of hardware acceleration, the available index may be retrieved from the device, an operation that can possibly fail. Reviewed-by: Parav Pandit <parav@mellanox.com> Signed-off-by: Eli Cohen <eli@mellanox.com> Link: https://lore.kernel.org/r/20200804162048.22587-9-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-08-05net/vdpa: Use struct for set/get vq stateEli Cohen4-12/+15
For now VQ state involves 16 bit available index value encoded in u64 variable. In the future it will be extended to contain more fields. Use struct to contain the state, now containing only a single u16 for the available index. In the future we can add fields to this struct. Reviewed-by: Parav Pandit <parav@mellanox.com> Acked-by: Jason Wang <jasowang@redhat.com> Signed-off-by: Eli Cohen <eli@mellanox.com> Link: https://lore.kernel.org/r/20200804162048.22587-8-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpa: remove hard coded virtq numMax Gurtovoy3-3/+7
This will enable vdpa providers to add support for multi queue feature and publish it to upper layers (vhost and virtio). Signed-off-by: Max Gurtovoy <maxg@mellanox.com> Reviewed-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200804162048.22587-7-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpasim: support batch updatingJason Wang1-3/+37
The vDPA simulator support both set_map() and dma_map()/dma_unmap() operations. But vhost-vdpa can only use one of them. So this patch introduce a module parameter (batch_mapping) that let vpda_sim to support only one of those dma operations. The batched mapping via set_map() is enabled by default. Signed-off-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200804162048.22587-6-eli@mellanox.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05ifcvf: implement vdpa_config_ops.get_vq_irq()Zhu Lingshan1-2/+16
This commit implemented vdpa_config_ops.get_vq_irq() in ifcvf, and initialized vq irq to -EINVAL. So that ifcvf can report irq number of a vq, or -EINVAL if the vq is not assigned an irq number. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> Suggested-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200731065533.4144-6-lingshan.zhu@intel.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpasim: protect concurrent access to iommu iotlbMax Gurtovoy1-4/+27
Iommu iotlb can be accessed by different cores for performing IO using multiple virt queues. Add a spinlock to synchronize iotlb accesses. This could be easily reproduced when using more than 1 pktgen threads to inject traffic to vdpa simulator. Fixes: 2c53d0f64c06f("vdpasim: vDPA device simulator") Cc: stable@vger.kernel.org Signed-off-by: Max Gurtovoy <maxg@mellanox.com> Signed-off-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200731073822.13326-1-jasowang@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpa_sim: fix endian-ness of config spaceMichael S. Tsirkin1-5/+28
VDPA sim accesses config space as native endian - this is wrong since it's a modern device and actually uses LE. It only supports modern guests so we could punt and just force LE, but let's use the full virtio APIs since people tend to copy/paste code, and this is not data path anyway. Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-05vdpa: make sure set_features is invoked for legacyMichael S. Tsirkin1-0/+1
Some legacy guests just assume features are 0 after reset. We detect that config space is accessed before features are set and set features to 0 automatically. Note: some legacy guests might not even access config space, if this is reported in the field we might need to catch a kick to handle these. Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-08-03virtio: VIRTIO_F_IOMMU_PLATFORM -> VIRTIO_F_ACCESS_PLATFORMMichael S. Tsirkin2-3/+3
Rename the bit to match latest virtio spec. Add a compat macro to avoid breaking existing userspace. Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Reviewed-by: David Hildenbrand <david@redhat.com>
2020-07-19dma-mapping: make support for dma ops optionalChristoph Hellwig1-0/+1
Avoid the overhead of the dma ops support for tiny builds that only use the direct mapping. Signed-off-by: Christoph Hellwig <hch@lst.de> Tested-by: Alexey Kardashevskiy <aik@ozlabs.ru> Reviewed-by: Alexey Kardashevskiy <aik@ozlabs.ru>
2020-06-22vdpa: fix typos in the comments for __vdpa_alloc_device()Jason Wang1-1/+1
Fix two typos in the comments for __vdpa_alloc_device(). Signed-off-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200527060528.9100-1-jasowang@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-06-06ifcvf: implement config interrupt in IFCVFZhu Lingshan3-1/+29
This commit implements config interrupt support in IFC VF Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> Acked-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/1591352835-22441-6-git-send-email-lingshan.zhu@intel.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-06-06ifcvf: ignore continuous setting same status valueZhu Lingshan1-0/+3
User space may try to set status of same value multiple times, this patch handles this case more efficiently by ignoring the setting of the same status value. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> Link: https://lore.kernel.org/r/1591352835-22441-3-git-send-email-lingshan.zhu@intel.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-06-04vdpasim: Fix some coccinelle warningsSamuel Zou1-3/+4
Fix below warnings reported by coccicheck: drivers/vdpa/vdpa_sim/vdpa_sim.c:104:1-10: WARNING: Assignment of 0/1 to bool variable drivers/vdpa/vdpa_sim/vdpa_sim.c:164:7-11: WARNING: Unsigned expression compared with zero: read <= 0 drivers/vdpa/vdpa_sim/vdpa_sim.c:169:7-12: WARNING: Unsigned expression compared with zero: write <= 0 1. The 'ready' variable in vdpasim_virtqueue struct is bool type. It is better to initialize vq->ready to false 2. Modify 'read' and 'write' variables type from size_t to ssize_t. And preserve the reverse christmas tree ordering of local variables. Fixes: 2c53d0f64c06 ("vdpasim: vDPA device simulator") Reported-by: Hulk Robot <hulkci@huawei.com> Signed-off-by: Samuel Zou <zou_wei@huawei.com> Link: https://lore.kernel.org/r/1588990802-28451-1-git-send-email-zou_wei@huawei.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-06-04ifcvf: move IRQ request/free to status change handlersZhu Lingshan1-47/+73
This commit move IRQ request and free operations from probe() to VIRTIO status change handler to comply with VIRTIO spec. VIRTIO spec 1.1, section 2.1.2 Device Requirements: Device Status Field The device MUST NOT consume buffers or send any used buffer notifications to the driver before DRIVER_OK. Signed-off-by: Zhu Lingshan <lingshan.zhu@intel.com> Link: https://lore.kernel.org/r/1589270444-3669-1-git-send-email-lingshan.zhu@intel.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-06-02vhost: revert "vhost: disable for OABI"Michael S. Tsirkin1-1/+1
This reverts commit d085eb8ce727 ("vhost: disable for OABI") With commit "virtio: force spec specified alignment on types" in place, we force proper alignment for all structures, so there's no longer a reason to blacklist OABI. Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-05-11vdpasim: remove unused variable 'ret'YueHaibing1-8/+7
drivers/vdpa/vdpa_sim/vdpa_sim.c:92:6: warning: variable ‘ret’ set but not used [-Wunused-but-set-variable] Signed-off-by: YueHaibing <yuehaibing@huawei.com> Link: https://lore.kernel.org/r/20200410115422.42308-1-yuehaibing@huawei.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-04-20vhost: disable for OABIMichael S. Tsirkin1-1/+1
vhost is currently broken on the some ARM configs. The reason is that the ring element addresses are passed between components with different alignments assumptions. Thus, if guest selects a pointer and host then gets and dereferences it, then alignment assumed by the host's compiler might be greater than the actual alignment of the pointer. compiler on the host from assuming pointer is aligned. This actually triggers on ARM with -mabi=apcs-gnu - which is a deprecated configuration. With this OABI, compiler assumes that all structures are 4 byte aligned - which is stronger than virtio guarantees for available and used rings, which are merely 2 bytes. Thus a guest without -mabi=apcs-gnu running on top of host with -mabi=apcs-gnu will be broken. The correct fix is to force alignment of structures - however that is an intrusive fix that's best deferred until the next release. We didn't previously support such ancient systems at all - this surfaced after vdpa support prompted removing dependency of vhost on VIRTULIZATION. So for now, let's just add something along the lines of depends on !ARM || AEABI to the virtio Kconfig declaration, and add a comment that it has to do with struct member alignment. Note: we can't make VHOST and VHOST_RING themselves have a dependency since these are selected. Add a new symbol for that. We should be able to drop this dependency down the road. Fixes: 20c384f1ea1a0bc7 ("vhost: refine vhost and vringh kconfig") Suggested-by: Ard Biesheuvel <ardb@kernel.org> Suggested-by: Richard Earnshaw <Richard.Earnshaw@arm.com> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-04-17vdpa: fix comment of vdpa_register_device()Jason Wang1-1/+1
The function should be called only after a success calling for vdpa_alloc_device(). Reported-by: Eli Cohen <eli@mellanox.com> Signed-off-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200413093738.28467-1-jasowang@redhat.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-04-17vdpa: make vhost, virtio depend on menuMichael S. Tsirkin1-11/+5
If user did not configure any vdpa drivers, neither vhost nor virtio vdpa are going to be useful. So there's no point in prompting for these and selecting vdpa core automatically. Simplify configuration by making virtio and vhost vdpa drivers depend on vdpa menu entry. Once done, we no longer need a separate menu entry, so also get rid of this. While at it, fix up the IFC entry: VDPA->vDPA for consistency with other places. Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Acked-by: Jason Wang <jasowang@redhat.com>
2020-04-17vdpa: allow a 32 bit vq alignmentMichael S. Tsirkin2-2/+2
get_vq_align returns u16 now, but that's not enough for systems/devices with 64K pages. All callers assign it to a u32 variable anyway, so let's just change the return value type to u32. Reported-by: Arnd Bergmann <arnd@arndb.de> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
2020-04-16vdpasim: Return status in vdpasim_get_statusYueHaibing1-1/+1
vdpasim->status should acquired under spin lock. Fixes: 870448c31775 ("vdpasim: vDPA device simulator") Signed-off-by: YueHaibing <yuehaibing@huawei.com> Acked-by: Jason Wang <jasowang@redhat.com> Link: https://lore.kernel.org/r/20200402024916.35192-1-yuehaibing@huawei.com Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
Copyright © 1996 – 2023 Jason A. Donenfeld. All Rights Reverse Engineered.