KekenoBSD/src
3
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Import the Hardware Trace (HWT) framework.

Browse Source 
The HWT framework provides infrastructure for hardware-assisted tracing. It
collects detailed information about software execution and records it as
"events" in highly compressed format into DRAM. The events cover information
about control flow changes of a program, whether branches taken or not,
exceptions taken, timing information, cycles elapsed and more. This allows
to reconstruct entire program flow of a given application.

This comes with separate machine-dependent tracing backends for trace
collection, trace decoder libraries and an instrumentation tool.

Reviewed by:	kib (sys/kern bits)
Sponsored by:	UKRI
Differential Revision:	https://reviews.freebsd.org/D40466
This commit is contained in:
Ruslan Bukin
2025-07-03 14:10:45 +01:00
parent 9d004de910
commit df114daef4
42 changed files with 4150 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sys/conf/files
+13
View File
@@ -1776,6 +1776,19 @@ dev/hwpmc/hwpmc_soft.c optional hwpmc
dev/hwreset/hwreset.c optional hwreset
dev/hwreset/hwreset_array.c optional hwreset
dev/hwreset/hwreset_if.m optional hwreset
dev/hwt/hwt.c optional hwt
dev/hwt/hwt_backend.c optional hwt
dev/hwt/hwt_config.c optional hwt
dev/hwt/hwt_context.c optional hwt
dev/hwt/hwt_contexthash.c optional hwt
dev/hwt/hwt_cpu.c optional hwt
dev/hwt/hwt_hook.c optional hwt
dev/hwt/hwt_ioctl.c optional hwt
dev/hwt/hwt_owner.c optional hwt
dev/hwt/hwt_ownerhash.c optional hwt
dev/hwt/hwt_record.c optional hwt
dev/hwt/hwt_thread.c optional hwt
dev/hwt/hwt_vm.c optional hwt
dev/ichiic/ig4_acpi.c optional ig4 acpi iicbus
dev/ichiic/ig4_iic.c optional ig4 iicbus
dev/ichiic/ig4_pci.c optional ig4 pci iicbus
sys/conf/options
+3
View File
@@ -885,6 +885,9 @@ DCONS_FORCE_GDB opt_dcons.h
HWPMC_DEBUG opt_global.h
HWPMC_HOOKS
# Hardware Trace (HWT) framework options
HWT_HOOKS
# 802.11 support layer
IEEE80211_DEBUG opt_wlan.h
IEEE80211_DEBUG_REFCNT opt_wlan.h
sys/dev/hwt/hwt.c
+242
View File
@@ -0,0 +1,242 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
/*
* Hardware Tracing framework.
*
* The framework manages hardware tracing units that collect information
* about software execution and store it as events in highly compressed format
* into DRAM. The events cover information about control flow changes of a
* program, whether branches taken or not, exceptions taken, timing information,
* cycles elapsed and more. That allows us to restore entire program flow of a
* given application without performance impact.
*
* Design overview.
*
* The framework provides character devices for mmap(2) and ioctl(2) system
* calls to allow user to manage CPU (hardware) tracing units.
*
* /dev/hwt:
* .ioctl:
* hwt_ioctl():
* a) HWT_IOC_ALLOC
* Allocates kernel tracing context CTX based on requested mode
* of operation. Verifies the information that comes with the
* request (pid, cpus), allocates unique ID for the context.
* Creates a new character device for CTX management.
*
* /dev/hwt_%d[_%d], ident[, thread_id]
* .mmap
* Maps tracing buffers of the corresponding thread to userspace.
* .ioctl
* hwt_thread_ioctl():
* a) HWT_IOC_START
* Enables tracing unit for a given context.
* b) HWT_IOC_RECORD_GET
* Transfers (small) record entries collected during program
* execution for a given context to userspace, such as mmaping
* tables of executable and dynamic libraries, interpreter,
* kernel mappings, tid of threads created, etc.
* c) HWT_IOC_SET_CONFIG
* Allows to specify backend-specific configuration of the
* trace unit.
* d) HWT_IOC_WAKEUP
* Wakes up a thread that is currently sleeping.
* e) HWT_IOC_BUFPTR_GET
* Transfers current hardware pointer in the filling buffer
* to the userspace.
* f) HWT_IOC_SVC_BUF
* To avoid data loss, userspace may notify kernel it has
* copied out the given buffer, so kernel is ok to overwrite
*
* HWT context lifecycle in THREAD mode of operation:
* 1. User invokes HWT_IOC_ALLOC ioctl with information about pid to trace and
* size of the buffers for the trace data to allocate.
* Some architectures may have different tracing units supported, so user
* also provides backend name to use for this context, e.g. "coresight".
* 2. Kernel allocates context, lookups the proc for the given pid. Then it
* creates first hwt_thread in the context and allocates trace buffers for
* it. Immediately, kernel initializes tracing backend.
* Kernel creates character device and returns unique identificator of
* trace context to the user.
* 3. To manage the new context, user opens the character device created.
* User invokes HWT_IOC_START ioctl, kernel marks context as RUNNING.
* At this point any HWT hook invocation by scheduler enables/disables
* tracing for threads associated with the context (threads of the proc).
* Any new threads creation (of the target proc) procedures will be invoking
* corresponding hooks in HWT framework, so that new hwt_thread and buffers
* allocated, character device for mmap(2) created on the fly.
* 4. User issues HWT_IOC_RECORD_GET ioctl to fetch information about mmaping
* tables and threads created during application startup.
* 5. User mmaps tracing buffers of each thread to userspace (using
* /dev/hwt_%d_%d % (ident, thread_id) character devices).
* 6. User can repeat 4 if expected thread is not yet created during target
* application execution.
* 7. User issues HWT_IOC_BUFPTR_GET ioctl to get current filling level of the
* hardware buffer of a given thread.
* 8. User invokes trace decoder library to process available data and see the
* results in human readable form.
* 9. User repeats 7 if needed.
*
* HWT context lifecycle in CPU mode of operation:
* 1. User invokes HWT_IOC_ALLOC ioctl providing a set of CPU to trace within
* single CTX.
* 2. Kernel verifies the set of CPU and allocates tracing context, creates
* a buffer for each CPU.
* Kernel creates a character device for every CPU provided in the request.
* Kernel initialized tracing backend.
* 3. User opens character devices of interest to map the buffers to userspace.
* User can start tracing by invoking HWT_IOC_START on any of character
* device within the context, entire context will be marked as RUNNING.
* 4. The rest is similar to the THREAD mode.
*
*/
#include <sys/param.h>
#include <sys/conf.h>
#include <sys/eventhandler.h>
#include <sys/kernel.h>
#include <sys/module.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_ownerhash.h>
#include <dev/hwt/hwt_backend.h>
#include <dev/hwt/hwt_record.h>
#include <dev/hwt/hwt_ioctl.h>
#include <dev/hwt/hwt_hook.h>
#define HWT_DEBUG
#undef HWT_DEBUG
#ifdef HWT_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static eventhandler_tag hwt_exit_tag;
static struct cdev *hwt_cdev;
static struct cdevsw hwt_cdevsw = {
.d_version = D_VERSION,
.d_name = "hwt",
.d_mmap_single = NULL,
.d_ioctl = hwt_ioctl
};
static void
hwt_process_exit(void *arg __unused, struct proc *p)
{
struct hwt_owner *ho;
/* Stop HWTs associated with exiting owner, if any. */
ho = hwt_ownerhash_lookup(p);
if (ho)
hwt_owner_shutdown(ho);
}
static int
hwt_load(void)
{
struct make_dev_args args;
int error;
make_dev_args_init(&args);
args.mda_devsw = &hwt_cdevsw;
args.mda_flags = MAKEDEV_CHECKNAME | MAKEDEV_WAITOK;
args.mda_uid = UID_ROOT;
args.mda_gid = GID_WHEEL;
args.mda_mode = 0660;
args.mda_si_drv1 = NULL;
hwt_backend_load();
hwt_ctx_load();
hwt_contexthash_load();
hwt_ownerhash_load();
hwt_record_load();
error = make_dev_s(&args, &hwt_cdev, "hwt");
if (error != 0)
return (error);
hwt_exit_tag = EVENTHANDLER_REGISTER(process_exit, hwt_process_exit,
NULL, EVENTHANDLER_PRI_ANY);
hwt_hook_load();
return (0);
}
static int
hwt_unload(void)
{
hwt_hook_unload();
EVENTHANDLER_DEREGISTER(process_exit, hwt_exit_tag);
destroy_dev(hwt_cdev);
hwt_record_unload();
hwt_ownerhash_unload();
hwt_contexthash_unload();
hwt_ctx_unload();
hwt_backend_unload();
return (0);
}
static int
hwt_modevent(module_t mod, int type, void *data)
{
int error;
switch (type) {
case MOD_LOAD:
error = hwt_load();
break;
case MOD_UNLOAD:
error = hwt_unload();
break;
default:
error = 0;
break;
}
return (error);
}
static moduledata_t hwt_mod = {
"hwt",
hwt_modevent,
NULL
};
DECLARE_MODULE(hwt, hwt_mod, SI_SUB_DRIVERS, SI_ORDER_FIRST);
MODULE_VERSION(hwt, 1);
sys/dev/hwt/hwt_backend.c
+289
View File
@@ -0,0 +1,289 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
/* Hardware Trace (HWT) framework. */
#include <sys/param.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mutex.h>
#include <sys/hwt.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_backend.h>
#define HWT_BACKEND_DEBUG
#undef HWT_BACKEND_DEBUG
#ifdef HWT_BACKEND_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static struct mtx hwt_backend_mtx;
struct hwt_backend_entry {
struct hwt_backend *backend;
LIST_ENTRY(hwt_backend_entry) next;
};
static LIST_HEAD(, hwt_backend_entry) hwt_backends;
static MALLOC_DEFINE(M_HWT_BACKEND, "hwt_backend", "HWT backend");
int
hwt_backend_init(struct hwt_context *ctx)
{
int error;
dprintf("%s\n", __func__);
error = ctx->hwt_backend->ops->hwt_backend_init(ctx);
return (error);
}
void
hwt_backend_deinit(struct hwt_context *ctx)
{
dprintf("%s\n", __func__);
ctx->hwt_backend->ops->hwt_backend_deinit(ctx);
}
int
hwt_backend_configure(struct hwt_context *ctx, int cpu_id, int thread_id)
{
int error;
dprintf("%s\n", __func__);
error = ctx->hwt_backend->ops->hwt_backend_configure(ctx, cpu_id,
thread_id);
return (error);
}
void
hwt_backend_enable(struct hwt_context *ctx, int cpu_id)
{
dprintf("%s\n", __func__);
ctx->hwt_backend->ops->hwt_backend_enable(ctx, cpu_id);
}
void
hwt_backend_disable(struct hwt_context *ctx, int cpu_id)
{
dprintf("%s\n", __func__);
ctx->hwt_backend->ops->hwt_backend_disable(ctx, cpu_id);
}
void
hwt_backend_enable_smp(struct hwt_context *ctx)
{
dprintf("%s\n", __func__);
ctx->hwt_backend->ops->hwt_backend_enable_smp(ctx);
}
void
hwt_backend_disable_smp(struct hwt_context *ctx)
{
dprintf("%s\n", __func__);
ctx->hwt_backend->ops->hwt_backend_disable_smp(ctx);
}
void __unused
hwt_backend_dump(struct hwt_context *ctx, int cpu_id)
{
dprintf("%s\n", __func__);
ctx->hwt_backend->ops->hwt_backend_dump(cpu_id);
}
int
hwt_backend_read(struct hwt_context *ctx, struct hwt_vm *vm, int *ident,
vm_offset_t *offset, uint64_t *data)
{
int error;
dprintf("%s\n", __func__);
error = ctx->hwt_backend->ops->hwt_backend_read(vm, ident,
offset, data);
return (error);
}
struct hwt_backend *
hwt_backend_lookup(const char *name)
{
struct hwt_backend_entry *entry;
struct hwt_backend *backend;
HWT_BACKEND_LOCK();
LIST_FOREACH(entry, &hwt_backends, next) {
backend = entry->backend;
if (strcmp(backend->name, name) == 0) {
HWT_BACKEND_UNLOCK();
return (backend);
}
}
HWT_BACKEND_UNLOCK();
return (NULL);
}
int
hwt_backend_register(struct hwt_backend *backend)
{
struct hwt_backend_entry *entry;
if (backend == NULL ||
backend->name == NULL ||
backend->ops == NULL)
return (EINVAL);
entry = malloc(sizeof(struct hwt_backend_entry), M_HWT_BACKEND,
M_WAITOK | M_ZERO);
entry->backend = backend;
HWT_BACKEND_LOCK();
LIST_INSERT_HEAD(&hwt_backends, entry, next);
HWT_BACKEND_UNLOCK();
return (0);
}
int
hwt_backend_unregister(struct hwt_backend *backend)
{
struct hwt_backend_entry *entry, *tmp;
if (backend == NULL)
return (EINVAL);
/* TODO: check if not in use */
HWT_BACKEND_LOCK();
LIST_FOREACH_SAFE(entry, &hwt_backends, next, tmp) {
if (entry->backend == backend) {
LIST_REMOVE(entry, next);
HWT_BACKEND_UNLOCK();
free(entry, M_HWT_BACKEND);
return (0);
}
}
HWT_BACKEND_UNLOCK();
return (ENOENT);
}
void
hwt_backend_load(void)
{
mtx_init(&hwt_backend_mtx, "hwt backend", NULL, MTX_DEF);
LIST_INIT(&hwt_backends);
}
void
hwt_backend_unload(void)
{
/* TODO: ensure all unregistered */
mtx_destroy(&hwt_backend_mtx);
}
void
hwt_backend_stop(struct hwt_context *ctx)
{
dprintf("%s\n", __func__);
ctx->hwt_backend->ops->hwt_backend_stop(ctx);
}
int
hwt_backend_svc_buf(struct hwt_context *ctx, void *data, size_t data_size,
int data_version)
{
int error;
dprintf("%s\n", __func__);
error = ctx->hwt_backend->ops->hwt_backend_svc_buf(ctx, data, data_size,
data_version);
return (error);
}
int
hwt_backend_thread_alloc(struct hwt_context *ctx, struct hwt_thread *thr)
{
int error;
dprintf("%s\n", __func__);
if (ctx->hwt_backend->ops->hwt_backend_thread_alloc == NULL)
return (0);
KASSERT(thr->private == NULL,
("%s: thread private data is not NULL\n", __func__));
error = ctx->hwt_backend->ops->hwt_backend_thread_alloc(thr);
return (error);
}
void
hwt_backend_thread_free(struct hwt_thread *thr)
{
dprintf("%s\n", __func__);
if (thr->backend->ops->hwt_backend_thread_free == NULL)
return;
KASSERT(thr->private != NULL,
("%s: thread private data is NULL\n", __func__));
thr->backend->ops->hwt_backend_thread_free(thr);
return;
}
sys/dev/hwt/hwt_backend.h
+87
View File
@@ -0,0 +1,87 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_BACKEND_H_
#define _DEV_HWT_HWT_BACKEND_H_
struct hwt_backend_ops {
int (*hwt_backend_init)(struct hwt_context *);
int (*hwt_backend_deinit)(struct hwt_context *);
int (*hwt_backend_configure)(struct hwt_context *, int cpu_id,
int thread_id);
int (*hwt_backend_svc_buf)(struct hwt_context *, void *data,
size_t data_size, int data_version);
void (*hwt_backend_enable)(struct hwt_context *, int cpu_id);
void (*hwt_backend_disable)(struct hwt_context *, int cpu_id);
int (*hwt_backend_read)(struct hwt_vm *, int *ident,
vm_offset_t *offset, uint64_t *data);
void (*hwt_backend_stop)(struct hwt_context *);
/* For backends that are tied to local CPU registers */
int (*hwt_backend_enable_smp)(struct hwt_context *);
int (*hwt_backend_disable_smp)(struct hwt_context *);
/* Allocation and initialization of backend-specific thread data. */
int (*hwt_backend_thread_alloc)(struct hwt_thread *);
void (*hwt_backend_thread_free)(struct hwt_thread *);
/* Debugging only. */
void (*hwt_backend_dump)(int cpu_id);
};
struct hwt_backend {
const char *name;
struct hwt_backend_ops *ops;
/* buffers require kernel virtual addresses */
bool kva_req;
};
int hwt_backend_init(struct hwt_context *ctx);
void hwt_backend_deinit(struct hwt_context *ctx);
int hwt_backend_configure(struct hwt_context *ctx, int cpu_id, int thread_id);
void hwt_backend_enable(struct hwt_context *ctx, int cpu_id);
void hwt_backend_disable(struct hwt_context *ctx, int cpu_id);
void hwt_backend_enable_smp(struct hwt_context *ctx);
void hwt_backend_disable_smp(struct hwt_context *ctx);
void hwt_backend_dump(struct hwt_context *ctx, int cpu_id);
int hwt_backend_read(struct hwt_context *ctx, struct hwt_vm *vm, int *ident,
vm_offset_t *offset, uint64_t *data);
int hwt_backend_register(struct hwt_backend *);
int hwt_backend_unregister(struct hwt_backend *);
void hwt_backend_stop(struct hwt_context *);
int hwt_backend_svc_buf(struct hwt_context *ctx, void *data, size_t data_size,
int data_version);
struct hwt_backend * hwt_backend_lookup(const char *name);
int hwt_backend_thread_alloc(struct hwt_context *ctx, struct hwt_thread *);
void hwt_backend_thread_free(struct hwt_thread *);
void hwt_backend_load(void);
void hwt_backend_unload(void);
#define HWT_BACKEND_LOCK() mtx_lock(&hwt_backend_mtx)
#define HWT_BACKEND_UNLOCK() mtx_unlock(&hwt_backend_mtx)
#endif /* !_DEV_HWT_HWT_BACKEND_H_ */
sys/dev/hwt/hwt_config.c
+108
View File
@@ -0,0 +1,108 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/proc.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mutex.h>
#include <sys/lock.h>
#include <sys/hwt.h>
#include <vm/vm.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_record.h>
#define HWT_MAXCONFIGSIZE PAGE_SIZE
#define HWT_CONFIG_DEBUG
#undef HWT_CONFIG_DEBUG
#ifdef HWT_CONFIG_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static MALLOC_DEFINE(M_HWT_CONFIG, "hwt_config", "HWT config");
int
hwt_config_set(struct thread *td, struct hwt_context *ctx,
struct hwt_set_config *sconf)
{
size_t config_size;
void *old_config;
void *config;
int error;
config_size = sconf->config_size;
if (config_size == 0)
return (0);
if (config_size > HWT_MAXCONFIGSIZE)
return (EFBIG);
config = malloc(config_size, M_HWT_CONFIG, M_WAITOK | M_ZERO);
error = copyin(sconf->config, config, config_size);
if (error) {
free(config, M_HWT_CONFIG);
return (error);
}
HWT_CTX_LOCK(ctx);
old_config = ctx->config;
ctx->config = config;
ctx->config_size = sconf->config_size;
ctx->config_version = sconf->config_version;
HWT_CTX_UNLOCK(ctx);
if (old_config != NULL)
free(old_config, M_HWT_CONFIG);
return (error);
}
void
hwt_config_free(struct hwt_context *ctx)
{
if (ctx->config == NULL)
return;
free(ctx->config, M_HWT_CONFIG);
ctx->config = NULL;
}
sys/dev/hwt/hwt_config.h
+36
View File
@@ -0,0 +1,36 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_CONFIG_H_
#define _DEV_HWT_HWT_CONFIG_H_
int hwt_config_set(struct thread *td, struct hwt_context *ctx,
struct hwt_set_config *sconf);
void hwt_config_free(struct hwt_context *ctx);
#endif /* !_DEV_HWT_HWT_CONFIG_H_ */
sys/dev/hwt/hwt_context.c
+201
View File
@@ -0,0 +1,201 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/bitstring.h>
#include <sys/conf.h>
#include <sys/proc.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mman.h>
#include <sys/mutex.h>
#include <sys/refcount.h>
#include <sys/rwlock.h>
#include <sys/hwt.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_vm.h>
#include <dev/hwt/hwt_cpu.h>
#define HWT_DEBUG
#undef HWT_DEBUG
#ifdef HWT_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static MALLOC_DEFINE(M_HWT_CTX, "hwt_ctx", "Hardware Trace");
static bitstr_t *ident_set;
static int ident_set_size;
static struct mtx ident_set_mutex;
static int
hwt_ctx_ident_alloc(int *new_ident)
{
mtx_lock(&ident_set_mutex);
bit_ffc(ident_set, ident_set_size, new_ident);
if (*new_ident == -1) {
mtx_unlock(&ident_set_mutex);
return (ENOMEM);
}
bit_set(ident_set, *new_ident);
mtx_unlock(&ident_set_mutex);
return (0);
}
static void
hwt_ctx_ident_free(int ident)
{
mtx_lock(&ident_set_mutex);
bit_clear(ident_set, ident);
mtx_unlock(&ident_set_mutex);
}
int
hwt_ctx_alloc(struct hwt_context **ctx0)
{
struct hwt_context *ctx;
int error;
ctx = malloc(sizeof(struct hwt_context), M_HWT_CTX, M_WAITOK | M_ZERO);
TAILQ_INIT(&ctx->records);
TAILQ_INIT(&ctx->threads);
TAILQ_INIT(&ctx->cpus);
mtx_init(&ctx->mtx, "ctx", NULL, MTX_SPIN);
mtx_init(&ctx->rec_mtx, "ctx_rec", NULL, MTX_DEF);
refcount_init(&ctx->refcnt, 0);
error = hwt_ctx_ident_alloc(&ctx->ident);
if (error) {
printf("could not allocate ident bit str\n");
return (error);
}
*ctx0 = ctx;
return (0);
}
static void
hwt_ctx_free_cpus(struct hwt_context *ctx)
{
struct hwt_cpu *cpu;
do {
HWT_CTX_LOCK(ctx);
cpu = TAILQ_FIRST(&ctx->cpus);
if (cpu)
TAILQ_REMOVE(&ctx->cpus, cpu, next);
HWT_CTX_UNLOCK(ctx);
if (cpu == NULL)
break;
/* TODO: move vm_free() to cpu_free()? */
hwt_vm_free(cpu->vm);
hwt_cpu_free(cpu);
} while (1);
}
static void
hwt_ctx_free_threads(struct hwt_context *ctx)
{
struct hwt_thread *thr;
dprintf("%s: remove threads\n", __func__);
do {
HWT_CTX_LOCK(ctx);
thr = TAILQ_FIRST(&ctx->threads);
if (thr)
TAILQ_REMOVE(&ctx->threads, thr, next);
HWT_CTX_UNLOCK(ctx);
if (thr == NULL)
break;
HWT_THR_LOCK(thr);
/* TODO: check if thr is sleeping before waking it up. */
wakeup(thr);
HWT_THR_UNLOCK(thr);
if (refcount_release(&thr->refcnt))
hwt_thread_free(thr);
} while (1);
}
void
hwt_ctx_free(struct hwt_context *ctx)
{
if (ctx->mode == HWT_MODE_CPU)
hwt_ctx_free_cpus(ctx);
else
hwt_ctx_free_threads(ctx);
hwt_config_free(ctx);
hwt_ctx_ident_free(ctx->ident);
free(ctx, M_HWT_CTX);
}
void
hwt_ctx_put(struct hwt_context *ctx)
{
refcount_release(&ctx->refcnt);
}
void
hwt_ctx_load(void)
{
ident_set_size = (1 << 8);
ident_set = bit_alloc(ident_set_size, M_HWT_CTX, M_WAITOK);
mtx_init(&ident_set_mutex, "ident set", NULL, MTX_DEF);
}
void
hwt_ctx_unload(void)
{
mtx_destroy(&ident_set_mutex);
free(ident_set, M_HWT_CTX);
}
sys/dev/hwt/hwt_context.h
+86
View File
@@ -0,0 +1,86 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_CONTEXT_H_
#define _DEV_HWT_HWT_CONTEXT_H_
enum hwt_ctx_state {
CTX_STATE_STOPPED,
CTX_STATE_RUNNING,
};
struct hwt_context {
TAILQ_HEAD(, hwt_record_entry) records;
LIST_ENTRY(hwt_context) next_hch; /* Entry in contexthash. */
LIST_ENTRY(hwt_context) next_hwts; /* Entry in ho->hwts. */
int mode;
int ident;
int kqueue_fd;
struct thread *hwt_td;
/* CPU mode. */
cpuset_t cpu_map;
TAILQ_HEAD(, hwt_cpu) cpus;
/* Thread mode. */
struct proc *proc; /* Target proc. */
pid_t pid; /* Target pid. */
TAILQ_HEAD(, hwt_thread) threads;
int thread_counter;
int pause_on_mmap;
size_t bufsize; /* Trace bufsize for each vm.*/
void *config;
size_t config_size;
int config_version;
struct hwt_owner *hwt_owner;
struct hwt_backend *hwt_backend;
struct mtx mtx;
struct mtx rec_mtx;
enum hwt_ctx_state state;
int refcnt;
};
#define HWT_CTX_LOCK(ctx) mtx_lock_spin(&(ctx)->mtx)
#define HWT_CTX_UNLOCK(ctx) mtx_unlock_spin(&(ctx)->mtx)
#define HWT_CTX_ASSERT_LOCKED(ctx) mtx_assert(&(ctx)->mtx, MA_OWNED)
int hwt_ctx_alloc(struct hwt_context **ctx0);
void hwt_ctx_free(struct hwt_context *ctx);
void hwt_ctx_put(struct hwt_context *ctx);
void hwt_ctx_load(void);
void hwt_ctx_unload(void);
#endif /* !_DEV_HWT_HWT_CONTEXT_H_ */
sys/dev/hwt/hwt_contexthash.c
+133
View File
@@ -0,0 +1,133 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mutex.h>
#include <sys/refcount.h>
#include <sys/hwt.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#define HWT_DEBUG
#undef HWT_DEBUG
#ifdef HWT_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
#define HWT_CONTEXTHASH_SIZE 1024
static MALLOC_DEFINE(M_HWT_CONTEXTHASH, "hwt_chash", "Hardware Trace");
/*
* Hash function. Discard the lower 2 bits of the pointer since
* these are always zero for our uses. The hash multiplier is
* round((2^LONG_BIT) * ((sqrt(5)-1)/2)).
*/
#define _HWT_HM 11400714819323198486u /* hash multiplier */
#define HWT_HASH_PTR(P, M) ((((unsigned long) (P) >> 2) * _HWT_HM) & (M))
static struct mtx hwt_contexthash_mtx;
static u_long hwt_contexthashmask;
static LIST_HEAD(hwt_contexthash, hwt_context) *hwt_contexthash;
/*
* To use by hwt_switch_in/out() and hwt_record() only.
* This function returns with refcnt acquired.
*/
struct hwt_context *
hwt_contexthash_lookup(struct proc *p)
{
struct hwt_contexthash *hch;
struct hwt_context *ctx;
int hindex;
hindex = HWT_HASH_PTR(p, hwt_contexthashmask);
hch = &hwt_contexthash[hindex];
HWT_CTXHASH_LOCK();
LIST_FOREACH(ctx, hch, next_hch) {
if (ctx->proc == p) {
refcount_acquire(&ctx->refcnt);
HWT_CTXHASH_UNLOCK();
return (ctx);
}
}
HWT_CTXHASH_UNLOCK();
return (NULL);
}
void
hwt_contexthash_insert(struct hwt_context *ctx)
{
struct hwt_contexthash *hch;
int hindex;
hindex = HWT_HASH_PTR(ctx->proc, hwt_contexthashmask);
hch = &hwt_contexthash[hindex];
HWT_CTXHASH_LOCK();
LIST_INSERT_HEAD(hch, ctx, next_hch);
HWT_CTXHASH_UNLOCK();
}
void
hwt_contexthash_remove(struct hwt_context *ctx)
{
HWT_CTXHASH_LOCK();
LIST_REMOVE(ctx, next_hch);
HWT_CTXHASH_UNLOCK();
}
void
hwt_contexthash_load(void)
{
hwt_contexthash = hashinit(HWT_CONTEXTHASH_SIZE, M_HWT_CONTEXTHASH,
&hwt_contexthashmask);
mtx_init(&hwt_contexthash_mtx, "hwt ctx hash", "hwt ctx", MTX_SPIN);
}
void
hwt_contexthash_unload(void)
{
mtx_destroy(&hwt_contexthash_mtx);
hashdestroy(hwt_contexthash, M_HWT_CONTEXTHASH, hwt_contexthashmask);
}
sys/dev/hwt/hwt_contexthash.h
+42
View File
@@ -0,0 +1,42 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_CONTEXTHASH_H_
#define _DEV_HWT_HWT_CONTEXTHASH_H_
struct hwt_context * hwt_contexthash_lookup(struct proc *p);
void hwt_contexthash_insert(struct hwt_context *ctx);
void hwt_contexthash_remove(struct hwt_context *ctx);
void hwt_contexthash_load(void);
void hwt_contexthash_unload(void);
#define HWT_CTXHASH_LOCK() mtx_lock_spin(&hwt_contexthash_mtx)
#define HWT_CTXHASH_UNLOCK() mtx_unlock_spin(&hwt_contexthash_mtx)
#endif /* !_DEV_HWT_HWT_CONTEXTHASH_H_ */
sys/dev/hwt/hwt_cpu.c
+115
View File
@@ -0,0 +1,115 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/proc.h>
#include <sys/kernel.h>
#include <sys/lock.h>
#include <sys/malloc.h>
#include <sys/mutex.h>
#include <sys/hwt.h>
#include <vm/vm.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_record.h>
#include <dev/hwt/hwt_cpu.h>
#define HWT_CPU_DEBUG
#undef HWT_CPU_DEBUG
#ifdef HWT_CPU_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static MALLOC_DEFINE(M_HWT_CPU, "hwt_cpu", "HWT cpu");
struct hwt_cpu *
hwt_cpu_alloc(void)
{
struct hwt_cpu *cpu;
cpu = malloc(sizeof(struct hwt_cpu), M_HWT_CPU, M_WAITOK | M_ZERO);
return (cpu);
}
void
hwt_cpu_free(struct hwt_cpu *cpu)
{
free(cpu, M_HWT_CPU);
}
struct hwt_cpu *
hwt_cpu_first(struct hwt_context *ctx)
{
struct hwt_cpu *cpu;
HWT_CTX_ASSERT_LOCKED(ctx);
cpu = TAILQ_FIRST(&ctx->cpus);
KASSERT(cpu != NULL, ("cpu is NULL"));
return (cpu);
}
struct hwt_cpu *
hwt_cpu_get(struct hwt_context *ctx, int cpu_id)
{
struct hwt_cpu *cpu, *tcpu;
HWT_CTX_ASSERT_LOCKED(ctx);
TAILQ_FOREACH_SAFE(cpu, &ctx->cpus, next, tcpu) {
KASSERT(cpu != NULL, ("cpu is NULL"));
if (cpu->cpu_id == cpu_id) {
return cpu;
}
}
return (NULL);
}
void
hwt_cpu_insert(struct hwt_context *ctx, struct hwt_cpu *cpu)
{
HWT_CTX_ASSERT_LOCKED(ctx);
TAILQ_INSERT_TAIL(&ctx->cpus, cpu, next);
}
sys/dev/hwt/hwt_cpu.h
+45
View File
@@ -0,0 +1,45 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_CPU_H_
#define _DEV_HWT_HWT_CPU_H_
struct hwt_cpu {
int cpu_id;
struct hwt_vm *vm;
TAILQ_ENTRY(hwt_cpu) next;
};
struct hwt_cpu * hwt_cpu_alloc(void);
void hwt_cpu_free(struct hwt_cpu *cpu);
struct hwt_cpu * hwt_cpu_first(struct hwt_context *ctx);
struct hwt_cpu * hwt_cpu_get(struct hwt_context *ctx, int cpu_id);
void hwt_cpu_insert(struct hwt_context *ctx, struct hwt_cpu *cpu);
#endif /* !_DEV_HWT_HWT_CPU_H_ */
sys/dev/hwt/hwt_hook.c
+323
View File
@@ -0,0 +1,323 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
/* Hardware Trace (HWT) framework. */
#include <sys/param.h>
#include <sys/proc.h>
#include <sys/malloc.h>
#include <sys/mman.h>
#include <sys/refcount.h>
#include <sys/hwt.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_backend.h>
#include <dev/hwt/hwt_record.h>
#include <dev/hwt/hwt_vm.h>
#define HWT_DEBUG
#undef HWT_DEBUG
#ifdef HWT_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static void
hwt_switch_in(struct thread *td)
{
struct hwt_context *ctx;
struct hwt_thread *thr;
struct proc *p;
int cpu_id;
p = td->td_proc;
cpu_id = PCPU_GET(cpuid);
ctx = hwt_contexthash_lookup(p);
if (ctx == NULL)
return;
if (ctx->state != CTX_STATE_RUNNING) {
hwt_ctx_put(ctx);
return;
}
thr = hwt_thread_lookup(ctx, td);
if (thr == NULL) {
hwt_ctx_put(ctx);
return;
}
dprintf("%s: thr %p index %d tid %d on cpu_id %d\n", __func__, thr,
thr->thread_id, td->td_tid, cpu_id);
hwt_backend_configure(ctx, cpu_id, thr->thread_id);
hwt_backend_enable(ctx, cpu_id);
hwt_ctx_put(ctx);
}
static void
hwt_switch_out(struct thread *td)
{
struct hwt_context *ctx;
struct hwt_thread *thr;
struct proc *p;
int cpu_id;
p = td->td_proc;
cpu_id = PCPU_GET(cpuid);
ctx = hwt_contexthash_lookup(p);
if (ctx == NULL)
return;
if (ctx->state != CTX_STATE_RUNNING) {
hwt_ctx_put(ctx);
return;
}
thr = hwt_thread_lookup(ctx, td);
if (thr == NULL) {
hwt_ctx_put(ctx);
return;
}
dprintf("%s: thr %p index %d tid %d on cpu_id %d\n", __func__, thr,
thr->thread_id, td->td_tid, cpu_id);
hwt_backend_disable(ctx, cpu_id);
hwt_ctx_put(ctx);
}
static void
hwt_hook_thread_exit(struct thread *td)
{
struct hwt_context *ctx;
struct hwt_thread *thr;
struct proc *p;
int cpu_id;
p = td->td_proc;
cpu_id = PCPU_GET(cpuid);
ctx = hwt_contexthash_lookup(p);
if (ctx == NULL)
return;
thr = hwt_thread_lookup(ctx, td);
if (thr == NULL) {
hwt_ctx_put(ctx);
return;
}
thr->state = HWT_THREAD_STATE_EXITED;
dprintf("%s: thr %p index %d tid %d on cpu_id %d\n", __func__, thr,
thr->thread_id, td->td_tid, cpu_id);
if (ctx->state == CTX_STATE_RUNNING)
hwt_backend_disable(ctx, cpu_id);
hwt_ctx_put(ctx);
}
static void
hwt_hook_mmap(struct thread *td)
{
struct hwt_context *ctx;
struct hwt_thread *thr;
struct proc *p;
int pause;
p = td->td_proc;
ctx = hwt_contexthash_lookup(p);
if (ctx == NULL)
return;
/* The ctx state could be any here. */
pause = ctx->pause_on_mmap ? 1 : 0;
thr = hwt_thread_lookup(ctx, td);
if (thr == NULL) {
hwt_ctx_put(ctx);
return;
}
/*
* msleep(9) atomically releases the mtx lock, so take refcount
* to ensure that thr is not destroyed.
* It could not be destroyed prior to this call as we are holding ctx
* refcnt.
*/
refcount_acquire(&thr->refcnt);
hwt_ctx_put(ctx);
if (pause) {
HWT_THR_LOCK(thr);
msleep(thr, &thr->mtx, PCATCH, "hwt-mmap", 0);
HWT_THR_UNLOCK(thr);
}
if (refcount_release(&thr->refcnt))
hwt_thread_free(thr);
}
static int
hwt_hook_thread_create(struct thread *td)
{
struct hwt_record_entry *entry;
struct hwt_context *ctx;
struct hwt_thread *thr;
char path[MAXPATHLEN];
size_t bufsize;
struct proc *p;
int thread_id, kva_req;
int error;
p = td->td_proc;
/* Step 1. Get CTX and collect information needed. */
ctx = hwt_contexthash_lookup(p);
if (ctx == NULL)
return (ENXIO);
thread_id = atomic_fetchadd_int(&ctx->thread_counter, 1);
bufsize = ctx->bufsize;
kva_req = ctx->hwt_backend->kva_req;
sprintf(path, "hwt_%d_%d", ctx->ident, thread_id);
hwt_ctx_put(ctx);
/* Step 2. Allocate some memory without holding ctx ref. */
error = hwt_thread_alloc(&thr, path, bufsize, kva_req);
if (error) {
printf("%s: could not allocate thread, error %d\n",
__func__, error);
return (error);
}
entry = hwt_record_entry_alloc();
entry->record_type = HWT_RECORD_THREAD_CREATE;
entry->thread_id = thread_id;
/* Step 3. Get CTX once again. */
ctx = hwt_contexthash_lookup(p);
if (ctx == NULL) {
hwt_record_entry_free(entry);
hwt_thread_free(thr);
/* ctx->thread_counter does not matter. */
return (ENXIO);
}
/* Allocate backend-specific thread data. */
error = hwt_backend_thread_alloc(ctx, thr);
if (error != 0) {
dprintf("%s: failed to allocate backend thread data\n",
__func__);
return (error);
}
thr->vm->ctx = ctx;
thr->ctx = ctx;
thr->backend = ctx->hwt_backend;
thr->thread_id = thread_id;
thr->td = td;
HWT_CTX_LOCK(ctx);
hwt_thread_insert(ctx, thr, entry);
HWT_CTX_UNLOCK(ctx);
/* Notify userspace. */
hwt_record_wakeup(ctx);
hwt_ctx_put(ctx);
return (0);
}
static void
hwt_hook_handler(struct thread *td, int func, void *arg)
{
struct proc *p;
p = td->td_proc;
if ((p->p_flag2 & P2_HWT) == 0)
return;
switch (func) {
case HWT_SWITCH_IN:
hwt_switch_in(td);
break;
case HWT_SWITCH_OUT:
hwt_switch_out(td);
break;
case HWT_THREAD_CREATE:
hwt_hook_thread_create(td);
break;
case HWT_THREAD_SET_NAME:
/* TODO. */
break;
case HWT_THREAD_EXIT:
hwt_hook_thread_exit(td);
break;
case HWT_EXEC:
case HWT_MMAP:
hwt_record_td(td, arg, M_WAITOK | M_ZERO);
hwt_hook_mmap(td);
break;
case HWT_RECORD:
hwt_record_td(td, arg, M_WAITOK | M_ZERO);
break;
};
}
void
hwt_hook_load(void)
{
hwt_hook = hwt_hook_handler;
}
void
hwt_hook_unload(void)
{
hwt_hook = NULL;
}
sys/dev/hwt/hwt_hook.h
+56
View File
@@ -0,0 +1,56 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/hwt_record.h>
#ifndef _DEV_HWT_HWT_HOOK_H_
#define _DEV_HWT_HWT_HOOK_H_
#define HWT_SWITCH_IN 0
#define HWT_SWITCH_OUT 1
#define HWT_THREAD_EXIT 2
#define HWT_THREAD_CREATE 3
#define HWT_THREAD_SET_NAME 4
#define HWT_RECORD 5
#define HWT_MMAP 6
#define HWT_EXEC 7
#define HWT_CALL_HOOK(td, func, arg) \
do { \
if (hwt_hook != NULL) \
(hwt_hook)((td), (func), (arg)); \
} while (0)
#define HWT_HOOK_INSTALLED (hwt_hook != NULL)
extern void (*hwt_hook)(struct thread *td, int func, void *arg);
void hwt_hook_load(void);
void hwt_hook_unload(void);
#endif /* !_DEV_HWT_HWT_HOOK_H_ */
sys/dev/hwt/hwt_intr.h
+33
View File
@@ -0,0 +1,33 @@
/*-
* Copyright (c) 2023-2025 Bojan Novković <bnovkov@freebsd.org>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_INTR_H_
#define _DEV_HWT_HWT_INTR_H_
#include <machine/frame.h>
extern int (*hwt_intr)(struct trapframe *tf);
#endif /* !_DEV_HWT_HWT_INTR_H_ */
sys/dev/hwt/hwt_ioctl.c
+443
View File
@@ -0,0 +1,443 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
/* Hardware Trace (HWT) framework. */
#include <sys/param.h>
#include <sys/proc.h>
#include <sys/ioccom.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mman.h>
#include <sys/mutex.h>
#include <sys/refcount.h>
#include <sys/rwlock.h>
#include <sys/smp.h>
#include <sys/hwt.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_cpu.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_ownerhash.h>
#include <dev/hwt/hwt_backend.h>
#include <dev/hwt/hwt_record.h>
#include <dev/hwt/hwt_ioctl.h>
#include <dev/hwt/hwt_vm.h>
#define HWT_IOCTL_DEBUG
#undef HWT_IOCTL_DEBUG
#ifdef HWT_IOCTL_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
/* No real reason for these limitations just sanity checks. */
#define HWT_MAXBUFSIZE (32UL * 1024 * 1024 * 1024) /* 32 GB */
static MALLOC_DEFINE(M_HWT_IOCTL, "hwt_ioctl", "Hardware Trace");
/*
* Check if owner process *o can trace target process *t.
*/
static int
hwt_priv_check(struct proc *o, struct proc *t)
{
struct ucred *oc, *tc;
int error;
int i;
PROC_LOCK(o);
oc = o->p_ucred;
crhold(oc);
PROC_UNLOCK(o);
PROC_LOCK_ASSERT(t, MA_OWNED);
tc = t->p_ucred;
crhold(tc);
error = 0;
/*
* The effective uid of the HWT owner should match at least one
* of the effective / real / saved uids of the target process.
*/
if (oc->cr_uid != tc->cr_uid &&
oc->cr_uid != tc->cr_svuid &&
oc->cr_uid != tc->cr_ruid) {
error = EPERM;
goto done;
}
/*
* Everyone of the target's group ids must be in the owner's
* group list.
*/
for (i = 0; i < tc->cr_ngroups; i++)
if (!groupmember(tc->cr_groups[i], oc)) {
error = EPERM;
goto done;
}
/* Check the read and saved GIDs too. */
if (!groupmember(tc->cr_rgid, oc) ||
!groupmember(tc->cr_svgid, oc)) {
error = EPERM;
goto done;
}
done:
crfree(tc);
crfree(oc);
return (error);
}
static int
hwt_ioctl_alloc_mode_thread(struct thread *td, struct hwt_owner *ho,
struct hwt_backend *backend, struct hwt_alloc *halloc)
{
struct thread **threads, *td1;
struct hwt_record_entry *entry;
struct hwt_context *ctx, *ctx1;
struct hwt_thread *thr;
char path[MAXPATHLEN];
struct proc *p;
int thread_id;
int error;
int cnt;
int i;
/* Check if the owner have this pid configured already. */
ctx = hwt_owner_lookup_ctx(ho, halloc->pid);
if (ctx)
return (EEXIST);
/* Allocate a new HWT context. */
error = hwt_ctx_alloc(&ctx);
if (error)
return (error);
ctx->bufsize = halloc->bufsize;
ctx->pid = halloc->pid;
ctx->hwt_backend = backend;
ctx->hwt_owner = ho;
ctx->mode = HWT_MODE_THREAD;
ctx->hwt_td = td;
ctx->kqueue_fd = halloc->kqueue_fd;
error = copyout(&ctx->ident, halloc->ident, sizeof(int));
if (error) {
hwt_ctx_free(ctx);
return (error);
}
/* Now get the victim proc. */
p = pfind(halloc->pid);
if (p == NULL) {
hwt_ctx_free(ctx);
return (ENXIO);
}
/* Ensure we can trace it. */
error = hwt_priv_check(td->td_proc, p);
if (error) {
PROC_UNLOCK(p);
hwt_ctx_free(ctx);
return (error);
}
/* Ensure it is not being traced already. */
ctx1 = hwt_contexthash_lookup(p);
if (ctx1) {
refcount_release(&ctx1->refcnt);
PROC_UNLOCK(p);
hwt_ctx_free(ctx);
return (EEXIST);
}
/* Allocate hwt threads and buffers. */
cnt = 0;
FOREACH_THREAD_IN_PROC(p, td1) {
cnt += 1;
}
KASSERT(cnt > 0, ("no threads"));
threads = malloc(sizeof(struct thread *) * cnt, M_HWT_IOCTL,
M_NOWAIT | M_ZERO);
if (threads == NULL) {
PROC_UNLOCK(p);
hwt_ctx_free(ctx);
return (ENOMEM);
}
i = 0;
FOREACH_THREAD_IN_PROC(p, td1) {
threads[i++] = td1;
}
ctx->proc = p;
PROC_UNLOCK(p);
for (i = 0; i < cnt; i++) {
thread_id = atomic_fetchadd_int(&ctx->thread_counter, 1);
sprintf(path, "hwt_%d_%d", ctx->ident, thread_id);
error = hwt_thread_alloc(&thr, path, ctx->bufsize,
ctx->hwt_backend->kva_req);
if (error) {
free(threads, M_HWT_IOCTL);
hwt_ctx_free(ctx);
return (error);
}
/* Allocate backend-specific thread data. */
error = hwt_backend_thread_alloc(ctx, thr);
if (error != 0) {
dprintf("%s: failed to allocate thread backend data\n",
__func__);
free(threads, M_HWT_IOCTL);
hwt_ctx_free(ctx);
return (error);
}
/*
* Insert a THREAD_CREATE record so userspace picks up
* the thread's tracing buffers.
*/
entry = hwt_record_entry_alloc();
entry->record_type = HWT_RECORD_THREAD_CREATE;
entry->thread_id = thread_id;
thr->vm->ctx = ctx;
thr->td = threads[i];
thr->ctx = ctx;
thr->backend = ctx->hwt_backend;
thr->thread_id = thread_id;
HWT_CTX_LOCK(ctx);
hwt_thread_insert(ctx, thr, entry);
HWT_CTX_UNLOCK(ctx);
}
free(threads, M_HWT_IOCTL);
error = hwt_backend_init(ctx);
if (error) {
hwt_ctx_free(ctx);
return (error);
}
/* hwt_owner_insert_ctx? */
mtx_lock(&ho->mtx);
LIST_INSERT_HEAD(&ho->hwts, ctx, next_hwts);
mtx_unlock(&ho->mtx);
/*
* Hooks are now in action after this, but the ctx is not in RUNNING
* state.
*/
hwt_contexthash_insert(ctx);
p = pfind(halloc->pid);
if (p) {
p->p_flag2 |= P2_HWT;
PROC_UNLOCK(p);
}
return (0);
}
static int
hwt_ioctl_alloc_mode_cpu(struct thread *td, struct hwt_owner *ho,
struct hwt_backend *backend, struct hwt_alloc *halloc)
{
struct hwt_context *ctx;
struct hwt_cpu *cpu;
struct hwt_vm *vm;
char path[MAXPATHLEN];
size_t cpusetsize;
cpuset_t cpu_map;
int cpu_count = 0;
int cpu_id;
int error;
CPU_ZERO(&cpu_map);
cpusetsize = min(halloc->cpusetsize, sizeof(cpuset_t));
error = copyin(halloc->cpu_map, &cpu_map, cpusetsize);
if (error)
return (error);
CPU_FOREACH_ISSET(cpu_id, &cpu_map) {
/* Ensure CPU is not halted. */
if (CPU_ISSET(cpu_id, &hlt_cpus_mask))
return (ENXIO);
#if 0
/* TODO: Check if the owner have this cpu configured already. */
ctx = hwt_owner_lookup_ctx_by_cpu(ho, halloc->cpu);
if (ctx)
return (EEXIST);
#endif
cpu_count++;
}
if (cpu_count == 0)
return (ENODEV);
/* Allocate a new HWT context. */
error = hwt_ctx_alloc(&ctx);
if (error)
return (error);
ctx->bufsize = halloc->bufsize;
ctx->hwt_backend = backend;
ctx->hwt_owner = ho;
ctx->mode = HWT_MODE_CPU;
ctx->cpu_map = cpu_map;
ctx->hwt_td = td;
ctx->kqueue_fd = halloc->kqueue_fd;
error = copyout(&ctx->ident, halloc->ident, sizeof(int));
if (error) {
hwt_ctx_free(ctx);
return (error);
}
CPU_FOREACH_ISSET(cpu_id, &cpu_map) {
sprintf(path, "hwt_%d_%d", ctx->ident, cpu_id);
error = hwt_vm_alloc(ctx->bufsize, ctx->hwt_backend->kva_req,
path, &vm);
if (error) {
/* TODO: remove all allocated cpus. */
hwt_ctx_free(ctx);
return (error);
}
cpu = hwt_cpu_alloc();
cpu->cpu_id = cpu_id;
cpu->vm = vm;
vm->cpu = cpu;
vm->ctx = ctx;
HWT_CTX_LOCK(ctx);
hwt_cpu_insert(ctx, cpu);
HWT_CTX_UNLOCK(ctx);
}
error = hwt_backend_init(ctx);
if (error) {
/* TODO: remove all allocated cpus. */
hwt_ctx_free(ctx);
return (error);
}
/* hwt_owner_insert_ctx? */
mtx_lock(&ho->mtx);
LIST_INSERT_HEAD(&ho->hwts, ctx, next_hwts);
mtx_unlock(&ho->mtx);
hwt_record_kernel_objects(ctx);
return (0);
}
static int
hwt_ioctl_alloc(struct thread *td, struct hwt_alloc *halloc)
{
char backend_name[HWT_BACKEND_MAXNAMELEN];
struct hwt_backend *backend;
struct hwt_owner *ho;
int error;
if (halloc->bufsize > HWT_MAXBUFSIZE)
return (EINVAL);
if (halloc->bufsize % PAGE_SIZE)
return (EINVAL);
if (halloc->backend_name == NULL)
return (EINVAL);
error = copyinstr(halloc->backend_name, (void *)backend_name,
HWT_BACKEND_MAXNAMELEN, NULL);
if (error)
return (error);
backend = hwt_backend_lookup(backend_name);
if (backend == NULL)
return (ENODEV);
/* First get the owner. */
ho = hwt_ownerhash_lookup(td->td_proc);
if (ho == NULL) {
/* Create a new owner. */
ho = hwt_owner_alloc(td->td_proc);
if (ho == NULL)
return (ENOMEM);
hwt_ownerhash_insert(ho);
}
switch (halloc->mode) {
case HWT_MODE_THREAD:
error = hwt_ioctl_alloc_mode_thread(td, ho, backend, halloc);
break;
case HWT_MODE_CPU:
error = hwt_ioctl_alloc_mode_cpu(td, ho, backend, halloc);
break;
default:
error = ENXIO;
};
return (error);
}
int
hwt_ioctl(struct cdev *dev, u_long cmd, caddr_t addr, int flags,
struct thread *td)
{
int error;
switch (cmd) {
case HWT_IOC_ALLOC:
/* Allocate HWT context. */
error = hwt_ioctl_alloc(td, (struct hwt_alloc *)addr);
return (error);
default:
return (ENXIO);
};
}
sys/dev/hwt/hwt_ioctl.h
+35
View File
@@ -0,0 +1,35 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_IOCTL_H
#define _DEV_HWT_HWT_IOCTL_H
int hwt_ioctl(struct cdev *dev, u_long cmd, caddr_t addr, int flags,
struct thread *td);
#endif /* !_DEV_HWT_HWT_IOCTL_H */
sys/dev/hwt/hwt_owner.c
+157
View File
@@ -0,0 +1,157 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mman.h>
#include <sys/mutex.h>
#include <sys/refcount.h>
#include <sys/rwlock.h>
#include <sys/hwt.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_cpu.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_ownerhash.h>
#include <dev/hwt/hwt_backend.h>
#include <dev/hwt/hwt_vm.h>
#include <dev/hwt/hwt_record.h>
#define HWT_DEBUG
#undef HWT_DEBUG
#ifdef HWT_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static MALLOC_DEFINE(M_HWT_OWNER, "hwt_owner", "Hardware Trace");
struct hwt_context *
hwt_owner_lookup_ctx(struct hwt_owner *ho, pid_t pid)
{
struct hwt_context *ctx;
mtx_lock(&ho->mtx);
LIST_FOREACH(ctx, &ho->hwts, next_hwts) {
if (ctx->pid == pid) {
mtx_unlock(&ho->mtx);
return (ctx);
}
}
mtx_unlock(&ho->mtx);
return (NULL);
}
#if 0
struct hwt_context *
hwt_owner_lookup_ctx_by_cpu(struct hwt_owner *ho, int cpu)
{
struct hwt_context *ctx;
mtx_lock(&ho->mtx);
LIST_FOREACH(ctx, &ho->hwts, next_hwts) {
if (ctx->cpu == cpu) {
mtx_unlock(&ho->mtx);
return (ctx);
}
}
mtx_unlock(&ho->mtx);
return (NULL);
}
#endif
struct hwt_owner *
hwt_owner_alloc(struct proc *p)
{
struct hwt_owner *ho;
ho = malloc(sizeof(struct hwt_owner), M_HWT_OWNER,
M_WAITOK | M_ZERO);
ho->p = p;
LIST_INIT(&ho->hwts);
mtx_init(&ho->mtx, "hwts", NULL, MTX_DEF);
return (ho);
}
void
hwt_owner_shutdown(struct hwt_owner *ho)
{
struct hwt_context *ctx;
dprintf("%s: stopping hwt owner\n", __func__);
while (1) {
mtx_lock(&ho->mtx);
ctx = LIST_FIRST(&ho->hwts);
if (ctx)
LIST_REMOVE(ctx, next_hwts);
mtx_unlock(&ho->mtx);
if (ctx == NULL)
break;
if (ctx->mode == HWT_MODE_THREAD)
hwt_contexthash_remove(ctx);
/*
* A hook could be still dealing with this ctx right here.
*/
HWT_CTX_LOCK(ctx);
ctx->state = 0;
HWT_CTX_UNLOCK(ctx);
/* Ensure hooks invocation is now completed. */
while (refcount_load(&ctx->refcnt) > 0)
continue;
/*
* Note that a thread could be still sleeping on msleep(9).
*/
hwt_backend_deinit(ctx);
hwt_record_free_all(ctx);
hwt_ctx_free(ctx);
}
hwt_ownerhash_remove(ho);
free(ho, M_HWT_OWNER);
}
sys/dev/hwt/hwt_owner.h
+45
View File
@@ -0,0 +1,45 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_OWNER_H_
#define _DEV_HWT_HWT_OWNER_H_
struct hwt_owner {
struct proc *p;
struct mtx mtx; /* Protects hwts. */
LIST_HEAD(, hwt_context) hwts; /* Owned HWTs. */
LIST_ENTRY(hwt_owner) next; /* Entry in hwt owner hash. */
};
struct hwt_context * hwt_owner_lookup_ctx(struct hwt_owner *ho, pid_t pid);
struct hwt_owner * hwt_owner_alloc(struct proc *p);
void hwt_owner_shutdown(struct hwt_owner *ho);
struct hwt_context * hwt_owner_lookup_ctx_by_cpu(struct hwt_owner *ho, int cpu);
#endif /* !_DEV_HWT_HWT_OWNER_H_ */
sys/dev/hwt/hwt_ownerhash.c
+141
View File
@@ -0,0 +1,141 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mman.h>
#include <sys/mutex.h>
#include <sys/rwlock.h>
#include <sys/hwt.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_ownerhash.h>
#define HWT_DEBUG
#undef HWT_DEBUG
#ifdef HWT_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
#define HWT_OWNERHASH_SIZE 1024
static MALLOC_DEFINE(M_HWT_OWNERHASH, "hwt_ohash", "Hardware Trace");
/*
* Hash function. Discard the lower 2 bits of the pointer since
* these are always zero for our uses. The hash multiplier is
* round((2^LONG_BIT) * ((sqrt(5)-1)/2)).
*/
#define _HWT_HM 11400714819323198486u /* hash multiplier */
#define HWT_HASH_PTR(P, M) ((((unsigned long) (P) >> 2) * _HWT_HM) & (M))
static struct mtx hwt_ownerhash_mtx;
static u_long hwt_ownerhashmask;
static LIST_HEAD(hwt_ownerhash, hwt_owner) *hwt_ownerhash;
struct hwt_owner *
hwt_ownerhash_lookup(struct proc *p)
{
struct hwt_ownerhash *hoh;
struct hwt_owner *ho;
int hindex;
hindex = HWT_HASH_PTR(p, hwt_ownerhashmask);
hoh = &hwt_ownerhash[hindex];
HWT_OWNERHASH_LOCK();
LIST_FOREACH(ho, hoh, next) {
if (ho->p == p) {
HWT_OWNERHASH_UNLOCK();
return (ho);
}
}
HWT_OWNERHASH_UNLOCK();
return (NULL);
}
void
hwt_ownerhash_insert(struct hwt_owner *ho)
{
struct hwt_ownerhash *hoh;
int hindex;
hindex = HWT_HASH_PTR(ho->p, hwt_ownerhashmask);
hoh = &hwt_ownerhash[hindex];
HWT_OWNERHASH_LOCK();
LIST_INSERT_HEAD(hoh, ho, next);
HWT_OWNERHASH_UNLOCK();
}
void
hwt_ownerhash_remove(struct hwt_owner *ho)
{
/* Destroy hwt owner. */
HWT_OWNERHASH_LOCK();
LIST_REMOVE(ho, next);
HWT_OWNERHASH_UNLOCK();
}
void
hwt_ownerhash_load(void)
{
hwt_ownerhash = hashinit(HWT_OWNERHASH_SIZE, M_HWT_OWNERHASH,
&hwt_ownerhashmask);
mtx_init(&hwt_ownerhash_mtx, "hwt-owner-hash", "hwt-owner", MTX_DEF);
}
void
hwt_ownerhash_unload(void)
{
struct hwt_ownerhash *hoh;
struct hwt_owner *ho, *tmp;
HWT_OWNERHASH_LOCK();
for (hoh = hwt_ownerhash;
hoh <= &hwt_ownerhash[hwt_ownerhashmask];
hoh++) {
LIST_FOREACH_SAFE(ho, hoh, next, tmp) {
/* TODO: module is in use ? */
}
}
HWT_OWNERHASH_UNLOCK();
mtx_destroy(&hwt_ownerhash_mtx);
hashdestroy(hwt_ownerhash, M_HWT_OWNERHASH, hwt_ownerhashmask);
}
sys/dev/hwt/hwt_ownerhash.h
+42
View File
@@ -0,0 +1,42 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_OWNERHASH_H_
#define _DEV_HWT_HWT_OWNERHASH_H_
struct hwt_owner * hwt_ownerhash_lookup(struct proc *p);
void hwt_ownerhash_insert(struct hwt_owner *ho);
void hwt_ownerhash_remove(struct hwt_owner *ho);
void hwt_ownerhash_load(void);
void hwt_ownerhash_unload(void);
#define HWT_OWNERHASH_LOCK() mtx_lock(&hwt_ownerhash_mtx)
#define HWT_OWNERHASH_UNLOCK() mtx_unlock(&hwt_ownerhash_mtx)
#endif /* !_DEV_HWT_HWT_OWNERHASH_H_ */
sys/dev/hwt/hwt_record.c
+302
View File
@@ -0,0 +1,302 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/proc.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mutex.h>
#include <sys/hwt.h>
#include <sys/linker.h>
#include <sys/pmckern.h> /* linker_hwpmc_list_objects */
#include <vm/vm.h>
#include <vm/uma.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_record.h>
#define HWT_RECORD_DEBUG
#undef HWT_RECORD_DEBUG
#ifdef HWT_RECORD_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static MALLOC_DEFINE(M_HWT_RECORD, "hwt_record", "Hardware Trace");
static uma_zone_t record_zone = NULL;
static struct hwt_record_entry *
hwt_record_clone(struct hwt_record_entry *ent, int flags)
{
struct hwt_record_entry *entry;
entry = uma_zalloc(record_zone, flags);
if (entry == NULL)
return (NULL);
memcpy(entry, ent, sizeof(struct hwt_record_entry));
switch (ent->record_type) {
case HWT_RECORD_MMAP:
case HWT_RECORD_EXECUTABLE:
case HWT_RECORD_KERNEL:
entry->fullpath = strdup(ent->fullpath, M_HWT_RECORD);
break;
default:
break;
}
return (entry);
}
static void
hwt_record_to_user(struct hwt_record_entry *ent,
struct hwt_record_user_entry *usr)
{
usr->record_type = ent->record_type;
switch (ent->record_type) {
case HWT_RECORD_MMAP:
case HWT_RECORD_EXECUTABLE:
case HWT_RECORD_KERNEL:
usr->addr = ent->addr;
usr->baseaddr = ent->baseaddr;
strncpy(usr->fullpath, ent->fullpath, MAXPATHLEN);
break;
case HWT_RECORD_BUFFER:
usr->buf_id = ent->buf_id;
usr->curpage = ent->curpage;
usr->offset = ent->offset;
break;
case HWT_RECORD_THREAD_CREATE:
case HWT_RECORD_THREAD_SET_NAME:
usr->thread_id = ent->thread_id;
break;
default:
break;
}
}
void
hwt_record_load(void)
{
record_zone = uma_zcreate("HWT records",
sizeof(struct hwt_record_entry), NULL, NULL, NULL, NULL, 0, 0);
}
void
hwt_record_unload(void)
{
uma_zdestroy(record_zone);
}
void
hwt_record_ctx(struct hwt_context *ctx, struct hwt_record_entry *ent, int flags)
{
struct hwt_record_entry *entry;
KASSERT(ent != NULL, ("ent is NULL"));
entry = hwt_record_clone(ent, flags);
if (entry == NULL) {
/* XXX: Not sure what to do here other than logging an error. */
return;
}
HWT_CTX_LOCK(ctx);
TAILQ_INSERT_TAIL(&ctx->records, entry, next);
HWT_CTX_UNLOCK(ctx);
hwt_record_wakeup(ctx);
}
void
hwt_record_td(struct thread *td, struct hwt_record_entry *ent, int flags)
{
struct hwt_record_entry *entry;
struct hwt_context *ctx;
struct proc *p;
p = td->td_proc;
KASSERT(ent != NULL, ("ent is NULL"));
entry = hwt_record_clone(ent, flags);
if (entry == NULL) {
/* XXX: Not sure what to do here other than logging an error. */
return;
}
ctx = hwt_contexthash_lookup(p);
if (ctx == NULL) {
hwt_record_entry_free(entry);
return;
}
HWT_CTX_LOCK(ctx);
TAILQ_INSERT_TAIL(&ctx->records, entry, next);
HWT_CTX_UNLOCK(ctx);
hwt_record_wakeup(ctx);
hwt_ctx_put(ctx);
}
struct hwt_record_entry *
hwt_record_entry_alloc(void)
{
return (uma_zalloc(record_zone, M_WAITOK | M_ZERO));
}
void
hwt_record_entry_free(struct hwt_record_entry *entry)
{
switch (entry->record_type) {
case HWT_RECORD_MMAP:
case HWT_RECORD_EXECUTABLE:
case HWT_RECORD_KERNEL:
free(entry->fullpath, M_HWT_RECORD);
break;
default:
break;
}
uma_zfree(record_zone, entry);
}
static int
hwt_record_grab(struct hwt_context *ctx,
struct hwt_record_user_entry *user_entry, int nitems_req, int wait)
{
struct hwt_record_entry *entry;
int i;
if (wait) {
mtx_lock(&ctx->rec_mtx);
if (TAILQ_FIRST(&ctx->records) == NULL) {
/* Wait until we have new records. */
msleep(ctx, &ctx->rec_mtx, PCATCH, "recsnd", 0);
}
mtx_unlock(&ctx->rec_mtx);
}
for (i = 0; i < nitems_req; i++) {
HWT_CTX_LOCK(ctx);
entry = TAILQ_FIRST(&ctx->records);
if (entry)
TAILQ_REMOVE_HEAD(&ctx->records, next);
HWT_CTX_UNLOCK(ctx);
if (entry == NULL)
break;
hwt_record_to_user(entry, &user_entry[i]);
hwt_record_entry_free(entry);
}
return (i);
}
void
hwt_record_free_all(struct hwt_context *ctx)
{
struct hwt_record_entry *entry;
while (1) {
HWT_CTX_LOCK(ctx);
entry = TAILQ_FIRST(&ctx->records);
if (entry)
TAILQ_REMOVE_HEAD(&ctx->records, next);
HWT_CTX_UNLOCK(ctx);
if (entry == NULL)
break;
hwt_record_entry_free(entry);
}
}
int
hwt_record_send(struct hwt_context *ctx, struct hwt_record_get *record_get)
{
struct hwt_record_user_entry *user_entry;
int nitems_req;
int error;
int i;
nitems_req = 0;
error = copyin(record_get->nentries, &nitems_req, sizeof(int));
if (error)
return (error);
if (nitems_req < 1 || nitems_req > 1024)
return (ENXIO);
user_entry = malloc(sizeof(struct hwt_record_user_entry) * nitems_req,
M_HWT_RECORD, M_WAITOK | M_ZERO);
i = hwt_record_grab(ctx, user_entry, nitems_req, record_get->wait);
if (i > 0)
error = copyout(user_entry, record_get->records,
sizeof(struct hwt_record_user_entry) * i);
if (error == 0)
error = copyout(&i, record_get->nentries, sizeof(int));
free(user_entry, M_HWT_RECORD);
return (error);
}
void
hwt_record_kernel_objects(struct hwt_context *ctx)
{
struct hwt_record_entry *entry;
struct pmckern_map_in *kobase;
int i;
kobase = linker_hwpmc_list_objects();
for (i = 0; kobase[i].pm_file != NULL; i++) {
entry = hwt_record_entry_alloc();
entry->record_type = HWT_RECORD_KERNEL;
entry->fullpath = strdup(kobase[i].pm_file, M_HWT_RECORD);
entry->addr = kobase[i].pm_address;
HWT_CTX_LOCK(ctx);
TAILQ_INSERT_HEAD(&ctx->records, entry, next);
HWT_CTX_UNLOCK(ctx);
}
free(kobase, M_LINKER);
}
void
hwt_record_wakeup(struct hwt_context *ctx)
{
wakeup(ctx);
}
sys/dev/hwt/hwt_record.h
+47
View File
@@ -0,0 +1,47 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_RECORD_H_
#define _DEV_HWT_HWT_RECORD_H_
struct hwt_record_get;
void hwt_record_load(void);
void hwt_record_unload(void);
int hwt_record_send(struct hwt_context *ctx, struct hwt_record_get *record_get);
void hwt_record_td(struct thread *td, struct hwt_record_entry *ent, int flags);
void hwt_record_ctx(struct hwt_context *ctx, struct hwt_record_entry *ent,
int flags);
struct hwt_record_entry * hwt_record_entry_alloc(void);
void hwt_record_entry_free(struct hwt_record_entry *entry);
void hwt_record_kernel_objects(struct hwt_context *ctx);
void hwt_record_free_all(struct hwt_context *ctx);
void hwt_record_wakeup(struct hwt_context *ctx);
#endif /* !_DEV_HWT_HWT_RECORD_H_ */
sys/dev/hwt/hwt_thread.c
+162
View File
@@ -0,0 +1,162 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/lock.h>
#include <sys/mutex.h>
#include <sys/refcount.h>
#include <sys/rwlock.h>
#include <sys/hwt.h>
#include <vm/vm.h>
#include <vm/vm_param.h>
#include <vm/vm_kern.h>
#include <vm/vm_page.h>
#include <vm/vm_object.h>
#include <vm/vm_pager.h>
#include <vm/vm_pageout.h>
#include <vm/vm_phys.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_ownerhash.h>
#include <dev/hwt/hwt_backend.h>
#include <dev/hwt/hwt_vm.h>
#include <dev/hwt/hwt_record.h>
#define HWT_THREAD_DEBUG
#undef HWT_THREAD_DEBUG
#ifdef HWT_THREAD_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static MALLOC_DEFINE(M_HWT_THREAD, "hwt_thread", "Hardware Trace");
struct hwt_thread *
hwt_thread_first(struct hwt_context *ctx)
{
struct hwt_thread *thr;
HWT_CTX_ASSERT_LOCKED(ctx);
thr = TAILQ_FIRST(&ctx->threads);
KASSERT(thr != NULL, ("thr is NULL"));
return (thr);
}
/*
* To use by hwt_switch_in/out() only.
*/
struct hwt_thread *
hwt_thread_lookup(struct hwt_context *ctx, struct thread *td)
{
struct hwt_thread *thr;
/* Caller of this func holds ctx refcnt right here. */
HWT_CTX_LOCK(ctx);
TAILQ_FOREACH(thr, &ctx->threads, next) {
if (thr->td == td) {
HWT_CTX_UNLOCK(ctx);
return (thr);
}
}
HWT_CTX_UNLOCK(ctx);
/*
* We are here because the hook on thread creation failed to allocate
* a thread.
*/
return (NULL);
}
int
hwt_thread_alloc(struct hwt_thread **thr0, char *path, size_t bufsize,
int kva_req)
{
struct hwt_thread *thr;
struct hwt_vm *vm;
int error;
error = hwt_vm_alloc(bufsize, kva_req, path, &vm);
if (error)
return (error);
thr = malloc(sizeof(struct hwt_thread), M_HWT_THREAD,
M_WAITOK | M_ZERO);
thr->vm = vm;
mtx_init(&thr->mtx, "thr", NULL, MTX_DEF);
refcount_init(&thr->refcnt, 1);
vm->thr = thr;
*thr0 = thr;
return (0);
}
void
hwt_thread_free(struct hwt_thread *thr)
{
hwt_vm_free(thr->vm);
/* Free private backend data, if any. */
if (thr->private != NULL)
hwt_backend_thread_free(thr);
free(thr, M_HWT_THREAD);
}
/*
* Inserts a new thread and a thread creation record into the
* context notifies userspace about the newly created thread.
*/
void
hwt_thread_insert(struct hwt_context *ctx, struct hwt_thread *thr,
struct hwt_record_entry *entry)
{
HWT_CTX_ASSERT_LOCKED(ctx);
TAILQ_INSERT_TAIL(&ctx->threads, thr, next);
TAILQ_INSERT_TAIL(&ctx->records, entry, next);
}
sys/dev/hwt/hwt_thread.h
+64
View File
@@ -0,0 +1,64 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_THREAD_H_
#define _DEV_HWT_HWT_THREAD_H_
struct hwt_record_entry;
struct hwt_thread {
struct hwt_vm *vm;
struct hwt_context *ctx;
struct hwt_backend *backend;
struct thread *td;
TAILQ_ENTRY(hwt_thread) next;
int thread_id;
int state;
#define HWT_THREAD_STATE_EXITED (1 << 0)
struct mtx mtx;
u_int refcnt;
int cpu_id; /* last cpu_id */
void *private; /* backend-specific private data */
};
/* Thread allocation. */
int hwt_thread_alloc(struct hwt_thread **thr0, char *path, size_t bufsize,
int kva_req);
void hwt_thread_free(struct hwt_thread *thr);
/* Thread list mgt. */
void hwt_thread_insert(struct hwt_context *ctx, struct hwt_thread *thr, struct hwt_record_entry *entry);
struct hwt_thread * hwt_thread_first(struct hwt_context *ctx);
struct hwt_thread * hwt_thread_lookup(struct hwt_context *ctx,
struct thread *td);
#define HWT_THR_LOCK(thr) mtx_lock(&(thr)->mtx)
#define HWT_THR_UNLOCK(thr) mtx_unlock(&(thr)->mtx)
#define HWT_THR_ASSERT_LOCKED(thr) mtx_assert(&(thr)->mtx, MA_OWNED)
#endif /* !_DEV_HWT_HWT_THREAD_H_ */
sys/dev/hwt/hwt_vm.c
+501
View File
@@ -0,0 +1,501 @@
/*-
* SPDX-License-Identifier: BSD-2-Clause
*
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <sys/param.h>
#include <sys/ioccom.h>
#include <sys/conf.h>
#include <sys/proc.h>
#include <sys/kernel.h>
#include <sys/malloc.h>
#include <sys/mman.h>
#include <sys/refcount.h>
#include <sys/rwlock.h>
#include <sys/hwt.h>
#include <sys/smp.h>
#include <vm/vm.h>
#include <vm/pmap.h>
#include <vm/vm_extern.h>
#include <vm/vm_param.h>
#include <vm/vm_kern.h>
#include <vm/vm_page.h>
#include <vm/vm_object.h>
#include <vm/vm_pager.h>
#include <vm/vm_pageout.h>
#include <vm/vm_phys.h>
#include <dev/hwt/hwt_hook.h>
#include <dev/hwt/hwt_context.h>
#include <dev/hwt/hwt_contexthash.h>
#include <dev/hwt/hwt_config.h>
#include <dev/hwt/hwt_cpu.h>
#include <dev/hwt/hwt_owner.h>
#include <dev/hwt/hwt_ownerhash.h>
#include <dev/hwt/hwt_thread.h>
#include <dev/hwt/hwt_backend.h>
#include <dev/hwt/hwt_vm.h>
#include <dev/hwt/hwt_record.h>
#define HWT_THREAD_DEBUG
#undef HWT_THREAD_DEBUG
#ifdef HWT_THREAD_DEBUG
#define dprintf(fmt, ...) printf(fmt, ##__VA_ARGS__)
#else
#define dprintf(fmt, ...)
#endif
static MALLOC_DEFINE(M_HWT_VM, "hwt_vm", "Hardware Trace");
static int
hwt_vm_fault(vm_object_t vm_obj, vm_ooffset_t offset,
int prot, vm_page_t *mres)
{
return (0);
}
static int
hwt_vm_ctor(void *handle, vm_ooffset_t size, vm_prot_t prot,
vm_ooffset_t foff, struct ucred *cred, u_short *color)
{
*color = 0;
return (0);
}
static void
hwt_vm_dtor(void *handle)
{
}
static struct cdev_pager_ops hwt_vm_pager_ops = {
.cdev_pg_fault = hwt_vm_fault,
.cdev_pg_ctor = hwt_vm_ctor,
.cdev_pg_dtor = hwt_vm_dtor
};
static int
hwt_vm_alloc_pages(struct hwt_vm *vm, int kva_req)
{
vm_paddr_t low, high, boundary;
vm_memattr_t memattr;
#ifdef __aarch64__
uintptr_t va;
#endif
int alignment;
vm_page_t m;
int pflags;
int tries;
int i;
alignment = PAGE_SIZE;
low = 0;
high = -1UL;
boundary = 0;
pflags = VM_ALLOC_NORMAL | VM_ALLOC_WIRED | VM_ALLOC_ZERO;
memattr = VM_MEMATTR_DEVICE;
if (kva_req) {
vm->kvaddr = kva_alloc(vm->npages * PAGE_SIZE);
if (!vm->kvaddr)
return (ENOMEM);
}
vm->obj = cdev_pager_allocate(vm, OBJT_MGTDEVICE,
&hwt_vm_pager_ops, vm->npages * PAGE_SIZE, PROT_READ, 0,
curthread->td_ucred);
for (i = 0; i < vm->npages; i++) {
tries = 0;
retry:
m = vm_page_alloc_noobj_contig(pflags, 1, low, high,
alignment, boundary, memattr);
if (m == NULL) {
if (tries < 3) {
if (!vm_page_reclaim_contig(pflags, 1, low,
high, alignment, boundary))
vm_wait(NULL);
tries++;
goto retry;
}
return (ENOMEM);
}
#if 0
/* TODO: could not clean device memory on arm64. */
if ((m->flags & PG_ZERO) == 0)
pmap_zero_page(m);
#endif
#ifdef __aarch64__
va = PHYS_TO_DMAP(VM_PAGE_TO_PHYS(m));
cpu_dcache_wb_range((void *)va, PAGE_SIZE);
#endif
m->valid = VM_PAGE_BITS_ALL;
m->oflags &= ~VPO_UNMANAGED;
m->flags |= PG_FICTITIOUS;
vm->pages[i] = m;
VM_OBJECT_WLOCK(vm->obj);
vm_page_insert(m, vm->obj, i);
if (kva_req)
pmap_qenter(vm->kvaddr + i * PAGE_SIZE, &m, 1);
VM_OBJECT_WUNLOCK(vm->obj);
}
return (0);
}
static int
hwt_vm_open(struct cdev *cdev, int oflags, int devtype, struct thread *td)
{
dprintf("%s\n", __func__);
return (0);
}
static int
hwt_vm_mmap_single(struct cdev *cdev, vm_ooffset_t *offset,
vm_size_t mapsize, struct vm_object **objp, int nprot)
{
struct hwt_vm *vm;
vm = cdev->si_drv1;
if (nprot != PROT_READ || *offset != 0)
return (ENXIO);
vm_object_reference(vm->obj);
*objp = vm->obj;
return (0);
}
static void
hwt_vm_start_cpu_mode(struct hwt_context *ctx)
{
cpuset_t enable_cpus;
int cpu_id;
CPU_ZERO(&enable_cpus);
CPU_FOREACH_ISSET(cpu_id, &ctx->cpu_map) {
/* Ensure CPU is not halted. */
if (CPU_ISSET(cpu_id, &hlt_cpus_mask))
continue;
hwt_backend_configure(ctx, cpu_id, cpu_id);
CPU_SET(cpu_id, &enable_cpus);
}
if (ctx->hwt_backend->ops->hwt_backend_enable_smp == NULL) {
CPU_FOREACH_ISSET(cpu_id, &enable_cpus)
hwt_backend_enable(ctx, cpu_id);
} else {
/* Some backends require enabling all CPUs at once. */
hwt_backend_enable_smp(ctx);
}
}
static int
hwt_vm_ioctl(struct cdev *dev, u_long cmd, caddr_t addr, int flags,
struct thread *td)
{
struct hwt_record_get *rget;
struct hwt_set_config *sconf;
struct hwt_bufptr_get *ptr_get;
struct hwt_svc_buf *sbuf;
struct hwt_context *ctx;
struct hwt_vm *vm;
struct hwt_owner *ho;
vm_offset_t offset;
int ident;
int error;
uint64_t data = 0;
void *data2;
size_t data_size;
int data_version;
vm = dev->si_drv1;
KASSERT(vm != NULL, ("si_drv1 is NULL"));
ctx = vm->ctx;
/* Ensure process is registered owner of this HWT. */
ho = hwt_ownerhash_lookup(td->td_proc);
if (ho == NULL)
return (ENXIO);
if (ctx->hwt_owner != ho)
return (EPERM);
switch (cmd) {
case HWT_IOC_START:
dprintf("%s: start tracing\n", __func__);
HWT_CTX_LOCK(ctx);
if (ctx->state == CTX_STATE_RUNNING) {
/* Already running ? */
HWT_CTX_UNLOCK(ctx);
return (ENXIO);
}
ctx->state = CTX_STATE_RUNNING;
HWT_CTX_UNLOCK(ctx);
if (ctx->mode == HWT_MODE_CPU)
hwt_vm_start_cpu_mode(ctx);
else {
/*
* Tracing backend will be configured and enabled
* during hook invocation. See hwt_hook.c.
*/
}
break;
case HWT_IOC_STOP:
if (ctx->state == CTX_STATE_STOPPED)
return (ENXIO);
hwt_backend_stop(ctx);
ctx->state = CTX_STATE_STOPPED;
break;
case HWT_IOC_RECORD_GET:
rget = (struct hwt_record_get *)addr;
error = hwt_record_send(ctx, rget);
if (error)
return (error);
break;
case HWT_IOC_SET_CONFIG:
if (ctx->state == CTX_STATE_RUNNING) {
return (ENXIO);
}
sconf = (struct hwt_set_config *)addr;
error = hwt_config_set(td, ctx, sconf);
if (error)
return (error);
ctx->pause_on_mmap = sconf->pause_on_mmap ? 1 : 0;
break;
case HWT_IOC_WAKEUP:
if (ctx->mode == HWT_MODE_CPU)
return (ENXIO);
KASSERT(vm->thr != NULL, ("thr is NULL"));
wakeup(vm->thr);
break;
case HWT_IOC_BUFPTR_GET:
ptr_get = (struct hwt_bufptr_get *)addr;
error = hwt_backend_read(ctx, vm, &ident, &offset, &data);
if (error)
return (error);
if (ptr_get->ident)
error = copyout(&ident, ptr_get->ident, sizeof(int));
if (error)
return (error);
if (ptr_get->offset)
error = copyout(&offset, ptr_get->offset,
sizeof(vm_offset_t));
if (error)
return (error);
if (ptr_get->data)
error = copyout(&data, ptr_get->data, sizeof(uint64_t));
if (error)
return (error);
break;
case HWT_IOC_SVC_BUF:
if (ctx->state == CTX_STATE_STOPPED) {
return (ENXIO);
}
sbuf = (struct hwt_svc_buf *)addr;
data_size = sbuf->data_size;
data_version = sbuf->data_version;
if (data_size == 0 || data_size > PAGE_SIZE)
return (EINVAL);
data2 = malloc(data_size, M_HWT_VM, M_WAITOK | M_ZERO);
error = copyin(sbuf->data, data2, data_size);
if (error) {
free(data2, M_HWT_VM);
return (error);
}
error = hwt_backend_svc_buf(ctx, data2, data_size, data_version);
if (error) {
free(data2, M_HWT_VM);
return (error);
}
free(data2, M_HWT_VM);
break;
default:
break;
}
return (0);
}
static struct cdevsw hwt_vm_cdevsw = {
.d_version = D_VERSION,
.d_name = "hwt",
.d_open = hwt_vm_open,
.d_mmap_single = hwt_vm_mmap_single,
.d_ioctl = hwt_vm_ioctl,
};
static int
hwt_vm_create_cdev(struct hwt_vm *vm, char *path)
{
struct make_dev_args args;
int error;
dprintf("%s: path %s\n", __func__, path);
make_dev_args_init(&args);
args.mda_devsw = &hwt_vm_cdevsw;
args.mda_flags = MAKEDEV_CHECKNAME | MAKEDEV_WAITOK;
args.mda_uid = UID_ROOT;
args.mda_gid = GID_WHEEL;
args.mda_mode = 0660;
args.mda_si_drv1 = vm;
error = make_dev_s(&args, &vm->cdev, "%s", path);
if (error != 0)
return (error);
return (0);
}
static int
hwt_vm_alloc_buffers(struct hwt_vm *vm, int kva_req)
{
int error;
vm->pages = malloc(sizeof(struct vm_page *) * vm->npages,
M_HWT_VM, M_WAITOK | M_ZERO);
error = hwt_vm_alloc_pages(vm, kva_req);
if (error) {
printf("%s: could not alloc pages\n", __func__);
return (error);
}
return (0);
}
static void
hwt_vm_destroy_buffers(struct hwt_vm *vm)
{
vm_page_t m;
int i;
if (vm->ctx->hwt_backend->kva_req && vm->kvaddr != 0) {
pmap_qremove(vm->kvaddr, vm->npages);
kva_free(vm->kvaddr, vm->npages * PAGE_SIZE);
}
VM_OBJECT_WLOCK(vm->obj);
for (i = 0; i < vm->npages; i++) {
m = vm->pages[i];
if (m == NULL)
break;
vm_page_busy_acquire(m, 0);
cdev_pager_free_page(vm->obj, m);
m->flags &= ~PG_FICTITIOUS;
vm_page_unwire_noq(m);
vm_page_free(m);
}
vm_pager_deallocate(vm->obj);
VM_OBJECT_WUNLOCK(vm->obj);
free(vm->pages, M_HWT_VM);
}
void
hwt_vm_free(struct hwt_vm *vm)
{
dprintf("%s\n", __func__);
if (vm->cdev)
destroy_dev_sched(vm->cdev);
hwt_vm_destroy_buffers(vm);
free(vm, M_HWT_VM);
}
int
hwt_vm_alloc(size_t bufsize, int kva_req, char *path, struct hwt_vm **vm0)
{
struct hwt_vm *vm;
int error;
vm = malloc(sizeof(struct hwt_vm), M_HWT_VM, M_WAITOK | M_ZERO);
vm->npages = bufsize / PAGE_SIZE;
error = hwt_vm_alloc_buffers(vm, kva_req);
if (error) {
free(vm, M_HWT_VM);
return (error);
}
error = hwt_vm_create_cdev(vm, path);
if (error) {
hwt_vm_free(vm);
return (error);
}
*vm0 = vm;
return (0);
}
sys/dev/hwt/hwt_vm.h
+47
View File
@@ -0,0 +1,47 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#ifndef _DEV_HWT_HWT_VM_H_
#define _DEV_HWT_HWT_VM_H_
struct hwt_vm {
vm_page_t *pages;
int npages;
vm_object_t obj;
vm_offset_t kvaddr;
struct cdev *cdev;
struct hwt_context *ctx;
struct hwt_cpu *cpu; /* cpu mode only. */
struct hwt_thread *thr; /* thr mode only. */
};
int hwt_vm_alloc(size_t bufsize, int kva_req, char *path, struct hwt_vm **vm0);
void hwt_vm_free(struct hwt_vm *vm);
#endif /* !_DEV_HWT_HWT_VM_H_ */
sys/kern/kern_exec.c
+19
View File
@@ -29,6 +29,7 @@
#include <sys/cdefs.h>
#include "opt_capsicum.h"
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include "opt_ktrace.h"
#include "opt_vm.h"
@@ -90,6 +91,10 @@
#include <sys/pmckern.h>
#endif
#ifdef HWT_HOOKS
#include <dev/hwt/hwt_hook.h>
#endif
#include <security/audit/audit.h>
#include <security/mac/mac_framework.h>
@@ -936,6 +941,20 @@ do_execve(struct thread *td, struct image_args *args, struct mac *mac_p,
}
#endif
#ifdef HWT_HOOKS
if ((td->td_proc->p_flag2 & P2_HWT) != 0) {
struct hwt_record_entry ent;
VOP_UNLOCK(imgp->vp);
ent.fullpath = imgp->execpath;
ent.addr = imgp->et_dyn_addr;
ent.baseaddr = imgp->reloc_base;
ent.record_type = HWT_RECORD_EXECUTABLE;
HWT_CALL_HOOK(td, HWT_EXEC, &ent);
vn_lock(imgp->vp, LK_SHARED | LK_RETRY);
}
#endif
/* Set values passed into the program in registers. */
(*p->p_sysent->sv_setregs)(td, imgp, stack_base);
sys/kern/kern_linker.c
+3 -2
View File
@@ -30,6 +30,7 @@
#include "opt_ddb.h"
#include "opt_kld.h"
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include <sys/param.h>
#include <sys/systm.h>
@@ -64,7 +65,7 @@
#include "linker_if.h"
#ifdef HWPMC_HOOKS
#if defined(HWPMC_HOOKS) || defined(HWT_HOOKS)
#include <sys/pmckern.h>
#endif
@@ -2184,7 +2185,7 @@ linker_basename(const char *path)
return (filename);
}
#ifdef HWPMC_HOOKS
#if defined(HWPMC_HOOKS) || defined(HWT_HOOKS)
/*
* Inform hwpmc about the set of kernel modules currently loaded.
*/
sys/kern/kern_pmc.c
+4
View File
@@ -72,6 +72,10 @@ int __read_mostly (*pmc_hook)(struct thread *td, int function, void *arg) = NULL
/* Interrupt handler */
int __read_mostly (*pmc_intr)(struct trapframe *tf) = NULL;
/* HWT hooks */
void __read_mostly (*hwt_hook)(struct thread *td, int func, void *arg) = NULL;
int __read_mostly (*hwt_intr)(struct trapframe *tf) = NULL;
DPCPU_DEFINE(uint8_t, pmc_sampled);
/*
sys/kern/kern_thr.c
+11 -1
View File
@@ -29,7 +29,7 @@
#include "opt_ktrace.h"
#include "opt_posix.h"
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include <sys/systm.h>
#include <sys/kernel.h>
#ifdef KTRACE
@@ -60,6 +60,9 @@
#ifdef HWPMC_HOOKS
#include <sys/pmckern.h>
#endif
#ifdef HWT_HOOKS
#include <dev/hwt/hwt_hook.h>
#endif
#include <machine/frame.h>
@@ -280,6 +283,10 @@ thread_create(struct thread *td, struct rtprio *rtp,
PMC_CALL_HOOK_UNLOCKED(newtd, PMC_FN_THR_CREATE_LOG, NULL);
#endif
#ifdef HWT_HOOKS
HWT_CALL_HOOK(newtd, HWT_THREAD_CREATE, NULL);
#endif
tidhash_add(newtd);
/* ignore timesharing class */
@@ -613,6 +620,9 @@ sys_thr_set_name(struct thread *td, struct thr_set_name_args *uap)
if (PMC_PROC_IS_USING_PMCS(p) || PMC_SYSTEM_SAMPLING_ACTIVE())
PMC_CALL_HOOK_UNLOCKED(ttd, PMC_FN_THR_CREATE_LOG, NULL);
#endif
#ifdef HWT_HOOKS
HWT_CALL_HOOK(ttd, HWT_THREAD_SET_NAME, NULL);
#endif
#ifdef KTR
sched_clear_tdname(ttd);
#endif
sys/kern/kern_thread.c
+9
View File
@@ -30,6 +30,7 @@
#include "opt_witness.h"
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include <sys/systm.h>
#include <sys/asan.h>
@@ -60,6 +61,9 @@
#ifdef HWPMC_HOOKS
#include <sys/pmckern.h>
#endif
#ifdef HWT_HOOKS
#include <dev/hwt/hwt_hook.h>
#endif
#include <sys/priv.h>
#include <security/audit/audit.h>
@@ -1002,6 +1006,11 @@ thread_exit(void)
} else if (PMC_SYSTEM_SAMPLING_ACTIVE())
PMC_CALL_HOOK_UNLOCKED(td, PMC_FN_THR_EXIT_LOG, NULL);
#endif
#ifdef HWT_HOOKS
HWT_CALL_HOOK(td, HWT_THREAD_EXIT, NULL);
#endif
PROC_UNLOCK(p);
PROC_STATLOCK(p);
thread_lock(td);
sys/kern/sched_4bsd.c
+21 -1
View File
@@ -36,6 +36,7 @@
#include <sys/cdefs.h>
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include "opt_sched.h"
#include <sys/param.h>
@@ -63,6 +64,10 @@
#include <sys/pmckern.h>
#endif
#ifdef HWT_HOOKS
#include <dev/hwt/hwt_hook.h>
#endif
#ifdef KDTRACE_HOOKS
#include <sys/dtrace_bsd.h>
int __read_mostly dtrace_vtime_active;
@@ -1075,6 +1080,11 @@ sched_switch(struct thread *td, int flags)
PMC_SWITCH_CONTEXT(td, PMC_FN_CSW_OUT);
#endif
#ifdef HWT_HOOKS
HWT_CALL_HOOK(td, HWT_SWITCH_OUT, NULL);
HWT_CALL_HOOK(newtd, HWT_SWITCH_IN, NULL);
#endif
SDT_PROBE2(sched, , , off__cpu, newtd, newtd->td_proc);
/* I feel sleepy */
@@ -1696,10 +1706,20 @@ sched_idletd(void *dummy)
static void
sched_throw_tail(struct thread *td)
{
struct thread *newtd;
mtx_assert(&sched_lock, MA_OWNED);
KASSERT(curthread->td_md.md_spinlock_count == 1, ("invalid count"));
cpu_throw(td, choosethread()); /* doesn't return */
newtd = choosethread();
#ifdef HWT_HOOKS
if (td)
HWT_CALL_HOOK(td, HWT_SWITCH_OUT, NULL);
HWT_CALL_HOOK(newtd, HWT_SWITCH_IN, NULL);
#endif
cpu_throw(td, newtd); /* doesn't return */
}
/*
sys/kern/sched_ule.c
+19
View File
@@ -39,6 +39,7 @@
#include <sys/cdefs.h>
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include "opt_sched.h"
#include <sys/param.h>
@@ -69,6 +70,10 @@
#include <sys/pmckern.h>
#endif
#ifdef HWT_HOOKS
#include <dev/hwt/hwt_hook.h>
#endif
#ifdef KDTRACE_HOOKS
#include <sys/dtrace_bsd.h>
int __read_mostly dtrace_vtime_active;
@@ -2432,6 +2437,12 @@ sched_switch(struct thread *td, int flags)
if (dtrace_vtime_active)
(*dtrace_vtime_switch_func)(newtd);
#endif
#ifdef HWT_HOOKS
HWT_CALL_HOOK(td, HWT_SWITCH_OUT, NULL);
HWT_CALL_HOOK(newtd, HWT_SWITCH_IN, NULL);
#endif
td->td_oncpu = NOCPU;
cpu_switch(td, newtd, mtx);
cpuid = td->td_oncpu = PCPU_GET(cpuid);
@@ -3252,6 +3263,10 @@ sched_ap_entry(void)
newtd = sched_throw_grab(tdq);
#ifdef HWT_HOOKS
HWT_CALL_HOOK(newtd, HWT_SWITCH_IN, NULL);
#endif
/* doesn't return */
cpu_throw(NULL, newtd);
}
@@ -3278,6 +3293,10 @@ sched_throw(struct thread *td)
newtd = sched_throw_grab(tdq);
#ifdef HWT_HOOKS
HWT_CALL_HOOK(newtd, HWT_SWITCH_IN, NULL);
#endif
/* doesn't return */
cpu_switch(td, newtd, TDQ_LOCKPTR(tdq));
}
sys/kern/vfs_vnops.c
+23
View File
@@ -41,6 +41,7 @@
*/
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include <sys/param.h>
#include <sys/systm.h>
@@ -86,6 +87,10 @@
#include <sys/pmckern.h>
#endif
#ifdef HWT_HOOKS
#include <dev/hwt/hwt_hook.h>
#endif
static fo_rdwr_t vn_read;
static fo_rdwr_t vn_write;
static fo_rdwr_t vn_io_fault;
@@ -3005,6 +3010,24 @@ vn_mmap(struct file *fp, vm_map_t map, vm_offset_t *addr, vm_size_t size,
}
}
#endif
#ifdef HWT_HOOKS
if (HWT_HOOK_INSTALLED && (prot & VM_PROT_EXECUTE) != 0 &&
error == 0) {
struct hwt_record_entry ent;
char *fullpath;
char *freepath;
if (vn_fullpath(vp, &fullpath, &freepath) == 0) {
ent.fullpath = fullpath;
ent.addr = (uintptr_t) *addr;
ent.record_type = HWT_RECORD_MMAP;
HWT_CALL_HOOK(td, HWT_MMAP, &ent);
free(freepath, M_TEMP);
}
}
#endif
return (error);
}
sys/modules/Makefile
+5
View File
@@ -141,6 +141,7 @@ SUBDIR= \
${_hptnr} \
${_hptrr} \
hwpmc \
${_hwt} \
${_hyperv} \
i2c \
${_iavf} \
@@ -859,6 +860,10 @@ _smartpqi= smartpqi
_p2sb= p2sb
.endif
.if ${MACHINE_CPUARCH} == "aarch64" || ${MACHINE_CPUARCH} == "amd64"
_hwt= hwt
.endif
.if ${MACHINE_CPUARCH} == "aarch64" || ${MACHINE_CPUARCH} == "amd64" || \
${MACHINE_CPUARCH} == "riscv"
.if ${MK_BHYVE} != "no" || defined(ALL_MODULES)
sys/modules/hwt/Makefile
+21
View File
@@ -0,0 +1,21 @@
# $FreeBSD$
.PATH: ${SRCTOP}/sys/dev/hwt
KMOD = hwt
SRCS = \
hwt.c \
hwt_backend.c \
hwt_config.c \
hwt_context.c \
hwt_contexthash.c \
hwt_cpu.c \
hwt_hook.c \
hwt_ioctl.c \
hwt_owner.c \
hwt_ownerhash.c \
hwt_record.c \
hwt_thread.c \
hwt_vm.c
.include <bsd.kmod.mk>
sys/sys/hwt.h
+129
View File
@@ -0,0 +1,129 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
/* User-visible header. */
#include <sys/param.h>
#include <sys/cpuset.h>
#include <sys/types.h>
#include <sys/hwt_record.h>
#ifndef _SYS_HWT_H_
#define _SYS_HWT_H_
#define HWT_MAGIC 0x42
#define HWT_IOC_ALLOC _IOW(HWT_MAGIC, 0x00, struct hwt_alloc)
#define HWT_IOC_START _IOW(HWT_MAGIC, 0x01, struct hwt_start)
#define HWT_IOC_STOP _IOW(HWT_MAGIC, 0x02, struct hwt_stop)
#define HWT_IOC_RECORD_GET _IOW(HWT_MAGIC, 0x03, struct hwt_record_get)
#define HWT_IOC_BUFPTR_GET _IOW(HWT_MAGIC, 0x04, struct hwt_bufptr_get)
#define HWT_IOC_SET_CONFIG _IOW(HWT_MAGIC, 0x05, struct hwt_set_config)
#define HWT_IOC_WAKEUP _IOW(HWT_MAGIC, 0x06, struct hwt_wakeup)
#define HWT_IOC_SVC_BUF _IOW(HWT_MAGIC, 0x07, struct hwt_svc_buf)
#define HWT_BACKEND_MAXNAMELEN 256
#define HWT_MODE_THREAD 1
#define HWT_MODE_CPU 2
struct hwt_alloc {
size_t bufsize;
int mode;
pid_t pid; /* thread mode */
cpuset_t *cpu_map; /* cpu mode only */
size_t cpusetsize;
const char *backend_name;
int *ident;
int kqueue_fd;
} __aligned(16);
struct hwt_start {
int reserved;
} __aligned(16);
struct hwt_stop {
int reserved;
} __aligned(16);
struct hwt_wakeup {
int reserved;
} __aligned(16);
struct hwt_record_user_entry {
enum hwt_record_type record_type;
union {
/*
* Used for MMAP, EXECUTABLE, INTERP,
* and KERNEL records.
*/
struct {
char fullpath[MAXPATHLEN];
uintptr_t addr;
uintptr_t baseaddr;
};
/* Used for BUFFER records. */
struct {
int buf_id;
int curpage;
vm_offset_t offset;
};
/* Used for THREAD_* records. */
int thread_id;
};
} __aligned(16);
struct hwt_record_get {
struct hwt_record_user_entry *records;
int *nentries;
int wait;
} __aligned(16);
struct hwt_bufptr_get {
int *ident;
vm_offset_t *offset;
uint64_t *data;
} __aligned(16);
struct hwt_set_config {
/* Configuration of ctx. */
int pause_on_mmap;
/* The following passed to backend as is. */
void *config;
size_t config_size;
int config_version;
} __aligned(16);
struct hwt_svc_buf {
/* The following passed to backend as is. */
void *data;
size_t data_size;
int data_version;
} __aligned(16);
#endif /* !_SYS_HWT_H_ */
sys/sys/hwt_record.h
+70
View File
@@ -0,0 +1,70 @@
/*-
* Copyright (c) 2023-2025 Ruslan Bukin <br@bsdpad.com>
*
* This work was supported by Innovate UK project 105694, "Digital Security
* by Design (DSbD) Technology Platform Prototype".
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
/* User-visible header. */
#ifndef _SYS_HWT_RECORD_H_
#define _SYS_HWT_RECORD_H_
enum hwt_record_type {
HWT_RECORD_MMAP,
HWT_RECORD_MUNMAP,
HWT_RECORD_EXECUTABLE,
HWT_RECORD_KERNEL,
HWT_RECORD_THREAD_CREATE,
HWT_RECORD_THREAD_SET_NAME,
HWT_RECORD_BUFFER
};
#ifdef _KERNEL
struct hwt_record_entry {
TAILQ_ENTRY(hwt_record_entry) next;
enum hwt_record_type record_type;
union {
/*
* Used for MMAP, EXECUTABLE, INTERP,
* and KERNEL records.
*/
struct {
char *fullpath;
uintptr_t addr;
uintptr_t baseaddr;
};
/* Used for BUFFER records. */
struct {
int buf_id;
int curpage;
vm_offset_t offset;
};
/* Used for THREAD_* records. */
int thread_id;
};
};
#endif
#endif /* !_SYS_HWT_RECORD_H_ */
sys/sys/proc.h
+2
View File
@@ -893,6 +893,8 @@ struct proc {
#define P2_LOGSIGEXIT_ENABLE 0x00800000 /* Disable logging on sigexit */
#define P2_LOGSIGEXIT_CTL 0x01000000 /* Override kern.logsigexit */
#define P2_HWT 0x02000000 /* Process is using HWT. */
/* Flags protected by proctree_lock, kept in p_treeflags. */
#define P_TREE_ORPHANED 0x00000001 /* Reparented, on orphan list */
#define P_TREE_FIRST_ORPHAN 0x00000002 /* First element of orphan
sys/vm/vm_mmap.c
+16
View File
@@ -41,6 +41,7 @@
*/
#include "opt_hwpmc_hooks.h"
#include "opt_hwt_hooks.h"
#include "opt_vm.h"
#define EXTERR_CATEGORY EXTERR_CAT_MMAP
@@ -95,6 +96,10 @@
#include <sys/pmckern.h>
#endif
#ifdef HWT_HOOKS
#include <dev/hwt/hwt_hook.h>
#endif
int old_mlock = 0;
SYSCTL_INT(_vm, OID_AUTO, old_mlock, CTLFLAG_RWTUN, &old_mlock, 0,
"Do not apply RLIMIT_MEMLOCK on mlockall");
@@ -613,6 +618,17 @@ kern_munmap(struct thread *td, uintptr_t addr0, size_t size)
#endif
rv = vm_map_delete(map, addr, end);
#ifdef HWT_HOOKS
if (HWT_HOOK_INSTALLED && rv == KERN_SUCCESS) {
struct hwt_record_entry ent;
ent.addr = (uintptr_t) addr;
ent.fullpath = NULL;
ent.record_type = HWT_RECORD_MUNMAP;
HWT_CALL_HOOK(td, HWT_RECORD, &ent);
}
#endif
#ifdef HWPMC_HOOKS
if (rv == KERN_SUCCESS && __predict_false(pmc_handled)) {
/* downgrade the lock to prevent a LOR with the pmc-sx lock */