KekenoBSD/src
3
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

dtrace_fbt.4: Document the DTrace fbt provider

Browse Source 
Reported by:	markj
Reviewed by:	christos, markj (earlier version), ziaee
Obtained from:	Mark Johnston, DTrace, FreeBSD Journal, May 2014
Obtained from:	https://wiki.freebsd.org/DTrace/One-Liners
MFC after:	2 weeks
Relnotes:	yes
This commit is contained in:
Mateusz Piotrowski
2025-06-14 21:26:48 +02:00
parent 31d1c080ab
commit 9388c28878
4 changed files with 343 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cddl/contrib/opensolaris/cmd/dtrace/dtrace.1
+2 -1
View File
@@ -20,7 +20,7 @@
.\"
.\" $FreeBSD$
.\"
.Dd July 14, 2025
.Dd July 16, 2025
.Dt DTRACE 1
.Os
.Sh NAME
@@ -1270,6 +1270,7 @@ Invalid command line options or arguments were specified.
.Xr dwatch 1 ,
.Xr dtrace_audit 4 ,
.Xr dtrace_dtrace 4 ,
.Xr dtrace_fbt 4 ,
.Xr dtrace_io 4 ,
.Xr dtrace_ip 4 ,
.Xr dtrace_kinst 4 ,
share/man/man4/Makefile
+1
View File
@@ -981,6 +981,7 @@ _ccd.4= ccd.4
.if ${MK_CDDL} != "no"
_dtrace_provs= dtrace_audit.4 \
dtrace_dtrace.4 \
dtrace_fbt.4 \
dtrace_io.4 \
dtrace_ip.4 \
dtrace_kinst.4 \
share/man/man4/dtrace_fbt.4
+332
View File
@@ -0,0 +1,332 @@
.\"
.\" SPDX-License-Identifier: BSD-2-Clause
.\"
.\" Copyright (c) 2025 Mateusz Piotrowski <0mp@FreeBSD.org>
.\"
.Dd July 16, 2025
.Dt DTRACE_FBT 4
.Os
.Sh NAME
.Nm dtrace_fbt
.Nd a DTrace provider for dynamic kernel tracing based on function boundaries
.Sh SYNOPSIS
.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry
.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return
.Sh DESCRIPTION
The Function Boundary Tracing
.Pq Nm fbt
provider instruments the entry and return of almost every kernel function
corresponding to an
.Xr elf 5
symbol in the kernel and loaded kernel modules.
.Pp
.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry
fires whenever the
.Ar function
is called.
.Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return
fires when the
.Ar function
returns.
.Pp
The
.Ar module
in the probe description is either the name of the loaded kernel module
or
.Ql kernel
for functions compiled into the kernel.
.Ss Function Boundary Instrumentation
The
.Nm fbt
will always instrument a function's entry, but
its return will be intsrumented so long as it can find a
.Ql ret
instruction.
.Pp
In some cases,
.Nm fbt
cannot instrument a function's entry and/or return.
Refer to subsection
.Sx Frame Pointer
for more details.
.Ss Probe Arguments
The arguments of the entry probe
.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:entry
are the arguments of the traced function call.
.Bl -column -offset indent "Entry Probe Argument" "Definition"
.It Sy Entry Probe Argument Ta Sy Definition
.It Fa args[0] Ta Function's first argument, typed
.Pq e.g., Xr malloc 9 Ap s Ft size_t Fa size
.It Fa args[1] Ta Function's second argument, typed
.Pq e.g., Xr malloc 9 Ap s Ft struct malloc_type Fa *type
.It Fa args[2] Ta Function's third argument, typed
.Pq e.g., Xr malloc 9 Ap s Ft int Fa flags
.It Fa ... Ta ...
.El
.Pp
The arguments of the return probe
.Pq Nm fbt Ns Cm \&: Ns Ar module Ns Cm \&: Ns Ar function Ns Cm \&:return
are
.Fa args[0]
.Po
the offset of the firing return instruction within the function;
useful to tell apart two different return statements in a single function
.Pc
and
.Fa args[1]
.Pq the return value, if any .
.Bl -column -offset indent "Return Probe Argument" "Definition"
.It Sy Return Probe Argument Ta Sy Definition
.It Fa args[0] Ta Offset of the traced return instruction
.It Fa args[1] Ta Function's return value
.Po e.g., a kernel virtual address if returning from a successful
.Xr malloc 9
.Pc
.El
.Pp
Subsection
.Sx Example 2 : Getting Details About Probe's Arguments
shows how to get probe's argument count and types directly with
.Xr dtrace 1
without having to resort to the reading function's source code
or documentation.
.Sh EXAMPLES
.Ss Example 1 : Listing Available FBT Probes
The following example shows how to list all the available
.Nm fbt
probes.
.Bd -literal -offset 2n
# dtrace -l -P fbt
ID PROVIDER MODULE FUNCTION NAME
[...]
31868 fbt kernel hammer_time entry
31869 fbt kernel hammer_time return
[...]
.Ed
.Pp
Since
.Fn hammer_time
is a part of the kernel and not a separate loaded module, the
.Ar module
column displays
.Ql kernel .
.Ss Example 2 : Getting Details About Probe's Arguments
The following example shows how to generate a program stability report of
.Xr malloc 9 Ap s
entry and return probes.
Those reports are useful to view
the probe's number of arguments and their types.
.Bd -literal -offset 2n
# dtrace -l -v -n fbt::malloc:entry
[...]
Argument Types
args[0]: size_t
args[1]: struct malloc_type *
args[2]: int
.Ed
.Pp
The count and types of
.Nm fbt Ns Cm \&::malloc:entry
arguments
match the function signature of
.Xr malloc 9 :
.Va args[0]
is
.Ft size_t ,
.Va args[1]
is
.Ft "struct malloc_type *" ,
and
.Va "args[2]"
is
.Ft int .
.Bd -literal -offset 2n
# dtrace -l -v -n fbt::malloc:return
[...]
Argument Types
args[0]: int
args[1]: void *
.Ed
.Pp
The
.Cm return
probe reports two arguments and their types:
the return instruction offset
.Pq the usual Ft int
and the function's return value, which in this case is
.Ft void * ,
as
.Xr malloc 9
returns a kernel virtual address.
.Ss Example 3 : Counting Kernel Slab Memory Allocation by Function
.Bd -literal -offset 2n
# dtrace -n 'fbt::kmem*:entry { @[probefunc] = count(); }'
dtrace: description 'fbt::kmem*:entry ' matched 47 probes
^C
kmem_alloc_contig 1
kmem_alloc_contig_domainset 1
kmem_cache_reap_active 1
kmem_alloc_contig_pages 2
kmem_free 2
kmem_std_destructor 19
kmem_std_constructor 26
kmem_cache_free 151
kmem_cache_alloc 181
.Ed
.Ss Example 4 : Counting Kernel Slab Memory Allocation by Calling Function
.Bd -literal -offset 2n
# dtrace -q -n 'fbt::kmem*:entry { @[caller] = count(); } END { printa("%40a %@16d\en", @); }'
^C
kernel`contigmalloc+0x33 1
kernel`free+0xd3 1
kernel`kmem_alloc_contig+0x29 1
kernel`kmem_alloc_contig_domainset+0x19a 1
zfs.ko`arc_reap_cb_check+0x16 1
.Ed
.Ss Example 5 : Counting Kernel malloc()'s by Calling Function
.Bd -literal -offset 2n
# dtrace -q -n 'fbt::malloc:entry { @[caller] = count(); } END { printa("%45a %@16d\en", @); }'
^C
kernel`devclass_get_devices+0xa8 1
kernel`sys_ioctl+0xb7 1
dtrace.ko`dtrace_ioctl+0x15c1 1
dtrace.ko`dtrace_ioctl+0x972 2
dtrace.ko`dtrace_dof_create+0x35 2
kernel`kern_poll_kfds+0x2f0 4
kernel`kern_poll_kfds+0x28a 19
.Ed
.Ss Example 6 : Counting Kernel malloc()'s by Kernel Stack Trace
.Bd -literal -offset 2n
# dtrace -q -n 'fbt::malloc:entry { @[stack()] = count(); }'
^C
dtrace.ko`dtrace_dof_create+0x35
dtrace.ko`dtrace_ioctl+0x827
kernel`devfs_ioctl+0xd1
kernel`VOP_IOCTL_APV+0x2a
kernel`vn_ioctl+0xb6
kernel`devfs_ioctl_f+0x1e
kernel`kern_ioctl+0x286
kernel`sys_ioctl+0x12f
kernel`amd64_syscall+0x169
kernel`0xffffffff81092b0b
2
.Ed
.Ss Example 7 : Summarizing vmem_alloc()'s by Arena Name and Size Distribution
.Bd -literal -offset 2n
# dtrace -q -n 'fbt::vmem_alloc:entry { @[args[0]->vm_name] = quantize(arg1); }'
^C
kernel arena dom
value ------------- Distribution ------------- count
2048 | 0
4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@ 4
8192 |@@@@@@@@@@@@@ 2
16384 | 0
.Ed
.Ss Example 8 : Measuring Total Time Spent Executing a Function
This DTrace script measures the total time spent in
.Fn vm_page*
kernel functions.
The
.Fn quantize
aggregation organizes the measurements into power-of-two buckets,
providing a time distribution in nanoseconds for each function.
.Bd -literal -offset 2n
fbt::vm_page*:entry {
self->start = timestamp;
}
fbt::vm_page*:return /self->start/ {
@[probefunc] = quantize(timestamp - self->start);
self->start = 0;
}
.Ed
.Sh SEE ALSO
.Xr dtrace 1 ,
.Xr dtrace_kinst 4 ,
.Xr tracing 7
.Rs
.%A Brendan Gregg
.%A Jim Mauro
.%B DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD
.%I Prentice Hall
.%P pp. 898\(en903
.%D 2011
.%U https://www.brendangregg.com/dtracebook/
.Re
.Rs
.%B The illumos Dynamic Tracing Guide
.%O Chapter fbt Provider
.%D 2008
.%U https://illumos.org/books/dtrace/chp-fbt.html#chp-fbt
.Re
.Sh AUTHORS
This manual page was written by
.An Mateusz Piotrowski Aq Mt 0mp@FreeBSD.org .
.Sh CAVEATS
.Ss Stability and Portability
.Nm fbt
probes are by definition tightly coupled to kernel code; if the code underlying
a script changes, the script may fail to run or may produce incorrect results.
Scripts written for one version of
.Fx
might not work on others,
and almost certainly will not work on other operating systems.
.Pp
Individual
.Nm fbt
probes often do not correspond nicely to logical system events.
For example, consider a DTrace script which prints the destination
address of every IP packet as the kernel hands them over
to the network card driver (NIC).
An
.Nm fbt Ns -based
implementation of such a script is a discouragingly difficult task:
it involves instrumenting at least four different functions in different parts
of the IPv4 and IPv6 code.
At the same time, with the
.Xr dtrace_ip 4
provider the script is a simple one-liner:
.Dl dtrace -n 'ip:::send {printf("%s", args[2]->ip_daddr);}'
.Pp
Make sure to review available
.Xr dtrace 1
providers first
before implementing a custom script with the
.Nm fbt
provider.
If none of the DTrace providers offer the desired probes,
consider adding new statically-defined tracing probes
.Pq Xr SDT 9 .
.Ss Frame Pointer
Inline functions are not instrumentable by
.Nm fbt
as they lack a frame pointer.
A developer might explicitly disable inlining by adding the
.Ql __noinline
attribute to a function definition,
but of course this requires a recompilation of the kernel.
Building the kernel with
.Fl fno-omit-frame-pointer
is another way of preserving frame pointers.
Note, that sometimes compilers will omit the frame pointer in leaf functions,
even when configured with
.Fl fno-omit-frame-pointer .
.Pp
Function returns via a tail call are also not instrumentable by
.Nm fbt .
As a result,
a function might have an entry probe
and a mix of instrumented and uninstrumentable returns.
.Pp
Use
.Xr dtrace_kinst 4
to trace arbitrary instructions inside kernel functions
and work around some of the
limitations
of
.Nm fbt .
.Ss Tracing DTrace
The
.Nm fbt
provider cannot attach to functions inside DTrace provider kernel modules.
share/man/man4/dtrace_kinst.4
+8 -4
View File
@@ -22,7 +22,7 @@
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.Dd February 27, 2023
.Dd July 16, 2025
.Dt DTRACE_KINST 4
.Os
.Sh NAME
@@ -43,10 +43,13 @@ creates probes on-demand, meaning it searches for and parses the function's
instructions each time
.Xr dtrace 1
is run, and not at module load time.
This is in contrast to FBT's load-time parsing, since
This is in contrast to
.Xr dtrace_fbt 4 Ap s
load-time parsing, since
.Nm kinst
can potentially create thousands of probes for just a single function, instead
of up to two (entry and return) in the case of FBT.
of up to two (entry and return) in the case of
.Xr dtrace_fbt 4 .
A result of this is that
.Cm dtrace -l -P kinst
will not match any probes.
@@ -79,7 +82,8 @@ Trace all instructions in
# dtrace -n 'kinst::amd64_syscall:'
.Ed
.Sh SEE ALSO
.Xr dtrace 1
.Xr dtrace 1 ,
.Xr dtrace_fbt 4
.Sh HISTORY
The
.Nm kinst