diff options
Diffstat (limited to 'Documentation/kprobes.txt')
-rw-r--r-- | Documentation/kprobes.txt | 506 |
1 files changed, 506 insertions, 0 deletions
diff --git a/Documentation/kprobes.txt b/Documentation/kprobes.txt new file mode 100644 index 0000000..a79633d --- /dev/null +++ b/Documentation/kprobes.txt @@ -0,0 +1,506 @@ +Title : Kernel Probes (Kprobes) +Authors : Jim Keniston <jkenisto@us.ibm.com> + : Prasanna S Panchamukhi <prasanna@in.ibm.com> + +CONTENTS + +1. Concepts: Kprobes, Jprobes, Return Probes +2. Architectures Supported +3. Configuring Kprobes +4. API Reference +5. Kprobes Features and Limitations +6. Probe Overhead +7. TODO +8. Kprobes Example +9. Jprobes Example +10. Kretprobes Example +Appendix A: The kprobes debugfs interface + +1. Concepts: Kprobes, Jprobes, Return Probes + +Kprobes enables you to dynamically break into any kernel routine and +collect debugging and performance information non-disruptively. You +can trap at almost any kernel code address, specifying a handler +routine to be invoked when the breakpoint is hit. + +There are currently three types of probes: kprobes, jprobes, and +kretprobes (also called return probes). A kprobe can be inserted +on virtually any instruction in the kernel. A jprobe is inserted at +the entry to a kernel function, and provides convenient access to the +function's arguments. A return probe fires when a specified function +returns. + +In the typical case, Kprobes-based instrumentation is packaged as +a kernel module. The module's init function installs ("registers") +one or more probes, and the exit function unregisters them. A +registration function such as register_kprobe() specifies where +the probe is to be inserted and what handler is to be called when +the probe is hit. + +There are also register_/unregister_*probes() functions for batch +registration/unregistration of a group of *probes. These functions +can speed up unregistration process when you have to unregister +a lot of probes at once. + +The next three subsections explain how the different types of +probes work. They explain certain things that you'll need to +know in order to make the best use of Kprobes -- e.g., the +difference between a pre_handler and a post_handler, and how +to use the maxactive and nmissed fields of a kretprobe. But +if you're in a hurry to start using Kprobes, you can skip ahead +to section 2. + +1.1 How Does a Kprobe Work? + +When a kprobe is registered, Kprobes makes a copy of the probed +instruction and replaces the first byte(s) of the probed instruction +with a breakpoint instruction (e.g., int3 on i386 and x86_64). + +When a CPU hits the breakpoint instruction, a trap occurs, the CPU's +registers are saved, and control passes to Kprobes via the +notifier_call_chain mechanism. Kprobes executes the "pre_handler" +associated with the kprobe, passing the handler the addresses of the +kprobe struct and the saved registers. + +Next, Kprobes single-steps its copy of the probed instruction. +(It would be simpler to single-step the actual instruction in place, +but then Kprobes would have to temporarily remove the breakpoint +instruction. This would open a small time window when another CPU +could sail right past the probepoint.) + +After the instruction is single-stepped, Kprobes executes the +"post_handler," if any, that is associated with the kprobe. +Execution then continues with the instruction following the probepoint. + +1.2 How Does a Jprobe Work? + +A jprobe is implemented using a kprobe that is placed on a function's +entry point. It employs a simple mirroring principle to allow +seamless access to the probed function's arguments. The jprobe +handler routine should have the same signature (arg list and return +type) as the function being probed, and must always end by calling +the Kprobes function jprobe_return(). + +Here's how it works. When the probe is hit, Kprobes makes a copy of +the saved registers and a generous portion of the stack (see below). +Kprobes then points the saved instruction pointer at the jprobe's +handler routine, and returns from the trap. As a result, control +passes to the handler, which is presented with the same register and +stack contents as the probed function. When it is done, the handler +calls jprobe_return(), which traps again to restore the original stack +contents and processor state and switch to the probed function. + +By convention, the callee owns its arguments, so gcc may produce code +that unexpectedly modifies that portion of the stack. This is why +Kprobes saves a copy of the stack and restores it after the jprobe +handler has run. Up to MAX_STACK_SIZE bytes are copied -- e.g., +64 bytes on i386. + +Note that the probed function's args may be passed on the stack +or in registers. The jprobe will work in either case, so long as the +handler's prototype matches that of the probed function. + +1.3 Return Probes + +1.3.1 How Does a Return Probe Work? + +When you call register_kretprobe(), Kprobes establishes a kprobe at +the entry to the function. When the probed function is called and this +probe is hit, Kprobes saves a copy of the return address, and replaces +the return address with the address of a "trampoline." The trampoline +is an arbitrary piece of code -- typically just a nop instruction. +At boot time, Kprobes registers a kprobe at the trampoline. + +When the probed function executes its return instruction, control +passes to the trampoline and that probe is hit. Kprobes' trampoline +handler calls the user-specified return handler associated with the +kretprobe, then sets the saved instruction pointer to the saved return +address, and that's where execution resumes upon return from the trap. + +While the probed function is executing, its return address is +stored in an object of type kretprobe_instance. Before calling +register_kretprobe(), the user sets the maxactive field of the +kretprobe struct to specify how many instances of the specified +function can be probed simultaneously. register_kretprobe() +pre-allocates the indicated number of kretprobe_instance objects. + +For example, if the function is non-recursive and is called with a +spinlock held, maxactive = 1 should be enough. If the function is +non-recursive and can never relinquish the CPU (e.g., via a semaphore +or preemption), NR_CPUS should be enough. If maxactive <= 0, it is +set to a default value. If CONFIG_PREEMPT is enabled, the default +is max(10, 2*NR_CPUS). Otherwise, the default is NR_CPUS. + +It's not a disaster if you set maxactive too low; you'll just miss +some probes. In the kretprobe struct, the nmissed field is set to +zero when the return probe is registered, and is incremented every +time the probed function is entered but there is no kretprobe_instance +object available for establishing the return probe. + +1.3.2 Kretprobe entry-handler + +Kretprobes also provides an optional user-specified handler which runs +on function entry. This handler is specified by setting the entry_handler +field of the kretprobe struct. Whenever the kprobe placed by kretprobe at the +function entry is hit, the user-defined entry_handler, if any, is invoked. +If the entry_handler returns 0 (success) then a corresponding return handler +is guaranteed to be called upon function return. If the entry_handler +returns a non-zero error then Kprobes leaves the return address as is, and +the kretprobe has no further effect for that particular function instance. + +Multiple entry and return handler invocations are matched using the unique +kretprobe_instance object associated with them. Additionally, a user +may also specify per return-instance private data to be part of each +kretprobe_instance object. This is especially useful when sharing private +data between corresponding user entry and return handlers. The size of each +private data object can be specified at kretprobe registration time by +setting the data_size field of the kretprobe struct. This data can be +accessed through the data field of each kretprobe_instance object. + +In case probed function is entered but there is no kretprobe_instance +object available, then in addition to incrementing the nmissed count, +the user entry_handler invocation is also skipped. + +2. Architectures Supported + +Kprobes, jprobes, and return probes are implemented on the following +architectures: + +- i386 +- x86_64 (AMD-64, EM64T) +- ppc64 +- ia64 (Does not support probes on instruction slot1.) +- sparc64 (Return probes not yet implemented.) +- arm +- ppc + +3. Configuring Kprobes + +When configuring the kernel using make menuconfig/xconfig/oldconfig, +ensure that CONFIG_KPROBES is set to "y". Under "Instrumentation +Support", look for "Kprobes". + +So that you can load and unload Kprobes-based instrumentation modules, +make sure "Loadable module support" (CONFIG_MODULES) and "Module +unloading" (CONFIG_MODULE_UNLOAD) are set to "y". + +Also make sure that CONFIG_KALLSYMS and perhaps even CONFIG_KALLSYMS_ALL +are set to "y", since kallsyms_lookup_name() is used by the in-kernel +kprobe address resolution code. + +If you need to insert a probe in the middle of a function, you may find +it useful to "Compile the kernel with debug info" (CONFIG_DEBUG_INFO), +so you can use "objdump -d -l vmlinux" to see the source-to-object +code mapping. + +4. API Reference + +The Kprobes API includes a "register" function and an "unregister" +function for each type of probe. The API also includes "register_*probes" +and "unregister_*probes" functions for (un)registering arrays of probes. +Here are terse, mini-man-page specifications for these functions and +the associated probe handlers that you'll write. See the files in the +samples/kprobes/ sub-directory for examples. + +4.1 register_kprobe + +#include <linux/kprobes.h> +int register_kprobe(struct kprobe *kp); + +Sets a breakpoint at the address kp->addr. When the breakpoint is +hit, Kprobes calls kp->pre_handler. After the probed instruction +is single-stepped, Kprobe calls kp->post_handler. If a fault +occurs during execution of kp->pre_handler or kp->post_handler, +or during single-stepping of the probed instruction, Kprobes calls +kp->fault_handler. Any or all handlers can be NULL. + +NOTE: +1. With the introduction of the "symbol_name" field to struct kprobe, +the probepoint address resolution will now be taken care of by the kernel. +The following will now work: + + kp.symbol_name = "symbol_name"; + +(64-bit powerpc intricacies such as function descriptors are handled +transparently) + +2. Use the "offset" field of struct kprobe if the offset into the symbol +to install a probepoint is known. This field is used to calculate the +probepoint. + +3. Specify either the kprobe "symbol_name" OR the "addr". If both are +specified, kprobe registration will fail with -EINVAL. + +4. With CISC architectures (such as i386 and x86_64), the kprobes code +does not validate if the kprobe.addr is at an instruction boundary. +Use "offset" with caution. + +register_kprobe() returns 0 on success, or a negative errno otherwise. + +User's pre-handler (kp->pre_handler): +#include <linux/kprobes.h> +#include <linux/ptrace.h> +int pre_handler(struct kprobe *p, struct pt_regs *regs); + +Called with p pointing to the kprobe associated with the breakpoint, +and regs pointing to the struct containing the registers saved when +the breakpoint was hit. Return 0 here unless you're a Kprobes geek. + +User's post-handler (kp->post_handler): +#include <linux/kprobes.h> +#include <linux/ptrace.h> +void post_handler(struct kprobe *p, struct pt_regs *regs, + unsigned long flags); + +p and regs are as described for the pre_handler. flags always seems +to be zero. + +User's fault-handler (kp->fault_handler): +#include <linux/kprobes.h> +#include <linux/ptrace.h> +int fault_handler(struct kprobe *p, struct pt_regs *regs, int trapnr); + +p and regs are as described for the pre_handler. trapnr is the +architecture-specific trap number associated with the fault (e.g., +on i386, 13 for a general protection fault or 14 for a page fault). +Returns 1 if it successfully handled the exception. + +4.2 register_jprobe + +#include <linux/kprobes.h> +int register_jprobe(struct jprobe *jp) + +Sets a breakpoint at the address jp->kp.addr, which must be the address +of the first instruction of a function. When the breakpoint is hit, +Kprobes runs the handler whose address is jp->entry. + +The handler should have the same arg list and return type as the probed +function; and just before it returns, it must call jprobe_return(). +(The handler never actually returns, since jprobe_return() returns +control to Kprobes.) If the probed function is declared asmlinkage +or anything else that affects how args are passed, the handler's +declaration must match. + +register_jprobe() returns 0 on success, or a negative errno otherwise. + +4.3 register_kretprobe + +#include <linux/kprobes.h> +int register_kretprobe(struct kretprobe *rp); + +Establishes a return probe for the function whose address is +rp->kp.addr. When that function returns, Kprobes calls rp->handler. +You must set rp->maxactive appropriately before you call +register_kretprobe(); see "How Does a Return Probe Work?" for details. + +register_kretprobe() returns 0 on success, or a negative errno +otherwise. + +User's return-probe handler (rp->handler): +#include <linux/kprobes.h> +#include <linux/ptrace.h> +int kretprobe_handler(struct kretprobe_instance *ri, struct pt_regs *regs); + +regs is as described for kprobe.pre_handler. ri points to the +kretprobe_instance object, of which the following fields may be +of interest: +- ret_addr: the return address +- rp: points to the corresponding kretprobe object +- task: points to the corresponding task struct +- data: points to per return-instance private data; see "Kretprobe + entry-handler" for details. + +The regs_return_value(regs) macro provides a simple abstraction to +extract the return value from the appropriate register as defined by +the architecture's ABI. + +The handler's return value is currently ignored. + +4.4 unregister_*probe + +#include <linux/kprobes.h> +void unregister_kprobe(struct kprobe *kp); +void unregister_jprobe(struct jprobe *jp); +void unregister_kretprobe(struct kretprobe *rp); + +Removes the specified probe. The unregister function can be called +at any time after the probe has been registered. + +NOTE: +If the functions find an incorrect probe (ex. an unregistered probe), +they clear the addr field of the probe. + +4.5 register_*probes + +#include <linux/kprobes.h> +int register_kprobes(struct kprobe **kps, int num); +int register_kretprobes(struct kretprobe **rps, int num); +int register_jprobes(struct jprobe **jps, int num); + +Registers each of the num probes in the specified array. If any +error occurs during registration, all probes in the array, up to +the bad probe, are safely unregistered before the register_*probes +function returns. +- kps/rps/jps: an array of pointers to *probe data structures +- num: the number of the array entries. + +NOTE: +You have to allocate(or define) an array of pointers and set all +of the array entries before using these functions. + +4.6 unregister_*probes + +#include <linux/kprobes.h> +void unregister_kprobes(struct kprobe **kps, int num); +void unregister_kretprobes(struct kretprobe **rps, int num); +void unregister_jprobes(struct jprobe **jps, int num); + +Removes each of the num probes in the specified array at once. + +NOTE: +If the functions find some incorrect probes (ex. unregistered +probes) in the specified array, they clear the addr field of those +incorrect probes. However, other probes in the array are +unregistered correctly. + +5. Kprobes Features and Limitations + +Kprobes allows multiple probes at the same address. Currently, +however, there cannot be multiple jprobes on the same function at +the same time. + +In general, you can install a probe anywhere in the kernel. +In particular, you can probe interrupt handlers. Known exceptions +are discussed in this section. + +The register_*probe functions will return -EINVAL if you attempt +to install a probe in the code that implements Kprobes (mostly +kernel/kprobes.c and arch/*/kernel/kprobes.c, but also functions such +as do_page_fault and notifier_call_chain). + +If you install a probe in an inline-able function, Kprobes makes +no attempt to chase down all inline instances of the function and +install probes there. gcc may inline a function without being asked, +so keep this in mind if you're not seeing the probe hits you expect. + +A probe handler can modify the environment of the probed function +-- e.g., by modifying kernel data structures, or by modifying the +contents of the pt_regs struct (which are restored to the registers +upon return from the breakpoint). So Kprobes can be used, for example, +to install a bug fix or to inject faults for testing. Kprobes, of +course, has no way to distinguish the deliberately injected faults +from the accidental ones. Don't drink and probe. + +Kprobes makes no attempt to prevent probe handlers from stepping on +each other -- e.g., probing printk() and then calling printk() from a +probe handler. If a probe handler hits a probe, that second probe's +handlers won't be run in that instance, and the kprobe.nmissed member +of the second probe will be incremented. + +As of Linux v2.6.15-rc1, multiple handlers (or multiple instances of +the same handler) may run concurrently on different CPUs. + +Kprobes does not use mutexes or allocate memory except during +registration and unregistration. + +Probe handlers are run with preemption disabled. Depending on the +architecture, handlers may also run with interrupts disabled. In any +case, your handler should not yield the CPU (e.g., by attempting to +acquire a semaphore). + +Since a return probe is implemented by replacing the return +address with the trampoline's address, stack backtraces and calls +to __builtin_return_address() will typically yield the trampoline's +address instead of the real return address for kretprobed functions. +(As far as we can tell, __builtin_return_address() is used only +for instrumentation and error reporting.) + +If the number of times a function is called does not match the number +of times it returns, registering a return probe on that function may +produce undesirable results. In such a case, a line: +kretprobe BUG!: Processing kretprobe d000000000041aa8 @ c00000000004f48c +gets printed. With this information, one will be able to correlate the +exact instance of the kretprobe that caused the problem. We have the +do_exit() case covered. do_execve() and do_fork() are not an issue. +We're unaware of other specific cases where this could be a problem. + +If, upon entry to or exit from a function, the CPU is running on +a stack other than that of the current task, registering a return +probe on that function may produce undesirable results. For this +reason, Kprobes doesn't support return probes (or kprobes or jprobes) +on the x86_64 version of __switch_to(); the registration functions +return -EINVAL. + +6. Probe Overhead + +On a typical CPU in use in 2005, a kprobe hit takes 0.5 to 1.0 +microseconds to process. Specifically, a benchmark that hits the same +probepoint repeatedly, firing a simple handler each time, reports 1-2 +million hits per second, depending on the architecture. A jprobe or +return-probe hit typically takes 50-75% longer than a kprobe hit. +When you have a return probe set on a function, adding a kprobe at +the entry to that function adds essentially no overhead. + +Here are sample overhead figures (in usec) for different architectures. +k = kprobe; j = jprobe; r = return probe; kr = kprobe + return probe +on same function; jr = jprobe + return probe on same function + +i386: Intel Pentium M, 1495 MHz, 2957.31 bogomips +k = 0.57 usec; j = 1.00; r = 0.92; kr = 0.99; jr = 1.40 + +x86_64: AMD Opteron 246, 1994 MHz, 3971.48 bogomips +k = 0.49 usec; j = 0.76; r = 0.80; kr = 0.82; jr = 1.07 + +ppc64: POWER5 (gr), 1656 MHz (SMT disabled, 1 virtual CPU per physical CPU) +k = 0.77 usec; j = 1.31; r = 1.26; kr = 1.45; jr = 1.99 + +7. TODO + +a. SystemTap (http://sourceware.org/systemtap): Provides a simplified +programming interface for probe-based instrumentation. Try it out. +b. Kernel return probes for sparc64. +c. Support for other architectures. +d. User-space probes. +e. Watchpoint probes (which fire on data references). + +8. Kprobes Example + +See samples/kprobes/kprobe_example.c + +9. Jprobes Example + +See samples/kprobes/jprobe_example.c + +10. Kretprobes Example + +See samples/kprobes/kretprobe_example.c + +For additional information on Kprobes, refer to the following URLs: +http://www-106.ibm.com/developerworks/library/l-kprobes.html?ca=dgr-lnxw42Kprobe +http://www.redhat.com/magazine/005mar05/features/kprobes/ +http://www-users.cs.umn.edu/~boutcher/kprobes/ +http://www.linuxsymposium.org/2006/linuxsymposium_procv2.pdf (pages 101-115) + + +Appendix A: The kprobes debugfs interface + +With recent kernels (> 2.6.20) the list of registered kprobes is visible +under the /debug/kprobes/ directory (assuming debugfs is mounted at /debug). + +/debug/kprobes/list: Lists all registered probes on the system + +c015d71a k vfs_read+0x0 +c011a316 j do_fork+0x0 +c03dedc5 r tcp_v4_rcv+0x0 + +The first column provides the kernel address where the probe is inserted. +The second column identifies the type of probe (k - kprobe, r - kretprobe +and j - jprobe), while the third column specifies the symbol+offset of +the probe. If the probed function belongs to a module, the module name +is also specified. + +/debug/kprobes/enabled: Turn kprobes ON/OFF + +Provides a knob to globally turn registered kprobes ON or OFF. By default, +all kprobes are enabled. By echoing "0" to this file, all registered probes +will be disarmed, till such time a "1" is echoed to this file. |