[lttv.git] / trunk / lttv / doc / developer / tracing_tools_overview.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <title>Tracing Tools</title>
</head>
  <body>

<h1>Tracing Tools</h1>

<p>Tracing is routinely used to help understanding the behavior and performance
of various aspects of the Linux kernel and associated drivers. 
Many of the 80K+ printk statements in the Linux kernel 
serve this purpose, although printk is relatively low
performance and unreliable. The small default printk buffer size coupled with
the low performance brings lost messages as soon as the volume becomes
significant.

<p>For this reason, a number of drivers include their own tracing macros
and infrastructure. A quick search looking for TRACE and related keywords
in the Linux kernel source reveals some form of tracing in at least 
the following files:

<UL>
<LI>./fs/hpfs/hpfs_fn.h
<LI>./fs/smbfs/smb_debug.h
<LI>./fs/autofs/autofs_i.h
<LI>./fs/jffs2/nodelist.h
<LI>./include/linux/wait.h
<LI>./include/linux/parport_pc.h
<LI>./include/linux/amigaffs.h
<LI>./include/linux/parport_pc.h
<LI>./include/linux/ncp_fs.h
<LI>drivers/net/wireless airport and orinoco
<LI>drivers/char/ftape
<LI>drivers/char/dtlk.c
<LI>drivers/char/mwave
<LI>drivers/char/n_r3964.c
<LI>drivers/scsi/qlogicfc.c
<LI>drivers/usb/pwc-if.c
<LI>drivers/usb/hpusbscsi.c
<LI>drivers/acpi/include/acmacros.h
<LI>arch/sparc/kernel/signal.c
<LI>arch/mips/math-emu/cp1emu.c
<LI>drivers/net/wavelan.c
<LI>drivers/net/hp100.c
<LI>drivers/net/wan/lmc/lmc_debug.c
<LI>drivers/net/skfp/h/targetos.h
<LI>drivers/char/ip2main.c
<LI>drivers/scsi/gdth.c
<LI>drivers/scsi/megaraid.c
<LI>drivers/scsi/qlogicisp.c
<LI>drivers/scsi/ips.c
<LI>drivers/scsi/qla1280.c
<LI>drivers/scsi/cpqfcTSstructs.h
<LI>drivers/cdrom/sjcd.c
<LI>drivers/isdn/eicon/sys.h
<LI>drivers/sbus/char/bbc_envctrl.c
<LI>drivers/ide/ide-tape.c
<LI>drivers/video/radeonfb.c
<LI>fs/intermezzo/sysctl.c
<LI>fs/ext3/balloc.c
<LI>net/ipv6/ip6_fib.c
<LI>net/irda/irnet/irnet.h
<UL>

<p>A number of tracing tools have been developed for the Linux kernel.
The best known, particularly in the embedded systems area, is the Linux Trace
Toolkit, <A HREF="http://www.opersys.com/LTT">LTT at 
http://www.opersys.com/LTT</A>. It
comes with a nice graphical user interface and is currently under active
development to add dynamically defined event types and graphical trace
analysis modules.

<P>
The <A HREF="http://lkst.sf.net">Linux Kernel State Tracer at
http://lkst.sf.net</A>was developed by Hitachi and offers basic,
low overhead, tracing functionality. There is no grahical user interface
available.

<P>
MAGNET was recently released. It was initially developed to trace the network
stack and drivers. Its performance has not been optimized for SMP systems.
It is available from 
<A HREF="http://public.lanl.gov/radiant/software/magnet.html">
http://public.lanl.gov/radiant/software/magnet.html
</A>. 

<P>
The IKD patch from Andrea Arcangeli 
<A HREF="ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/">
ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/
</A>
includes ktrace which adds the -pg gcc compilation option 
to specified source files. This adds a call to function <i>mcount</i> 
upon entry in any function compiled with that option. A function <i>mcount</i>
is provided which records in a trace the address of the function entered.
Using the system map, this is later translated into a trace of names of
functions entered.

<H2>Reliability, Availability and Serviceability</H2>

<P>
Tracing may be placed in the larger context of Reliability, Availability and
Serviceability (RAS). The Linux RAS project is probably the most active and
well organized,
<A HREF="http://systemras.sourceforge.net/">
http://systemras.sourceforge.net/
</A>
<A HREF="http://www-124.ibm.com/linux/projects/linuxras/">
http://www-124.ibm.com/linux/projects/linuxras/
</A>.
It links to several underlying projects, including the Linux Trace Toolkit
<A HREF="http://www.opersys.com/LTT">LTT</A>.

<P>
Several other projects within Linux RAS directly relate to tracing.

<H3>Enterprise Event Logging</H3>

<p>The Enterprise Event Logging project,
<A HREF="http://evlog.sourceforge.net/">EVLOG project
at http://evlog.sourceforge.net/</A>, produces traces and thus shares a number
of underlying implementation needs 
(events recording, kernel to user mode transfer,
trace analysis and viewing tools, event types format). The intended purpose
and thus implementation constraints differ significantly, however. 
EVLOG records important system events for two purposes,
to trigger service and security alarms (e.g. weak signals in a magnetic disk,
unauthorized access attempt) and to provide permament records. The volume
is typically low and full context is required for each event. While logging
(EVLOG) is therefore implemented separately from tracing (LTT), some
underlying technology may be reused as appropriate (e.g. kernel hooks,
kernel to user mode data relay...).

<H3>Kernel Crash Dump</H3>

<P>A common symptom of a serious kernel problem is a crash. Traces may
be extremely useful to understand the problem except that, because of the
crash, the important last events in the current trace buffer cannot be 
stored on disk. The Linux Kernel Crash Dump facility (LKCD) at
<A HREF="http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/">
http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/
</A> is used to recover such information, when <i>warm</i> rebooting from a
crash while this information is still available in memory.

<P>LKCD needs to be told how to find the tracing buffers in the memory
(address in a map or signature to look for) and in which file to save
their content.

<H3>Kernel Hooks</H3>

<p>
Kernel hooks, at
<A HREF="http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/">
http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/
</A> are a mechanism to insert hooks at desired locations in the kernel.
Handlers may later be registered to be called at these hooks locations.
When no handler is registered, the cost associated with a hook is almost
negligeable, a few NOPs. Skipping NOPs is even faster than testing a 
global boolean variable. Kernel hooks would be ideally suited for the
dynamic activation of trace points. Furthermore, kernel hooks allow registering
multiple handlers. A same location could have a tracing handler and a
performance tool handler, reducing the number of points needed to be 
inserted in the kernel source code.

<p>Interactive tools may be used to rapidly select groups of hooks to be
activated based on facilities (networking, block devices...), level
of details (core events, detailed events) or severity level (warning, info,
debug).

<p>As part of Kernel Hooks and Dynamic Probes, were defined handlers
which produce tracing information. The tracing data models for Dynamic Probes 
and LTT are fairly similar and may eventually be consolidated.

<H3>Dynamic Probes</H3>

<p>The Dynamic Probes,
<A HREF="http://www-124.ibm.com/linux/projects/kprobes/">
http://www-124.ibm.com/linux/projects/kprobes/
</A>,
allow inserting kernel hooks dynamically in a running kernel, just like
breakpoints in debuggers. The instruction
at the desired location is saved and replaced by an interrupt instruction.
When the interrupt instruction is executed, the handlers are called, the
original instruction restored and executed in single step mode, and the
interrupt instruction is reinserted.

</body>
</html>
Commit	Line	Data
	1	<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
	2	<html>
	3	<head>
	4	<title>Tracing Tools</title>
	5	</head>
	6	<body>
	7
	8	<h1>Tracing Tools</h1>
	9
	10	<p>Tracing is routinely used to help understanding the behavior and performance
	11	of various aspects of the Linux kernel and associated drivers.
	12	Many of the 80K+ printk statements in the Linux kernel
	13	serve this purpose, although printk is relatively low
	14	performance and unreliable. The small default printk buffer size coupled with
	15	the low performance brings lost messages as soon as the volume becomes
	16	significant.
	17
	18	<p>For this reason, a number of drivers include their own tracing macros
	19	and infrastructure. A quick search looking for TRACE and related keywords
	20	in the Linux kernel source reveals some form of tracing in at least
	21	the following files:
	22
	23	<UL>
	24	<LI>./fs/hpfs/hpfs_fn.h
	25	<LI>./fs/smbfs/smb_debug.h
	26	<LI>./fs/autofs/autofs_i.h
	27	<LI>./fs/jffs2/nodelist.h
	28	<LI>./include/linux/wait.h
	29	<LI>./include/linux/parport_pc.h
	30	<LI>./include/linux/amigaffs.h
	31	<LI>./include/linux/parport_pc.h
	32	<LI>./include/linux/ncp_fs.h
	33	<LI>drivers/net/wireless airport and orinoco
	34	<LI>drivers/char/ftape
	35	<LI>drivers/char/dtlk.c
	36	<LI>drivers/char/mwave
	37	<LI>drivers/char/n_r3964.c
	38	<LI>drivers/scsi/qlogicfc.c
	39	<LI>drivers/usb/pwc-if.c
	40	<LI>drivers/usb/hpusbscsi.c
	41	<LI>drivers/acpi/include/acmacros.h
	42	<LI>arch/sparc/kernel/signal.c
	43	<LI>arch/mips/math-emu/cp1emu.c
	44	<LI>drivers/net/wavelan.c
	45	<LI>drivers/net/hp100.c
	46	<LI>drivers/net/wan/lmc/lmc_debug.c
	47	<LI>drivers/net/skfp/h/targetos.h
	48	<LI>drivers/char/ip2main.c
	49	<LI>drivers/scsi/gdth.c
	50	<LI>drivers/scsi/megaraid.c
	51	<LI>drivers/scsi/qlogicisp.c
	52	<LI>drivers/scsi/ips.c
	53	<LI>drivers/scsi/qla1280.c
	54	<LI>drivers/scsi/cpqfcTSstructs.h
	55	<LI>drivers/cdrom/sjcd.c
	56	<LI>drivers/isdn/eicon/sys.h
	57	<LI>drivers/sbus/char/bbc_envctrl.c
	58	<LI>drivers/ide/ide-tape.c
	59	<LI>drivers/video/radeonfb.c
	60	<LI>fs/intermezzo/sysctl.c
	61	<LI>fs/ext3/balloc.c
	62	<LI>net/ipv6/ip6_fib.c
	63	<LI>net/irda/irnet/irnet.h
	64	<UL>
	65
	66	<p>A number of tracing tools have been developed for the Linux kernel.
	67	The best known, particularly in the embedded systems area, is the Linux Trace
	68	Toolkit, <A HREF="http://www.opersys.com/LTT">LTT at
	69	http://www.opersys.com/LTT</A>. It
	70	comes with a nice graphical user interface and is currently under active
	71	development to add dynamically defined event types and graphical trace
	72	analysis modules.
	73
	74	<P>
	75	The <A HREF="http://lkst.sf.net">Linux Kernel State Tracer at
	76	http://lkst.sf.net</A>was developed by Hitachi and offers basic,
	77	low overhead, tracing functionality. There is no grahical user interface
	78	available.
	79
	80	<P>
	81	MAGNET was recently released. It was initially developed to trace the network
	82	stack and drivers. Its performance has not been optimized for SMP systems.
	83	It is available from
	84	<A HREF="http://public.lanl.gov/radiant/software/magnet.html">
	85	http://public.lanl.gov/radiant/software/magnet.html
	86	</A>.
	87
	88	<P>
	89	The IKD patch from Andrea Arcangeli
	90	<A HREF="ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/">
	91	ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/
	92	</A>
	93	includes ktrace which adds the -pg gcc compilation option
	94	to specified source files. This adds a call to function <i>mcount</i>
	95	upon entry in any function compiled with that option. A function <i>mcount</i>
	96	is provided which records in a trace the address of the function entered.
	97	Using the system map, this is later translated into a trace of names of
	98	functions entered.
	99
	100	<H2>Reliability, Availability and Serviceability</H2>
	101
	102	<P>
	103	Tracing may be placed in the larger context of Reliability, Availability and
	104	Serviceability (RAS). The Linux RAS project is probably the most active and
	105	well organized,
	106	<A HREF="http://systemras.sourceforge.net/">
	107	http://systemras.sourceforge.net/
	108	</A>
	109	<A HREF="http://www-124.ibm.com/linux/projects/linuxras/">
	110	http://www-124.ibm.com/linux/projects/linuxras/
	111	</A>.
	112	It links to several underlying projects, including the Linux Trace Toolkit
	113	<A HREF="http://www.opersys.com/LTT">LTT</A>.
	114
	115	<P>
	116	Several other projects within Linux RAS directly relate to tracing.
	117
	118	<H3>Enterprise Event Logging</H3>
	119
	120	<p>The Enterprise Event Logging project,
	121	<A HREF="http://evlog.sourceforge.net/">EVLOG project
	122	at http://evlog.sourceforge.net/</A>, produces traces and thus shares a number
	123	of underlying implementation needs
	124	(events recording, kernel to user mode transfer,
	125	trace analysis and viewing tools, event types format). The intended purpose
	126	and thus implementation constraints differ significantly, however.
	127	EVLOG records important system events for two purposes,
	128	to trigger service and security alarms (e.g. weak signals in a magnetic disk,
	129	unauthorized access attempt) and to provide permament records. The volume
	130	is typically low and full context is required for each event. While logging
	131	(EVLOG) is therefore implemented separately from tracing (LTT), some
	132	underlying technology may be reused as appropriate (e.g. kernel hooks,
	133	kernel to user mode data relay...).
	134
	135	<H3>Kernel Crash Dump</H3>
	136
	137	<P>A common symptom of a serious kernel problem is a crash. Traces may
	138	be extremely useful to understand the problem except that, because of the
	139	crash, the important last events in the current trace buffer cannot be
	140	stored on disk. The Linux Kernel Crash Dump facility (LKCD) at
	141	<A HREF="http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/">
	142	http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/
	143	</A> is used to recover such information, when <i>warm</i> rebooting from a
	144	crash while this information is still available in memory.
	145
	146	<P>LKCD needs to be told how to find the tracing buffers in the memory
	147	(address in a map or signature to look for) and in which file to save
	148	their content.
	149
	150	<H3>Kernel Hooks</H3>
	151
	152	<p>
	153	Kernel hooks, at
	154	<A HREF="http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/">
	155	http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/
	156	</A> are a mechanism to insert hooks at desired locations in the kernel.
	157	Handlers may later be registered to be called at these hooks locations.
	158	When no handler is registered, the cost associated with a hook is almost
	159	negligeable, a few NOPs. Skipping NOPs is even faster than testing a
	160	global boolean variable. Kernel hooks would be ideally suited for the
	161	dynamic activation of trace points. Furthermore, kernel hooks allow registering
	162	multiple handlers. A same location could have a tracing handler and a
	163	performance tool handler, reducing the number of points needed to be
	164	inserted in the kernel source code.
	165
	166	<p>Interactive tools may be used to rapidly select groups of hooks to be
	167	activated based on facilities (networking, block devices...), level
	168	of details (core events, detailed events) or severity level (warning, info,
	169	debug).
	170
	171	<p>As part of Kernel Hooks and Dynamic Probes, were defined handlers
	172	which produce tracing information. The tracing data models for Dynamic Probes
	173	and LTT are fairly similar and may eventually be consolidated.
	174
	175	<H3>Dynamic Probes</H3>
	176
	177	<p>The Dynamic Probes,
	178	<A HREF="http://www-124.ibm.com/linux/projects/kprobes/">
	179	http://www-124.ibm.com/linux/projects/kprobes/
	180	</A>,
	181	allow inserting kernel hooks dynamically in a running kernel, just like
	182	breakpoints in debuggers. The instruction
	183	at the desired location is saved and replaced by an interrupt instruction.
	184	When the interrupt instruction is executed, the handlers are called, the
	185	original instruction restored and executed in single step mode, and the
	186	interrupt instruction is reinserted.
	187
	188	</body>
	189	</html>