[lttv.git] / doc / developer / tracing_tools_overview.html

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <title>Tracing Tools</title>
</head>
  <body>

<h1>Tracing Tools</h1>

<p>Tracing is routinely used to help understanding the behavior and performance
of various aspects of the Linux kernel and associated drivers. 
Many of the 80K+ printk statements in the Linux kernel 
serve this purpose, although printk is relatively low
performance and unreliable. The small default printk buffer size coupled with
the low performance brings lost messages as soon as the volume becomes
significant.

<p>For this reason, a number of drivers include their own tracing macros
and infrastructure. A quick search looking for TRACE and related keywords
in the Linux kernel source reveals some form of tracing in at least 
the following files:

<UL>
<LI>./fs/hpfs/hpfs_fn.h
<LI>./fs/smbfs/smb_debug.h
<LI>./fs/autofs/autofs_i.h
<LI>./fs/jffs2/nodelist.h
<LI>./include/linux/wait.h
<LI>./include/linux/parport_pc.h
<LI>./include/linux/amigaffs.h
<LI>./include/linux/parport_pc.h
<LI>./include/linux/ncp_fs.h
<LI>drivers/net/wireless airport and orinoco
<LI>drivers/char/ftape
<LI>drivers/char/dtlk.c
<LI>drivers/char/mwave
<LI>drivers/char/n_r3964.c
<LI>drivers/scsi/qlogicfc.c
<LI>drivers/usb/pwc-if.c
<LI>drivers/usb/hpusbscsi.c
<LI>drivers/acpi/include/acmacros.h
<LI>arch/sparc/kernel/signal.c
<LI>arch/mips/math-emu/cp1emu.c
<LI>drivers/net/wavelan.c
<LI>drivers/net/hp100.c
<LI>drivers/net/wan/lmc/lmc_debug.c
<LI>drivers/net/skfp/h/targetos.h
<LI>drivers/char/ip2main.c
<LI>drivers/scsi/gdth.c
<LI>drivers/scsi/megaraid.c
<LI>drivers/scsi/qlogicisp.c
<LI>drivers/scsi/ips.c
<LI>drivers/scsi/qla1280.c
<LI>drivers/scsi/cpqfcTSstructs.h
<LI>drivers/cdrom/sjcd.c
<LI>drivers/isdn/eicon/sys.h
<LI>drivers/sbus/char/bbc_envctrl.c
<LI>drivers/ide/ide-tape.c
<LI>drivers/video/radeonfb.c
<LI>fs/intermezzo/sysctl.c
<LI>fs/ext3/balloc.c
<LI>net/ipv6/ip6_fib.c
<LI>net/irda/irnet/irnet.h
<UL>

<p>A number of tracing tools have been developed for the Linux kernel.
The best known, particularly in the embedded systems area, is the Linux Trace
Toolkit, <A HREF="http://www.opersys.com/LTT">LTT at 
http://www.opersys.com/LTT</A>. It
comes with a nice graphical user interface and is currently under active
development to add dynamically defined event types and graphical trace
analysis modules.

<P>
The <A HREF="http://lkst.sf.net">Linux Kernel State Tracer at
http://lkst.sf.net</A>was developed by Hitachi and offers basic,
low overhead, tracing functionality. There is no grahical user interface
available.

<P>
MAGNET was recently released. It was initially developed to trace the network
stack and drivers. Its performance has not been optimized for SMP systems.
It is available from 
<A HREF="http://public.lanl.gov/radiant/software/magnet.html">
http://public.lanl.gov/radiant/software/magnet.html
</A>. 

<P>
The IKD patch from Andrea Arcangeli 
<A HREF="ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/">
ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/
</A>
includes ktrace which adds the -pg gcc compilation option 
to specified source files. This adds a call to function <i>mcount</i> 
upon entry in any function compiled with that option. A function <i>mcount</i>
is provided which records in a trace the address of the function entered.
Using the system map, this is later translated into a trace of names of
functions entered.

<H2>Reliability, Availability and Serviceability</H2>

<P>
Tracing may be placed in the larger context of Reliability, Availability and
Serviceability (RAS). The Linux RAS project is probably the most active and
well organized,
<A HREF="http://systemras.sourceforge.net/">
http://systemras.sourceforge.net/
</A>
<A HREF="http://www-124.ibm.com/linux/projects/linuxras/">
http://www-124.ibm.com/linux/projects/linuxras/
</A>.
It links to several underlying projects, including the Linux Trace Toolkit
<A HREF="http://www.opersys.com/LTT">LTT</A>.

<P>
Several other projects within Linux RAS directly relate to tracing.

<H3>Enterprise Event Logging</H3>

<p>The Enterprise Event Logging project,
<A HREF="http://evlog.sourceforge.net/">EVLOG project
at http://evlog.sourceforge.net/</A>, produces traces and thus shares a number
of underlying implementation needs 
(events recording, kernel to user mode transfer,
trace analysis and viewing tools, event types format). The intended purpose
and thus implementation constraints differ significantly, however. 
EVLOG records important system events for two purposes,
to trigger service and security alarms (e.g. weak signals in a magnetic disk,
unauthorized access attempt) and to provide permament records. The volume
is typically low and full context is required for each event. While logging
(EVLOG) is therefore implemented separately from tracing (LTT), some
underlying technology may be reused as appropriate (e.g. kernel hooks,
kernel to user mode data relay...).

<H3>Kernel Crash Dump</H3>

<P>A common symptom of a serious kernel problem is a crash. Traces may
be extremely useful to understand the problem except that, because of the
crash, the important last events in the current trace buffer cannot be 
stored on disk. The Linux Kernel Crash Dump facility (LKCD) at
<A HREF="http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/">
http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/
</A> is used to recover such information, when <i>warm</i> rebooting from a
crash while this information is still available in memory.

<P>LKCD needs to be told how to find the tracing buffers in the memory
(address in a map or signature to look for) and in which file to save
their content.

<H3>Kernel Hooks</H3>

<p>
Kernel hooks, at
<A HREF="http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/">
http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/
</A> are a mechanism to insert hooks at desired locations in the kernel.
Handlers may later be registered to be called at these hooks locations.
When no handler is registered, the cost associated with a hook is almost
negligeable, a few NOPs. Skipping NOPs is even faster than testing a 
global boolean variable. Kernel hooks would be ideally suited for the
dynamic activation of trace points. Furthermore, kernel hooks allow registering
multiple handlers. A same location could have a tracing handler and a
performance tool handler, reducing the number of points needed to be 
inserted in the kernel source code.

<p>Interactive tools may be used to rapidly select groups of hooks to be
activated based on facilities (networking, block devices...), level
of details (core events, detailed events) or severity level (warning, info,
debug).

<p>As part of Kernel Hooks and Dynamic Probes, were defined handlers
which produce tracing information. The tracing data models for Dynamic Probes 
and LTT are fairly similar and may eventually be consolidated.

<H3>Dynamic Probes</H3>

<p>The Dynamic Probes,
<A HREF="http://www-124.ibm.com/linux/projects/kprobes/">
http://www-124.ibm.com/linux/projects/kprobes/
</A>,
allow inserting kernel hooks dynamically in a running kernel, just like
breakpoints in debuggers. The instruction
at the desired location is saved and replaced by an interrupt instruction.
When the interrupt instruction is executed, the handlers are called, the
original instruction restored and executed in single step mode, and the
interrupt instruction is reinserted.

</body>
</html>
Commit	Line	Data
99416aeb	1	<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
	2	<html>
	3	<head>
	4	<title>Tracing Tools</title>
	5	</head>
	6	<body>
	7
	8	<h1>Tracing Tools</h1>
	9
	10	<p>Tracing is routinely used to help understanding the behavior and performance
	11	of various aspects of the Linux kernel and associated drivers.
	12	Many of the 80K+ printk statements in the Linux kernel
	13	serve this purpose, although printk is relatively low
	14	performance and unreliable. The small default printk buffer size coupled with
	15	the low performance brings lost messages as soon as the volume becomes
	16	significant.
	17
	18	<p>For this reason, a number of drivers include their own tracing macros
	19	and infrastructure. A quick search looking for TRACE and related keywords
	20	in the Linux kernel source reveals some form of tracing in at least
	21	the following files:
	22
	23	<UL>
	24	<LI>./fs/hpfs/hpfs_fn.h
	25	<LI>./fs/smbfs/smb_debug.h
	26	<LI>./fs/autofs/autofs_i.h
	27	<LI>./fs/jffs2/nodelist.h
	28	<LI>./include/linux/wait.h
	29	<LI>./include/linux/parport_pc.h
	30	<LI>./include/linux/amigaffs.h
	31	<LI>./include/linux/parport_pc.h
	32	<LI>./include/linux/ncp_fs.h
	33	<LI>drivers/net/wireless airport and orinoco
	34	<LI>drivers/char/ftape
	35	<LI>drivers/char/dtlk.c
	36	<LI>drivers/char/mwave
	37	<LI>drivers/char/n_r3964.c
	38	<LI>drivers/scsi/qlogicfc.c
	39	<LI>drivers/usb/pwc-if.c
	40	<LI>drivers/usb/hpusbscsi.c
	41	<LI>drivers/acpi/include/acmacros.h
	42	<LI>arch/sparc/kernel/signal.c
	43	<LI>arch/mips/math-emu/cp1emu.c
	44	<LI>drivers/net/wavelan.c
	45	<LI>drivers/net/hp100.c
	46	<LI>drivers/net/wan/lmc/lmc_debug.c
	47	<LI>drivers/net/skfp/h/targetos.h
	48	<LI>drivers/char/ip2main.c
	49	<LI>drivers/scsi/gdth.c
	50	<LI>drivers/scsi/megaraid.c
	51	<LI>drivers/scsi/qlogicisp.c
	52	<LI>drivers/scsi/ips.c
	53	<LI>drivers/scsi/qla1280.c
	54	<LI>drivers/scsi/cpqfcTSstructs.h
	55	<LI>drivers/cdrom/sjcd.c
	56	<LI>drivers/isdn/eicon/sys.h
	57	<LI>drivers/sbus/char/bbc_envctrl.c
	58	<LI>drivers/ide/ide-tape.c
	59	<LI>drivers/video/radeonfb.c
	60	<LI>fs/intermezzo/sysctl.c
	61	<LI>fs/ext3/balloc.c
	62	<LI>net/ipv6/ip6_fib.c
	63	<LI>net/irda/irnet/irnet.h
	64	<UL>
65
66	<p>A number of tracing tools have been developed for the Linux kernel.
67	The best known, particularly in the embedded systems area, is the Linux Trace
68	Toolkit, <A HREF="http://www.opersys.com/LTT">LTT at
69	http://www.opersys.com/LTT</A>. It
70	comes with a nice graphical user interface and is currently under active
71	development to add dynamically defined event types and graphical trace
72	analysis modules.
73
74	<P>
75	The <A HREF="http://lkst.sf.net">Linux Kernel State Tracer at
76	http://lkst.sf.net</A>was developed by Hitachi and offers basic,
77	low overhead, tracing functionality. There is no grahical user interface
78	available.
79
80	<P>
81	MAGNET was recently released. It was initially developed to trace the network
82	stack and drivers. Its performance has not been optimized for SMP systems.
83	It is available from
84	<A HREF="http://public.lanl.gov/radiant/software/magnet.html">
85	http://public.lanl.gov/radiant/software/magnet.html
86	</A>.
87
88	<P>
89	The IKD patch from Andrea Arcangeli
90	<A HREF="ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/">
91	ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/ikd/
92	</A>
93	includes ktrace which adds the -pg gcc compilation option
94	to specified source files. This adds a call to function <i>mcount</i>
95	upon entry in any function compiled with that option. A function <i>mcount</i>
96	is provided which records in a trace the address of the function entered.
97	Using the system map, this is later translated into a trace of names of
98	functions entered.
99
100	<H2>Reliability, Availability and Serviceability</H2>
101
102	<P>
103	Tracing may be placed in the larger context of Reliability, Availability and
104	Serviceability (RAS). The Linux RAS project is probably the most active and
105	well organized,
106	<A HREF="http://systemras.sourceforge.net/">
107	http://systemras.sourceforge.net/
108	</A>
109	<A HREF="http://www-124.ibm.com/linux/projects/linuxras/">
110	http://www-124.ibm.com/linux/projects/linuxras/
111	</A>.
112	It links to several underlying projects, including the Linux Trace Toolkit
113	<A HREF="http://www.opersys.com/LTT">LTT</A>.
114
115	<P>
116	Several other projects within Linux RAS directly relate to tracing.
117
118	<H3>Enterprise Event Logging</H3>
119
120	<p>The Enterprise Event Logging project,
121	<A HREF="http://evlog.sourceforge.net/">EVLOG project
122	at http://evlog.sourceforge.net/</A>, produces traces and thus shares a number
123	of underlying implementation needs
124	(events recording, kernel to user mode transfer,
125	trace analysis and viewing tools, event types format). The intended purpose
126	and thus implementation constraints differ significantly, however.
127	EVLOG records important system events for two purposes,
128	to trigger service and security alarms (e.g. weak signals in a magnetic disk,
129	unauthorized access attempt) and to provide permament records. The volume
130	is typically low and full context is required for each event. While logging
131	(EVLOG) is therefore implemented separately from tracing (LTT), some
132	underlying technology may be reused as appropriate (e.g. kernel hooks,
133	kernel to user mode data relay...).
134
135	<H3>Kernel Crash Dump</H3>
136
137	<P>A common symptom of a serious kernel problem is a crash. Traces may
138	be extremely useful to understand the problem except that, because of the
139	crash, the important last events in the current trace buffer cannot be
140	stored on disk. The Linux Kernel Crash Dump facility (LKCD) at
141	<A HREF="http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/">
142	http://oss.software.ibm.com/developer/opensource/linux/projects/flexdump/
143	</A> is used to recover such information, when <i>warm</i> rebooting from a
144	crash while this information is still available in memory.
145
146	<P>LKCD needs to be told how to find the tracing buffers in the memory
147	(address in a map or signature to look for) and in which file to save
148	their content.
149
150	<H3>Kernel Hooks</H3>
151
152	<p>
153	Kernel hooks, at
154	<A HREF="http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/">
155	http://www-124.ibm.com/developerworks/oss/linux/projects/kernelhooks/
156	</A> are a mechanism to insert hooks at desired locations in the kernel.
157	Handlers may later be registered to be called at these hooks locations.
158	When no handler is registered, the cost associated with a hook is almost
159	negligeable, a few NOPs. Skipping NOPs is even faster than testing a
160	global boolean variable. Kernel hooks would be ideally suited for the
161	dynamic activation of trace points. Furthermore, kernel hooks allow registering
162	multiple handlers. A same location could have a tracing handler and a
163	performance tool handler, reducing the number of points needed to be
164	inserted in the kernel source code.
165
166	<p>Interactive tools may be used to rapidly select groups of hooks to be
167	activated based on facilities (networking, block devices...), level
168	of details (core events, detailed events) or severity level (warning, info,
169	debug).
170
171	<p>As part of Kernel Hooks and Dynamic Probes, were defined handlers
172	which produce tracing information. The tracing data models for Dynamic Probes
173	and LTT are fairly similar and may eventually be consolidated.
174
175	<H3>Dynamic Probes</H3>
176
177	<p>The Dynamic Probes,
178	<A HREF="http://www-124.ibm.com/linux/projects/kprobes/">
179	http://www-124.ibm.com/linux/projects/kprobes/
180	</A>,
181	allow inserting kernel hooks dynamically in a running kernel, just like
182	breakpoints in debuggers. The instruction
183	at the desired location is saved and replaced by an interrupt instruction.
184	When the interrupt instruction is executed, the handlers are called, the
185	original instruction restored and executed in single step mode, and the
186	interrupt instruction is reinserted.
187
188	</body>
189	</html>