[lttv.git] / ltt / branches / poly / doc / developer / format.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/2002/REC-xhtml1-20020801/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">

<head>
  <title>The LTTng trace format</title>
</head>

<body>

<h1>The LTTng trace format</h1>

<p>
<em>Last update: 2008/06/02</em>
</p>

<p>
This document describes the LTTng trace format. It should be useful mainly to
developers who code the LTTng tracer or the traceread LTTV library, as this
library offers all the necessary abstractions on top of the raw trace data.
</p>

<p>
A trace is contained in a directory tree. To send a trace remotely, the
directory tree may be tar-gzipped. The trace <tt>foo</tt>, placed in the home
directory of user john, /home/john, would have the following contents:
</p>

<pre><tt>
$ cd /home/john
$ tree foo
foo/
|-- control
|   |-- facilities_0
|   |-- facilities_1
|   |-- facilities_...
|   |-- interrupts_0
|   |-- interrupts_1
|   |-- interrupts_...
|   |-- modules_0
|   |-- modules_1
|   |-- modules_...
|   `-- processes_0
|   `-- processes_1
|   `-- processes_...
|-- cpu_0
|-- cpu_1
`-- cpu_...

</tt></pre>

<p>
The root directory contains a tracefile for each cpu, numbered from 0, 
in .trace format. A uniprocessor thus only contains the file cpu_0. 
A multi-processor with some unused (possibly hotplug) CPU slots may have some
unused CPU numbers. For instance an 8 way SMP board with 6 CPUs randomly 
installed may produce tracefiles named 0, 1, 2, 4, 6, 7.
</p>

<p>
The files in the control directory also follow the .trace format and are
also per cpu.  The "facilities" files only contain "core" marker_id,
marker_format and time_heartbeat events. The first two are used to describe the
events that are in the trace. The other control files contain the initial
system state and various subsequent important events, for example process
creations and exit. The interest of placing such subsequent events in control
trace files instead of (or in addition to) in the per cpu trace files is that
they may be accessed more quickly/conveniently and that they may be kept even
when the per cpu files are overwritten in "flight recorder mode".
</p>

<h2>Trace format</h2>

<p>
Each tracefile is divided into equal size blocks with a header at the beginning
of the block. Events are packed sequentially in the block starting right after
the block header.
</p>

<p>
Each block consists of :
</p>

<pre><tt>
block start/end header
trace header
event 1 header
event 1 variable length data
event 2 header
event 2 variable length data
....
padding
</tt></pre>

<h3>The block start/end header</h3>

<pre><tt>
begin
	* the beginning of buffer information
	uint64 cycle_count
		* TSC at the beginning of the buffer
	uint64 freq
		* frequency of the CPUs at the beginning of the buffer.
end
	* the end of buffer information
	uint64 cycle_count
		* TSC at the end of the buffer
	uint64 freq
		* frequency of the CPUs at the end of the buffer.
uint32 lost_size
	* number of bytes of padding at the end of the buffer.
uint32 buf_size
	* size of the sub-buffer.
</tt></pre>


<h3>The trace header</h3>

<pre><tt>
uint32 magic_number
	* 0x00D6B7ED, used to check the trace byte order vs host byte order.
uint32 arch_type
	* Architecture type of the traced machine.
uint32 arch_variant
	* Architecture variant of the traced machine. May be unused on some arch.
uint32 float_word_order
	* Byte order of floats and doubles, sometimes different from integer byte
	  order. Useful only for user space traces.
uint8 arch_size
	* Size (in bytes) of the void * on the traced machine.
uint8 major_version
	* major version of the trace.
uint8 minor_version
	* minor version of the trace.
uint8 flight_recorder
	* Is flight recorder mode activated ? If yes, data might be missing
	  (overwritten) in the trace.
uint8	has_heartbeat
	* Does this trace have heartbeat timer event activated ?
		Yes (1) -> Event header has 32 bits TSC
		No (0) -> Event header has 64 bits TSC
uint8 alignment
	* Are event headers in this trace aligned ?
		Yes -> the value indicates the alignment
		No (0) -> data is packed.
uint8 tsc_lsb_truncate
uint8 tscbits
uint8 compact_data_shift
uint32 freq_scale
		event time is always calculated from :
			trace_start_time + ((event_tsc - trace_start_tsc) * (freq / freq_scale))
uint64 start_freq
	* CPUs clock frequency at the beginnig of the trace.
uint64 start_tsc
	* TSC at the beginning of the trace.
uint64 start_monotonic
	* monotonically increasing time at the beginning of the trace.
		(currently not supported)
start_time
	* Real time at the beginning of the trace (as given by date, adjusted by NTP)
		This is the only time reference with the real world : the rest of the trace
		has monotonically increasing time from this point (with TSC difference and
		clock frequency).
	uint32 seconds
	uint32 nanoseconds
</tt></pre>


<h3>Event header</h3>

<p>
Event headers differ according to the following conditions : does the
traced system have a heartbeat timer? Is tracing alignment activated?
</p>

<p>
Event header :
</p>
<pre><tt>
{ uint32 timestamp
	or
	uint64 timestamp }
	* if has_heartbeat : 32 LSB of the cycle counter at the event record time.
	* else : 64 bits complete cycle counter.
uint8 facility_id
	* Numerical ID of the facility corresponding to the event. See the facility
	  tracefile to know which facility ID matches which facility name and
		description.
uint8 event_id
	* Numerical ID of the event inside the facility.
uint16 event_size
	* Size of the variable length data that follows this header.
</tt></pre>

<p>
Event header alignment
</p>

<p>
If trace alignment is activated (<tt>alignment</tt>), the event header is
aligned. In addition, padding is automatically added after the event header so
the variable length data is automatically aligned on the architecture size.
</p>

<!--
<h2>System description</h2>

<p>
The system type description, in system.xml, looks like:
</p>

<pre><tt>
&lt;system 
 node_name="vaucluse"
 domainname="polymtl.ca" 
 cpu=4
 arch_size="ILP32" 
 endian="little" 
 kernel_name="Linux" 
 kernel_release="2.4.18-686-smp" 
 kernel_version="#1 SMP Sun Apr 14 12:07:19 EST 2002"
 machine="i686" 
 processor="unknown" 
 hardware_platform="unknown"
 operating_system="Linux" 
 ltt_major_version="2"
 ltt_minor_version="0"
 ltt_block_size="100000"
&gt;
Some comments about the system
&lt;/system&gt;
</tt></pre>

<p>
The system attributes kernel_name, node_name, kernel_release,
 kernel_version, machine, processor, hardware_platform and operating_system
come from the uname(1) program. The domainname attribute is obtained from
the "hostname &#045;&#045;domain" command. The arch_size attribute is one of
LP32, ILP32, LP64 or ILP64 and specifies the length in bits of integers (I),
long (L) and pointers (P). The endian attribute is "little" or "big".
While the arch_size and endian attributes could be deduced from the platform
type, having these explicit allows analysing traces from yet unknown
platforms. The cpu attribute specifies the maximum number of processors in
the system; only tracefiles 0 to this maximum - 1 may exist in the cpu
directory.
</p>

<p>
Within the system element, the text enclosed may describe further the
system traced.
</p>


<h2>Bookmarks</h2>

<p>
Bookmarks are user supplied information added to a trace. They contain user
annotations attached to a time interval.
</p>


<pre><tt>
&lt;bookmarks&gt;
  &lt;location name=name cpu=n start_time=t end_time=t&gt;Some text&lt;/location&gt;
  ...
&lt;/bookmarks&gt;
</tt></pre>

<p>
The interval is defined using either "time=" or "start_time=" and 
"end_time=", or "cycle=" or "start_cycle=" and "end_cycle=". 
The time is in seconds with decimals up to nanoseconds and cycle counts 
are unsigned integers with a 64 bits range. The cpu attribute is optional.
</p>

-->
</body>
</html>
Commit	Line	Data
	1	<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/2002/REC-xhtml1-20020801/DTD/xhtml1-strict.dtd">
	2	<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
	3
	4	<head>
	5	<title>The LTTng trace format</title>
	6	</head>
	7
	8	<body>
	9
	10	<h1>The LTTng trace format</h1>
	11
	12	<p>
	13	<em>Last update: 2008/06/02</em>
	14	</p>
	15
	16	<p>
	17	This document describes the LTTng trace format. It should be useful mainly to
	18	developers who code the LTTng tracer or the traceread LTTV library, as this
	19	library offers all the necessary abstractions on top of the raw trace data.
	20	</p>
	21
	22	<p>
	23	A trace is contained in a directory tree. To send a trace remotely, the
	24	directory tree may be tar-gzipped. The trace <tt>foo</tt>, placed in the home
	25	directory of user john, /home/john, would have the following contents:
	26	</p>
	27
	28	<pre><tt>
	29	$ cd /home/john
	30	$ tree foo
	31	foo/
	32	\|-- control
	33	\| \|-- facilities_0
	34	\| \|-- facilities_1
	35	\| \|-- facilities_...
	36	\| \|-- interrupts_0
	37	\| \|-- interrupts_1
	38	\| \|-- interrupts_...
	39	\| \|-- modules_0
	40	\| \|-- modules_1
	41	\| \|-- modules_...
	42	\| `-- processes_0
	43	\| `-- processes_1
	44	\| `-- processes_...
	45	\|-- cpu_0
	46	\|-- cpu_1
	47	`-- cpu_...
	48
	49	</tt></pre>
	50
	51	<p>
	52	The root directory contains a tracefile for each cpu, numbered from 0,
	53	in .trace format. A uniprocessor thus only contains the file cpu_0.
	54	A multi-processor with some unused (possibly hotplug) CPU slots may have some
	55	unused CPU numbers. For instance an 8 way SMP board with 6 CPUs randomly
	56	installed may produce tracefiles named 0, 1, 2, 4, 6, 7.
	57	</p>
	58
	59	<p>
	60	The files in the control directory also follow the .trace format and are
	61	also per cpu. The "facilities" files only contain "core" marker_id,
	62	marker_format and time_heartbeat events. The first two are used to describe the
	63	events that are in the trace. The other control files contain the initial
	64	system state and various subsequent important events, for example process
	65	creations and exit. The interest of placing such subsequent events in control
	66	trace files instead of (or in addition to) in the per cpu trace files is that
	67	they may be accessed more quickly/conveniently and that they may be kept even
	68	when the per cpu files are overwritten in "flight recorder mode".
	69	</p>
	70
	71	<h2>Trace format</h2>
	72
	73	<p>
	74	Each tracefile is divided into equal size blocks with a header at the beginning
	75	of the block. Events are packed sequentially in the block starting right after
	76	the block header.
	77	</p>
	78
	79	<p>
	80	Each block consists of :
	81	</p>
	82
	83	<pre><tt>
	84	block start/end header
	85	trace header
	86	event 1 header
	87	event 1 variable length data
	88	event 2 header
	89	event 2 variable length data
	90	....
	91	padding
	92	</tt></pre>
	93
	94	<h3>The block start/end header</h3>
	95
	96	<pre><tt>
	97	begin
	98	* the beginning of buffer information
	99	uint64 cycle_count
	100	* TSC at the beginning of the buffer
	101	uint64 freq
	102	* frequency of the CPUs at the beginning of the buffer.
	103	end
	104	* the end of buffer information
	105	uint64 cycle_count
	106	* TSC at the end of the buffer
	107	uint64 freq
	108	* frequency of the CPUs at the end of the buffer.
	109	uint32 lost_size
	110	* number of bytes of padding at the end of the buffer.
	111	uint32 buf_size
	112	* size of the sub-buffer.
	113	</tt></pre>
	114
	115
	116
	117	<h3>The trace header</h3>
	118
	119	<pre><tt>
	120	uint32 magic_number
	121	* 0x00D6B7ED, used to check the trace byte order vs host byte order.
	122	uint32 arch_type
	123	* Architecture type of the traced machine.
	124	uint32 arch_variant
	125	* Architecture variant of the traced machine. May be unused on some arch.
	126	uint32 float_word_order
	127	* Byte order of floats and doubles, sometimes different from integer byte
	128	order. Useful only for user space traces.
	129	uint8 arch_size
	130	* Size (in bytes) of the void * on the traced machine.
	131	uint8 major_version
	132	* major version of the trace.
	133	uint8 minor_version
	134	* minor version of the trace.
	135	uint8 flight_recorder
	136	* Is flight recorder mode activated ? If yes, data might be missing
	137	(overwritten) in the trace.
	138	uint8 has_heartbeat
	139	* Does this trace have heartbeat timer event activated ?
	140	Yes (1) -> Event header has 32 bits TSC
	141	No (0) -> Event header has 64 bits TSC
	142	uint8 alignment
	143	* Are event headers in this trace aligned ?
	144	Yes -> the value indicates the alignment
	145	No (0) -> data is packed.
	146	uint8 tsc_lsb_truncate
	147	uint8 tscbits
	148	uint8 compact_data_shift
	149	uint32 freq_scale
	150	event time is always calculated from :
	151	trace_start_time + ((event_tsc - trace_start_tsc) * (freq / freq_scale))
	152	uint64 start_freq
	153	* CPUs clock frequency at the beginnig of the trace.
	154	uint64 start_tsc
	155	* TSC at the beginning of the trace.
	156	uint64 start_monotonic
	157	* monotonically increasing time at the beginning of the trace.
	158	(currently not supported)
	159	start_time
	160	* Real time at the beginning of the trace (as given by date, adjusted by NTP)
	161	This is the only time reference with the real world : the rest of the trace
	162	has monotonically increasing time from this point (with TSC difference and
	163	clock frequency).
	164	uint32 seconds
	165	uint32 nanoseconds
	166	</tt></pre>
	167
	168
	169	<h3>Event header</h3>
	170
	171	<p>
	172	Event headers differ according to the following conditions : does the
	173	traced system have a heartbeat timer? Is tracing alignment activated?
	174	</p>
	175
	176	<p>
	177	Event header :
	178	</p>
	179	<pre><tt>
	180	{ uint32 timestamp
	181	or
	182	uint64 timestamp }
	183	* if has_heartbeat : 32 LSB of the cycle counter at the event record time.
	184	* else : 64 bits complete cycle counter.
	185	uint8 facility_id
	186	* Numerical ID of the facility corresponding to the event. See the facility
	187	tracefile to know which facility ID matches which facility name and
	188	description.
	189	uint8 event_id
	190	* Numerical ID of the event inside the facility.
	191	uint16 event_size
	192	* Size of the variable length data that follows this header.
	193	</tt></pre>
	194
	195	<p>
	196	Event header alignment
	197	</p>
	198
	199	<p>
	200	If trace alignment is activated (<tt>alignment</tt>), the event header is
	201	aligned. In addition, padding is automatically added after the event header so
	202	the variable length data is automatically aligned on the architecture size.
	203	</p>
	204
	205	<!--
	206	<h2>System description</h2>
	207
	208	<p>
	209	The system type description, in system.xml, looks like:
	210	</p>
	211
	212	<pre><tt>
	213	<system
	214	node_name="vaucluse"
	215	domainname="polymtl.ca"
	216	cpu=4
	217	arch_size="ILP32"
	218	endian="little"
	219	kernel_name="Linux"
	220	kernel_release="2.4.18-686-smp"
	221	kernel_version="#1 SMP Sun Apr 14 12:07:19 EST 2002"
	222	machine="i686"
	223	processor="unknown"
	224	hardware_platform="unknown"
	225	operating_system="Linux"
	226	ltt_major_version="2"
	227	ltt_minor_version="0"
	228	ltt_block_size="100000"
	229	>
	230	Some comments about the system
	231	</system>
	232	</tt></pre>
	233
	234	<p>
	235	The system attributes kernel_name, node_name, kernel_release,
	236	kernel_version, machine, processor, hardware_platform and operating_system
	237	come from the uname(1) program. The domainname attribute is obtained from
	238	the "hostname --domain" command. The arch_size attribute is one of
	239	LP32, ILP32, LP64 or ILP64 and specifies the length in bits of integers (I),
	240	long (L) and pointers (P). The endian attribute is "little" or "big".
	241	While the arch_size and endian attributes could be deduced from the platform
	242	type, having these explicit allows analysing traces from yet unknown
	243	platforms. The cpu attribute specifies the maximum number of processors in
	244	the system; only tracefiles 0 to this maximum - 1 may exist in the cpu
	245	directory.
	246	</p>
	247
	248	<p>
	249	Within the system element, the text enclosed may describe further the
	250	system traced.
	251	</p>
	252
	253
	254	<h2>Bookmarks</h2>
	255
	256	<p>
	257	Bookmarks are user supplied information added to a trace. They contain user
	258	annotations attached to a time interval.
	259	</p>
	260
	261
	262	<pre><tt>
	263	<bookmarks>
	264	<location name=name cpu=n start_time=t end_time=t>Some text</location>
	265	...
	266	</bookmarks>
	267	</tt></pre>
	268
	269	<p>
	270	The interval is defined using either "time=" or "start_time=" and
	271	"end_time=", or "cycle=" or "start_cycle=" and "end_cycle=".
	272	The time is in seconds with decimals up to nanoseconds and cycle counts
	273	are unsigned integers with a 64 bits range. The cpu attribute is optional.
	274	</p>
	275
	276	-->
	277	</body>
	278	</html>