Fix: consumerd: consumed size miscomputed during statistics sampling

Issue observed
==============

When specifying a size-based automatic rotation, sessions sometimes continuously
rotate after the first size-based rotation is triggered.

Steps to reproduce:

  $ ./lttng create the_test
  $ ./lttng enable-channel -u the_channel
  $ ./lttng enable-rotation --size=32M
  $ ./lttng enable-event -ua --channel the_channel
  $ ./lttng start
  # Produce more than 32MiB of tracing data and stop the application(s), then
  # observe a rotation occuring on every expiration of the channel's monitor timer.

Cause
=====

The consumer daemon samples statistics of the various streams it monitors on
every expiration of their channel's "monitor" timer. One of the statistic it
maintains is the channel's total consumed size.

Once the stats are collected, the consumer sends a channel statistics sample
message to the session daemon. One of the fields of this message is the amount
of data that was consumed for the channel since the last statistics sample was
sent. The session daemon relies on that information to aggregate the consumed
size a session based on the consumed sizes of its various channels.

The consumer maintains a per-channel 'last_consumed_size_sample_sent' which is
used to compute the amount of data consumed since the last statistics sample was
sent to the session daemon. The computation is erroneous since the count sent to
the session daemon is 'the channel's total consumed data amount' minus 'the data
consumed since the last statistics sample'.

Solution
========

The counter maintained on a per-channel basis is now
consumed_size_as_of_last_sample_sent: the total amount of data consumed for that
channel as of the last statistics sample. On every expiration of the monitor
timer, the total amount of data consumed for the channel is determined and the
difference from that counter is sent to the sessiond daemon. The counter is then
updated with the latest computed value.

Known drawbacks
===============

None.

Change-Id: I327fde946826ab1da0b6511a13132f3afba1ac16
Signed-off-by: Jérémie Galarneau <jeremie.galarneau@efficios.com>

diff --git a/src/common/consumer/consumer-timer.cpp b/src/common/consumer/consumer-timer.cpp
index 2ae7f0f73d1b2fdaffe59ed6b21b3814b78e8c22..70094ef4f98e30c5da46e9c8b087e3bd5dfdd7da 100644 (file)
--- a/src/common/consumer/consumer-timer.cpp
+++ b/src/common/consumer/consumer-timer.cpp
@@ -687,7 +687,8 @@ void sample_and_send_channel_buffer_stats(struct lttng_consumer_channel *channel
 
        msg.highest = highest;
        msg.lowest = lowest;
-       msg.consumed_since_last_sample = total_consumed - channel->last_consumed_size_sample_sent;
+       msg.consumed_since_last_sample =
+               total_consumed - channel->consumed_size_as_of_last_sample_sent;
 
        /*
         * Writes performed here are assumed to be atomic which is only
@@ -712,7 +713,7 @@ void sample_and_send_channel_buffer_stats(struct lttng_consumer_channel *channel
                    channel->key,
                    msg.highest,
                    msg.lowest);
-               channel->last_consumed_size_sample_sent = msg.consumed_since_last_sample;
+               channel->consumed_size_as_of_last_sample_sent = total_consumed;
        }
 }
 

diff --git a/src/common/consumer/consumer.hpp b/src/common/consumer/consumer.hpp
index 851b3e2b8137eaef4d5904b19deb81a3f9921bdf..cad2e2c5611139d814168807e73a0a3abad1fcc4 100644 (file)
--- a/src/common/consumer/consumer.hpp
+++ b/src/common/consumer/consumer.hpp
@@ -258,7 +258,7 @@ struct lttng_consumer_channel {
        uint64_t lost_packets = 0;
 
        bool streams_sent_to_relayd = false;
-       uint64_t last_consumed_size_sample_sent = false;
+       uint64_t consumed_size_as_of_last_sample_sent = 0;
 };
 
 struct stream_subbuffer {