From: Francis Deslauriers Date: Tue, 23 Mar 2021 23:50:30 +0000 (-0400) Subject: Fix: sessiond: notification: use after free of trigger object X-Git-Tag: v2.13.0-rc1~170 X-Git-Url: https://git.lttng.org./?a=commitdiff_plain;h=091fa780af74dc1a93eaeff50d8c0baf1de8c41f;p=lttng-tools.git Fix: sessiond: notification: use after free of trigger object Background ========== * Clients can subscribe to certain specific conditions (e.g. buffer usage) using the `lttng_notification_channel_subscribe()` function. * This subscription is only useful once a trigger with that condition AND at least one notify action is registered. * The sessiond keeps a list for client subscribed to each registered condition. * More than one trigger with the same condition may be registered the sessiond at the same time if they have different actions. Issue ===== Currently, when registering a trigger (T1) the sessiond looks if there is already a client list for the condition of this trigger. If not, the sessiond links the newly created client list object to that trigger T1 by keeping a pointer to it. This means that if another trigger (T2) is registered with the same condition (but a different name, or different actions) it will reuse the same client list object and use the pointer to the T1. This causes problems if T1 is unregistered before T2. In that case, the pointer to T1 in the client list object is pointing to a deallocated trigger object. This issue is not encountered with the current test suite, namely the `test_notification_multi_app` test case, because triggers with the same condition also had the same action, so they are considered identical and are not registered. This issue was first witnessed when adding a trigger name comparison in the `lttng_trigger_is_equal()` function. Fix === Change the client list object so that it has its own copy of the condition and a list of dependent triggers. Each trigger with that condition has a reference on this client list object. When unregistering a trigger, the notification thread removes it for the client list's triggers list and put its reference on the client list object. Tests ===== This commit adds a parameter to the base_client that dictates if the notify action should be in a group. This is a trick to create triggers that are not equal but have the same behaviour. The `test_notification_multi_app` test case is modified to turn on this option every other trigger registration. Semi-related cleanups ===================== * Merge `notification_client_list_create()` and `publish_notification_client_list()` functions since they are used together anyway. This removes the need to call `notification_client_list_put()` at the end of the `_register_trigger()` function * Rename `trigger_applies_to_client()` to `condition_applies_to_client()`. Signed-off-by: Francis Deslauriers Signed-off-by: Jérémie Galarneau Change-Id: I3ebb90a1a64236a440a085e6fc1b82726a0e5af9 --- diff --git a/src/bin/lttng-sessiond/condition-internal.c b/src/bin/lttng-sessiond/condition-internal.c index 07b32f86d..50bb6cb1a 100644 --- a/src/bin/lttng-sessiond/condition-internal.c +++ b/src/bin/lttng-sessiond/condition-internal.c @@ -135,3 +135,35 @@ unsigned long lttng_condition_hash(const struct lttng_condition *condition) abort(); } } + +LTTNG_HIDDEN +struct lttng_condition *lttng_condition_copy(const struct lttng_condition *condition) +{ + int ret; + struct lttng_payload copy_buffer; + struct lttng_condition *copy = NULL; + + lttng_payload_init(©_buffer); + + ret = lttng_condition_serialize(condition, ©_buffer); + if (ret < 0) { + goto end; + } + + { + struct lttng_payload_view view = + lttng_payload_view_from_payload( + ©_buffer, 0, -1); + + ret = lttng_condition_create_from_payload( + &view, ©); + if (ret < 0) { + copy = NULL; + goto end; + } + } + +end: + lttng_payload_reset(©_buffer); + return copy; +} diff --git a/src/bin/lttng-sessiond/condition-internal.h b/src/bin/lttng-sessiond/condition-internal.h index 2863f7bd8..270a8b1af 100644 --- a/src/bin/lttng-sessiond/condition-internal.h +++ b/src/bin/lttng-sessiond/condition-internal.h @@ -17,4 +17,6 @@ */ unsigned long lttng_condition_hash(const struct lttng_condition *condition); +struct lttng_condition *lttng_condition_copy( + const struct lttng_condition *condition); #endif /* LTTNG_SESSIOND_CONDITION_INTERNAL_H */ diff --git a/src/bin/lttng-sessiond/notification-thread-events.c b/src/bin/lttng-sessiond/notification-thread-events.c index 5ec3ed279..ee20725dd 100644 --- a/src/bin/lttng-sessiond/notification-thread-events.c +++ b/src/bin/lttng-sessiond/notification-thread-events.c @@ -122,6 +122,7 @@ struct lttng_trigger_ht_element { struct lttng_trigger *trigger; struct cds_lfht_node node; struct cds_lfht_node node_by_name_uid; + struct cds_list_head client_list_trigger_node; /* call_rcu delayed reclaim. */ struct rcu_head rcu_node; }; @@ -314,7 +315,7 @@ int match_client_list_condition(struct cds_lfht_node *node, const void *key) client_list = caa_container_of(node, struct notification_client_list, notification_trigger_clients_ht_node); - condition = lttng_trigger_get_const_condition(client_list->trigger); + condition = client_list->condition; return !!lttng_condition_is_equal(condition_key, condition); } @@ -661,62 +662,121 @@ void notification_client_list_release(struct urcu_ref *list_ref) container_of(list_ref, typeof(*list), ref); struct notification_client_list_element *client_list_element, *tmp; + lttng_condition_put(list->condition); + if (list->notification_trigger_clients_ht) { rcu_read_lock(); + cds_lfht_del(list->notification_trigger_clients_ht, &list->notification_trigger_clients_ht_node); rcu_read_unlock(); list->notification_trigger_clients_ht = NULL; } cds_list_for_each_entry_safe(client_list_element, tmp, - &list->list, node) { + &list->clients_list, node) { free(client_list_element); } + + assert(cds_list_empty(&list->triggers_list)); + pthread_mutex_destroy(&list->lock); call_rcu(&list->rcu_node, free_notification_client_list_rcu); } +static +bool condition_applies_to_client(const struct lttng_condition *condition, + struct notification_client *client) +{ + bool applies = false; + struct lttng_condition_list_element *condition_list_element; + + cds_list_for_each_entry(condition_list_element, &client->condition_list, + node) { + applies = lttng_condition_is_equal( + condition_list_element->condition, + condition); + if (applies) { + break; + } + } + + return applies; +} + static struct notification_client_list *notification_client_list_create( - const struct lttng_trigger *trigger) + struct notification_thread_state *state, + const struct lttng_condition *condition) { - struct notification_client_list *client_list = - zmalloc(sizeof(*client_list)); + struct notification_client *client; + struct cds_lfht_iter iter; + struct notification_client_list *client_list; + client_list = zmalloc(sizeof(*client_list)); if (!client_list) { - goto error; + PERROR("Failed to allocate notification client list"); + goto end; } + pthread_mutex_init(&client_list->lock, NULL); + /* + * The trigger that owns the condition has the first reference to this + * client list. + */ urcu_ref_init(&client_list->ref); cds_lfht_node_init(&client_list->notification_trigger_clients_ht_node); - CDS_INIT_LIST_HEAD(&client_list->list); - client_list->trigger = trigger; -error: - return client_list; -} + CDS_INIT_LIST_HEAD(&client_list->clients_list); + CDS_INIT_LIST_HEAD(&client_list->triggers_list); -static -void publish_notification_client_list( - struct notification_thread_state *state, - struct notification_client_list *list) -{ - const struct lttng_condition *condition = - lttng_trigger_get_const_condition(list->trigger); + /* + * Create a copy of the condition so that it's independent of any + * trigger. The client list may outlive the trigger object (which owns + * the condition) that is used to create it. + */ + client_list->condition = lttng_condition_copy(condition); + + /* Build a list of clients to which this new condition applies. */ + cds_lfht_for_each_entry (state->client_socket_ht, &iter, client, + client_socket_ht_node) { + struct notification_client_list_element *client_list_element; - assert(!list->notification_trigger_clients_ht); - notification_client_list_get(list); + if (!condition_applies_to_client(condition, client)) { + continue; + } + + client_list_element = zmalloc(sizeof(*client_list_element)); + if (!client_list_element) { + goto error_put_client_list; + } - list->notification_trigger_clients_ht = + CDS_INIT_LIST_HEAD(&client_list_element->node); + client_list_element->client = client; + cds_list_add(&client_list_element->node, &client_list->clients_list); + } + + client_list->notification_trigger_clients_ht = state->notification_trigger_clients_ht; rcu_read_lock(); - cds_lfht_add(state->notification_trigger_clients_ht, - lttng_condition_hash(condition), - &list->notification_trigger_clients_ht_node); + /* + * Add the client list to the global list of client list. + */ + cds_lfht_add_unique(state->notification_trigger_clients_ht, + lttng_condition_hash(client_list->condition), + match_client_list_condition, + client_list->condition, + &client_list->notification_trigger_clients_ht_node); rcu_read_unlock(); + goto end; + +error_put_client_list: + notification_client_list_put(client_list); + client_list = NULL; + +end: + return client_list; } -LTTNG_HIDDEN void notification_client_list_put(struct notification_client_list *list) { if (!list) { @@ -1004,12 +1064,11 @@ int evaluate_condition_for_client(const struct lttng_trigger *trigger, * subscribing. */ cds_lfht_node_init(&client_list.notification_trigger_clients_ht_node); - CDS_INIT_LIST_HEAD(&client_list.list); - client_list.trigger = trigger; + CDS_INIT_LIST_HEAD(&client_list.clients_list); CDS_INIT_LIST_HEAD(&client_list_element.node); client_list_element.client = client; - cds_list_add(&client_list_element.node, &client_list.list); + cds_list_add(&client_list_element.node, &client_list.clients_list); /* Send evaluation result to the newly-subscribed client. */ DBG("[notification-thread] Newly subscribed-to condition evaluated to true, notifying client"); @@ -1083,13 +1142,19 @@ int notification_thread_client_subscribe(struct notification_client *client, * This is correct since the list doesn't own the trigger and the * object is immutable. */ - if (evaluate_condition_for_client(client_list->trigger, condition, - client, state)) { - WARN("[notification-thread] Evaluation of a condition on client subscription failed, aborting."); - ret = -1; - free(client_list_element); - goto end; + struct lttng_trigger_ht_element *trigger_ht_element; + pthread_mutex_lock(&client_list->lock); + cds_list_for_each_entry(trigger_ht_element, + &client_list->triggers_list, client_list_trigger_node) { + if (evaluate_condition_for_client(trigger_ht_element->trigger, condition, + client, state)) { + WARN("[notification-thread] Evaluation of a condition on client subscription failed, aborting."); + ret = -1; + free(client_list_element); + goto end; + } } + pthread_mutex_unlock(&client_list->lock); /* * Add the client to the list of clients interested in a given trigger @@ -1100,7 +1165,7 @@ int notification_thread_client_subscribe(struct notification_client *client, CDS_INIT_LIST_HEAD(&client_list_element->node); pthread_mutex_lock(&client_list->lock); - cds_list_add(&client_list_element->node, &client_list->list); + cds_list_add(&client_list_element->node, &client_list->clients_list); pthread_mutex_unlock(&client_list->lock); end: if (_status) { @@ -1171,7 +1236,7 @@ int notification_thread_client_unsubscribe( pthread_mutex_lock(&client_list->lock); cds_list_for_each_entry_safe(client_list_element, client_tmp, - &client_list->list, node) { + &client_list->clients_list, node) { if (client_list_element->client->id != client->id) { continue; } @@ -1364,25 +1429,6 @@ fail: return false; } -static -bool trigger_applies_to_client(struct lttng_trigger *trigger, - struct notification_client *client) -{ - bool applies = false; - struct lttng_condition_list_element *condition_list_element; - - cds_list_for_each_entry(condition_list_element, &client->condition_list, - node) { - applies = lttng_condition_is_equal( - condition_list_element->condition, - lttng_trigger_get_condition(trigger)); - if (applies) { - break; - } - } - return applies; -} - /* Must be called with RCU read lock held. */ static struct lttng_session_trigger_list *get_session_trigger_list( @@ -2566,12 +2612,9 @@ int handle_notification_thread_command_register_trigger( { int ret = 0; struct lttng_condition *condition; - struct notification_client *client; struct notification_client_list *client_list = NULL; struct lttng_trigger_ht_element *trigger_ht_element = NULL; - struct notification_client_list_element *client_list_element; struct cds_lfht_node *node; - struct cds_lfht_iter iter; const char* trigger_name; bool free_trigger = true; struct lttng_evaluation *evaluation = NULL; @@ -2670,58 +2713,50 @@ int handle_notification_thread_command_register_trigger( } } - /* - * Ownership of the trigger and of its wrapper was transfered to - * the triggers_ht. Same for token ht element if necessary. - */ - trigger_ht_element = NULL; - free_trigger = false; - /* * The rest only applies to triggers that have a "notify" action. * It is not skipped as this is the only action type currently * supported. */ if (is_trigger_action_notify(trigger)) { - client_list = notification_client_list_create(trigger); + /* + * Find or create the client list of this condition. It may + * already be present if another trigger is already registered + * with the same condition. + */ + client_list = get_client_list_from_condition(state, condition); if (!client_list) { - ret = -1; - goto error_free_ht_element; - } - - /* Build a list of clients to which this new trigger applies. */ - cds_lfht_for_each_entry (state->client_socket_ht, &iter, client, - client_socket_ht_node) { - if (!trigger_applies_to_client(trigger, client)) { - continue; - } - - client_list_element = - zmalloc(sizeof(*client_list_element)); - if (!client_list_element) { - ret = -1; - goto error_put_client_list; + /* + * No client list for this condition yet. We create new + * one and build it up. + */ + client_list = notification_client_list_create(state, condition); + if (!client_list) { + ERR("Error creating notification client list for trigger %s", trigger->name); + goto error_free_ht_element; } - - CDS_INIT_LIST_HEAD(&client_list_element->node); - client_list_element->client = client; - cds_list_add(&client_list_element->node, - &client_list->list); } - /* - * Client list ownership transferred to the - * notification_trigger_clients_ht. - */ - publish_notification_client_list(state, client_list); + CDS_INIT_LIST_HEAD(&trigger_ht_element->client_list_trigger_node); + + pthread_mutex_lock(&client_list->lock); + cds_list_add(&trigger_ht_element->client_list_trigger_node, &client_list->triggers_list); + pthread_mutex_unlock(&client_list->lock); } + /* + * Ownership of the trigger and of its wrapper was transfered to + * the triggers_ht. Same for token ht element if necessary. + */ + trigger_ht_element = NULL; + free_trigger = false; + switch (get_condition_binding_object(condition)) { case LTTNG_OBJECT_TYPE_SESSION: /* Add the trigger to the list if it matches a known session. */ ret = bind_trigger_to_matching_session(trigger, state); if (ret) { - goto error_put_client_list; + goto error_free_ht_element; } break; case LTTNG_OBJECT_TYPE_CHANNEL: @@ -2731,7 +2766,7 @@ int handle_notification_thread_command_register_trigger( */ ret = bind_trigger_to_matching_channels(trigger, state); if (ret) { - goto error_put_client_list; + goto error_free_ht_element; } break; case LTTNG_OBJECT_TYPE_NONE: @@ -2739,7 +2774,7 @@ int handle_notification_thread_command_register_trigger( default: ERR("Unknown object type on which to bind a newly registered trigger was encountered"); ret = -1; - goto error_put_client_list; + goto error_free_ht_element; } /* @@ -2797,7 +2832,7 @@ int handle_notification_thread_command_register_trigger( if (ret) { /* Fatal error. */ - goto error_put_client_list; + goto error_free_ht_element; } DBG("Newly registered trigger's condition evaluated to %s", @@ -2805,7 +2840,7 @@ int handle_notification_thread_command_register_trigger( if (!evaluation) { /* Evaluation yielded nothing. Normal exit. */ ret = 0; - goto end; + goto success; } /* @@ -2826,7 +2861,7 @@ int handle_notification_thread_command_register_trigger( */ ERR("Fatal error occurred while enqueuing action associated to newly registered trigger"); ret = -1; - goto error_put_client_list; + goto error_free_ht_element; case ACTION_EXECUTOR_STATUS_OVERFLOW: /* * TODO Add trigger identification (name/id) when @@ -2836,18 +2871,16 @@ int handle_notification_thread_command_register_trigger( */ WARN("No space left when enqueuing action associated to newly registered trigger"); ret = 0; - goto end; + goto success; default: abort(); } -end: +success: *cmd_result = LTTNG_OK; DBG("Registered trigger: name = `%s`, tracer token = %" PRIu64, trigger_name, trigger_tracer_token); - -error_put_client_list: - notification_client_list_put(client_list); + goto end; error_free_ht_element: if (trigger_ht_element) { @@ -2859,6 +2892,7 @@ error: if (free_trigger) { lttng_trigger_destroy(trigger); } +end: rcu_read_unlock(); return ret; } @@ -2959,6 +2993,9 @@ int handle_notification_thread_command_unregister_trigger( teardown_tracer_notifier(state, trigger); } + trigger_ht_element = caa_container_of(triggers_ht_node, + struct lttng_trigger_ht_element, node); + if (is_trigger_action_notify(trigger)) { /* * Remove and release the client list from @@ -2967,15 +3004,16 @@ int handle_notification_thread_command_unregister_trigger( client_list = get_client_list_from_condition(state, condition); assert(client_list); + pthread_mutex_lock(&client_list->lock); + cds_list_del(&trigger_ht_element->client_list_trigger_node); + pthread_mutex_unlock(&client_list->lock); + /* Put new reference and the hashtable's reference. */ notification_client_list_put(client_list); notification_client_list_put(client_list); client_list = NULL; } - trigger_ht_element = caa_container_of(triggers_ht_node, - struct lttng_trigger_ht_element, node); - /* Remove trigger from triggers_ht. */ notif_thread_state_remove_trigger_ht_elem(state, trigger_ht_element); @@ -4266,7 +4304,7 @@ int notification_client_list_send_evaluation( pthread_mutex_lock(&client_list->lock); cds_list_for_each_entry_safe(client_list_element, tmp, - &client_list->list, node) { + &client_list->clients_list, node) { enum client_transmission_status transmission_status; struct notification_client *client = client_list_element->client; @@ -4548,7 +4586,7 @@ int dispatch_one_event_notifier_notification(struct notification_thread_state *s /* Warn clients that a notification (or more) was dropped. */ pthread_mutex_lock(&client_list->lock); cds_list_for_each_entry_safe(client_list_element, tmp, - &client_list->list, node) { + &client_list->clients_list, node) { enum client_transmission_status transmission_status; struct notification_client *client = client_list_element->client; diff --git a/src/bin/lttng-sessiond/notification-thread-internal.h b/src/bin/lttng-sessiond/notification-thread-internal.h index e6467dc27..38d968a4d 100644 --- a/src/bin/lttng-sessiond/notification-thread-internal.h +++ b/src/bin/lttng-sessiond/notification-thread-internal.h @@ -113,8 +113,9 @@ struct notification_client_list_element { struct notification_client_list { pthread_mutex_t lock; struct urcu_ref ref; - const struct lttng_trigger *trigger; - struct cds_list_head list; + struct lttng_condition *condition; + struct cds_list_head triggers_list; + struct cds_list_head clients_list; /* Weak reference to container. */ struct cds_lfht *notification_trigger_clients_ht; struct cds_lfht_node notification_trigger_clients_ht_node; diff --git a/tests/regression/tools/notification/base_client.c b/tests/regression/tools/notification/base_client.c index 70ad763ab..9ba1340cb 100644 --- a/tests/regression/tools/notification/base_client.c +++ b/tests/regression/tools/notification/base_client.c @@ -17,6 +17,7 @@ #include #include +#include #include #include #include @@ -35,6 +36,7 @@ static const char *channel_name = NULL; static double threshold_ratio = 0.0; static uint64_t threshold_bytes = 0; static bool is_threshold_ratio = false; +static bool use_action_group = false; static enum lttng_condition_type buffer_usage_type = LTTNG_CONDITION_TYPE_UNKNOWN; static enum lttng_domain_type domain_type = LTTNG_DOMAIN_NONE; @@ -50,6 +52,7 @@ int parse_arguments(char **argv) const char *buffer_usage_threshold_type = NULL; const char *buffer_usage_threshold_value = NULL; const char *nr_expected_notifications_string = NULL; + const char *use_action_group_value = NULL; session_name = argv[1]; channel_name = argv[2]; @@ -58,6 +61,7 @@ int parse_arguments(char **argv) buffer_usage_threshold_type = argv[5]; buffer_usage_threshold_value = argv[6]; nr_expected_notifications_string = argv[7]; + use_action_group_value = argv[8]; /* Parse arguments */ /* Domain type */ @@ -98,6 +102,11 @@ int parse_arguments(char **argv) /* Number of notification to expect */ sscanf(nr_expected_notifications_string, "%d", &nr_expected_notifications); + /* Put notify action in a group. */ + if (!strcasecmp("1", use_action_group_value)) { + use_action_group = true; + } + return 0; error: return 1; @@ -107,6 +116,7 @@ int main(int argc, char **argv) { int ret = 0; enum lttng_condition_status condition_status; + enum lttng_action_status action_status; enum lttng_notification_channel_status nc_status; struct lttng_notification_channel *notification_channel = NULL; struct lttng_condition *condition = NULL; @@ -120,7 +130,7 @@ int main(int argc, char **argv) */ setbuf(stdout, NULL); - if (argc < 8) { + if (argc < 9) { printf("error: Missing arguments for tests\n"); ret = 1; goto end; @@ -196,11 +206,41 @@ int main(int argc, char **argv) goto end; } - action = lttng_action_notify_create(); - if (!action) { - printf("error: Could not create action notify\n"); - ret = 1; - goto end; + if (use_action_group) { + struct lttng_action *notify, *group; + + group = lttng_action_group_create(); + if (!group) { + printf("error: Could not create action group\n"); + ret = 1; + goto end; + } + + notify = lttng_action_notify_create(); + if (!notify) { + lttng_action_destroy(group); + printf("error: Could not create action notify\n"); + ret = 1; + goto end; + } + + action_status = lttng_action_group_add_action(group, notify); + if (action_status != LTTNG_ACTION_STATUS_OK) { + printf("error: Could not add action notify to action group\n"); + lttng_action_destroy(group); + lttng_action_destroy(notify); + ret = 1; + goto end; + } + + action = group; + } else { + action = lttng_action_notify_create(); + if (!action) { + printf("error: Could not create action notify\n"); + ret = 1; + goto end; + } } trigger = lttng_trigger_create(condition, action); @@ -265,7 +305,7 @@ int main(int argc, char **argv) goto end; default: /* Unhandled conditions / errors. */ - printf("error: Unknown notification channel status\n"); + printf("error: Unknown notification channel status (%d) \n", status); ret = 1; goto end; } diff --git a/tests/regression/tools/notification/test_notification_multi_app b/tests/regression/tools/notification/test_notification_multi_app index afac94d10..5d5427c9d 100755 --- a/tests/regression/tools/notification/test_notification_multi_app +++ b/tests/regression/tools/notification/test_notification_multi_app @@ -54,8 +54,9 @@ function start_client { local buffer_usage_threshold_type=$6 local buffer_usage_threshold_value=$7 local nr_expected_notification=$8 + local use_action_group=$9 - ${CURDIR}/base_client ${session_name} ${channel_name} ${domain_type} ${buffer_usage_type} ${buffer_usage_threshold_type} ${buffer_usage_threshold_value} ${nr_expected_notification} > ${output_file} & + ${CURDIR}/base_client ${session_name} ${channel_name} ${domain_type} ${buffer_usage_type} ${buffer_usage_threshold_type} ${buffer_usage_threshold_value} ${nr_expected_notification} ${use_action_group} > ${output_file} & pid=$! app_pids+=("$pid") @@ -177,8 +178,8 @@ function test_multi_app () for (( i = 0; i < $nr_client_app; i++ )); do low_app_output_file=$output_dir/${low_output_file_pattern}${i} high_app_output_file=$output_dir/${high_output_file_pattern}${i} - start_client $low_app_output_file $SESSION_NAME $CHANNEL_NAME $domain_string LOW RATIO 0.0 $nr_notification_expected - start_client $high_app_output_file $SESSION_NAME $CHANNEL_NAME $domain_string HIGH RATIO 0.420 $nr_notification_expected + start_client $low_app_output_file $SESSION_NAME $CHANNEL_NAME $domain_string LOW RATIO 0.0 $nr_notification_expected $(( $i % 2)) + start_client $high_app_output_file $SESSION_NAME $CHANNEL_NAME $domain_string HIGH RATIO 0.420 $nr_notification_expected $(( $i % 2)) done wait_for_message $output_dir "${low_output_file_pattern}" "sync: ready" @@ -362,7 +363,7 @@ function test_on_register_evaluation () high_app_output_file=${high_output_file_pattern}.first_receiver high_app_output_path=$output_dir/${high_app_output_file} - start_client $high_app_output_path $SESSION_NAME $CHANNEL_NAME $domain_string HIGH RATIO 0.420 1 + start_client $high_app_output_path $SESSION_NAME $CHANNEL_NAME $domain_string HIGH RATIO 0.420 1 0 wait_for_message $output_dir "${high_app_output_file}" "sync: ready" @@ -379,7 +380,7 @@ function test_on_register_evaluation () # notification on subscription high_app_output_file=${high_output_file_pattern}.second_receiver high_app_output_path=$output_dir/${high_app_output_file} - start_client $high_app_output_path $SESSION_NAME $CHANNEL_NAME $domain_string HIGH RATIO 0.420 1 + start_client $high_app_output_path $SESSION_NAME $CHANNEL_NAME $domain_string HIGH RATIO 0.420 1 0 wait_for_message $output_dir "${high_app_output_file}" "sync: ready" wait_for_message $output_dir "${high_app_output_file}" "notification: high 0"