The time-series database (TSDB)

Object metrics are defined on values of one or more attributes present on rows on the RTB datastore tables; we will use this type of metric in our first example.

Object metrics values can be of two separate datatypes:

The first step in our configuration is to define the value we intend to monitor: this means identifying the table in the BDS datastore which contains the values we are interested in, the rows of the table which contains the value and the actual attribute to collect.

Suppose for example that we are interested in tracking the number of IS-IS neighbors in the default instance which are in UP state; first we need to identify where this information is stored in order to collect it.

From the output, we can see that a good candidate is the level1_neighbor_count attribute, if we select the row of the datastore table which corresponds to the default instance by using the condition instance_name=default

So to define this metric, we will need to configure:

Then, within the table, we need to specify the row we are interested in (the one with instance_name=default), and the attribute (the column in the table) which contains the value we need to monitor.

To do this, we will need two extra configuration statements:

The actual metrics configuration should look similar to this:

Now that our gauge metric is defined, we need to define a metric profile that defines the collection interval, and associate it with our metric.

We can define a profile named every_10_secs which collects data every 10 seconds with this configuration:

Then we can associate this profile with our metric, commit and go back to operational mode:

From this point on, the metric will be collected and stored in the time-series database and will be available both to internal alerting subsystem, and for collection and aggregation by a centralized Prometheus instance.

Unlike object metrics, index metrics are not meant to capture the values of attributes the rows of the RtBrick Datastore tables; specifically, they count the number of object in an index defined on a table.

Index metrics are less intuitive than object metrics, as their value depends on how tables are indexed in order to efficiently retrieve single rows.

To address this complexity, we will show examples one of the most common and useful use case for index metrics: tables where an index metric is equivalent to the number of object in the table (the row count).

Let’s consider for example the table that stores the set of active IPv4 routes in the default instance, that is, the routes actually installed and used to forward IPv4 packets: the table default.ribd.fib-local.ipv4.unicast

It is interesting from operations point of view to monitor the size of this table in the table, to prevent resource issues (that is, to make sure the number of active routes is well within the system hardware capabilities).

First, let’s check which indexes are defined on the default.ribd.fib-local.ipv4.unicast table.

The primary index (of type radix) is used to do longest-prefix lookup on the IPv4 routing table, and is a good candidate for our metric: the active-entry-count of this index will match the number of active IPv4 unicast routes.

Now we have all information we need, and we can define our index metric on the default.ribd.fib-local.ipv4.unicast table.

If we name our metric ipv4_unicast_routes, the definition (under the time-series hierarchy) should look similar to this:

Once metrics are defined and are being collected, it is possible to define alerts triggered by conditions on their values.

An alert configuration contains, among other things:

As a simple example, we can define an alert named isis_below_three, that is fired if the number of is-is neighbors drops below three over any 10-second interval. We can assign a severity level of 2 (critical) to this event, and attach to it a label repeat_interval with a value of 10m, which we will use later on when we define the alert-profile.

The alert configuration should look similar to this:

To simulate a failure and trigger the alarm, we can disable one of the three IS-IS neighbors in the configuration.

After committing, let’s verify the state of alarms by running show time-series alert. Remember the condition is triggered after 10 seconds; after this, you should see the alert transitioning to pending and then firing

You can get the details of any of the active alarms by specifying their name; this will also tell you when the alarm was first fired:

Alert profiles allows to group several alerts and specify how often notifications should be re-sent if the alert condition persists.

As an example, you could define a profile called hadware-alert that groups several individual alerts that indicate potential hardware issues (for example temperature, fans or power supply issues); once the network operation team has been advised of this, they can then check and troubleshoot the specific problem.

Similarly, you could consider grouping alerts related to individual switch resources (e.g. excessive CPU usage, high memory utilization or low disk space) into a resource-alert group, to warn about issues that could potentially lead to resource exhaustion. Alerts can then be relayed via the CtrlD process to external Network Faut Management Systems, to notify the network operation team of the potential issue before it has any actual impact.

Alert profiles are configured under the time-series configuration hierarchy, and are associate to alerts via the labels attached to each alert.

The real usefulness of alert-profiles is the capability to group alerts to simplify their management; but, just as an example, we can configure a simple alert profile associated to the isis_below_three alert we defined above, and request that the alert is re-sent every 10 minutes if active:

The matchers statement is what ties the alert profile to the alert, via the repeat_interval alert label.