RBFS Logging

This document provides you information about logging in the entire RBFS ecosystem. It includes the RBFS container and the host OS which is ONL in case of hardware switches.

RBFS logging involves recording log messages during the execution of events. It generates reports about activities within the entire RBFS ecosystem across various functional areas. Almost every component in RBFS ecosystem can generate log files. Logging enables you to trace application errors in real time.

Understanding RBFS Logging

RBFS’s logging system captures events from various components, including Brick Daemons (BDs), non-BD daemons like Prometheus, and host-level services such as CtrlD and ApiGwD. These logs provide insights into the entire RBFS ecosystem, facilitating real-time monitoring and analysis.

RBFS Logging allows you:

  • Enable Logging

  • View Logs (at var log file location and by using operational commands)

  • Export Logs to an External Server

An RBFS ecosystem contains multiple microservices and these microservices are known as Brick Daemons (BD) and other (non-BD) daemons. CtrlD and ApiGwD daemons are part of ONL, but they do not reside inside of RBFS container. BDS provides in-built infrastructure for logging which can be used by all BDS applications.

You can view all the logs generated and stored in the var log files and using operational commands. All logs can be exported to an external management server where you can view and analyze the real-time data.

Log Level

Log level indicates level of the severity of the events (logs). You can configure logging based on different severity levels. RBFS supports the following log levels, listed in order of priority:

  • Emergency

  • Alert

  • Critical

  • Error

  • Warning

  • Notice

  • Info

  • Debug

  • None

Any level above the level Warning indicates that you should perform logging with caution as a scaled environment may cause a system instability.

About RBFS Logging System

RBFS provides logging for the entire RBFS ecosystem that includes Brick Daemons (BD) and also for other (non-BD) daemons. Brick Daemons are built on top of BDS and other (non-BD) daemons (such as Prometheus) are the ones which are not dependent on BDS.

The RBFS container logging infrastructure provides in-memory (BDS) and traditional (BD) logging support for RBFS applications. The BDS logging is a low-latency and in-memory logging which can be used in a high scale system without compromising much in performance whereas BD logging is a direct write to a file hence CPU-heavy.

RBFS logging

In RBFS, logs are generated from various components or sources such as BDS, Syslog, CtrlD, APIGwD, and Prometheus. All these logs can be finally exported to a log management server.

BDS Logging

BDS logs are stored in a log table. For every unique event, a log ID is created in RBFS. Whenever that particular event is logged, a log entry gets added to the log table. A log table is created for a module only when that module has at least one event logged. Every module in RBFS has at least one log table named in this format: <modulename>.<bd-name>.log.

BDS logging

Log Tables

BDS logs are stored in a BDS table. BDS creates a log table for each module in a BD. One entry is added to this log table for every log. Older entries are removed from the table when the number of entries exceeds more than 10,000.

Log Maps

Every log is mapped to one specific event that is logged by the application. For the optimized usage of memory, RBFS does not store the verbose strings; instead, it stores the log map as an identifier to the actual string message.

Log map and log ID refer to the same entity.

You can access these log maps at the following location:

/usr/share/rtbrick/liblog/logs/

You can see the log maps, organized based on the modules that they belong to, at:

supervisor@rtbrick:/usr/share/rtbrick/liblog/logs$ ls
bds  bgp  fib  fwdinfra  ifm  lldpv2  pd  policy  pubsub  resmon  rib  snapshot  static  time_series

In the preceding example, you can see the modules that have registered with the log maps.

If you want to know more about a particular log map, you can perform a grep of the log map in this directory.

Log Groups

A log group is a collection of log maps or log IDs. Groups have been introduced to simplify the log configuration tasks. For example, to debug a BGP peer issue, instead of enabling logs for individual log IDs that are related to BGP peer, you can enable log for a log group BGP peer.

Log Modules

Every BDS application consists of multiple modules. Logging can be configured for each BDS modules separately.

The following are RBFS modules:

bcm_q2c
bgp
fib
hostconfd
ifm
igmp
ipoe
isis
l2tp
lag
ldp
license
lldp
mrib
ospf
pim
policy
poold
pppoe
resmon
rib
rtbrick-cli
secure_management
snapshot
subsMgmt
static
time_series
vpp

Log Export to an External Server (Plugin Alias)

Any logs in RBFS can be exported to an external logging destination. Currently, CtrlD supports GELF and syslog as external plugins.

CtrlD is the egress node for all the GELF (Graylog Extended Log Format) messages. The brick daemons which are configured to send GELF messages to CtrlD and CtrlD forwards them to the configured endpoints such as syslog or GELF endpoint. This is because CtrlD enhances the GELF message with switch-global settings (for example, the serial number of the switch).

graylog 
supervisor@rtbrick: cfg> set system host log-alias default endpoint http:10.1.1.1:2002/gelf
  <cr>
  level                 Log level for the alias
  network               Log alias endpoint port
  queue-size            Queue size that will be used for the fanout [Range: <0-500>]
  type                  Log alias server type
supervisor@rtbrick: cfg> set system host log-alias default endpoint http:10.1.1.1:2002/gelf network
  http                  Log alias endpoint port
  tcp                   Log alias endpoint port
  udp                   Log alias endpoint port

Supported Logging Format

RBFS supports two types of logging format: GELF and Syslog.

Export Format

GELF:

GELF is a logging format for Graylog. It uses JSON to structure log messages in a way that makes them searchable and more structured.

The supported protocol for Graylog is HTTP.

Example:

supervisor@rtbrick: cfg> set system host log-alias graylog endpoint http://10.1.1.1:2002/gelf type
 gelf                 syslog

Syslog:

Syslog is a format used by Linux systems and many networking devices for logging. It is a plain text protocol with a defined structure.

The supported transportation protocols for Syslog format are TCP and UDP.

You can configure log-alias to send logs to a centralized logging endpoint, like Graylog, using the GELF.

Example:

supervisor@rtbrick: cfg> set system host log-alias graylog endpoint http://10.1.1.1:2002/gelf network
http
tcp
udp

Guidelines and Limitations for BDS Logging

  • By default, BDS logging is enabled and the log level is set to 'Error'.

  • By default, logging for BDS and PUBSUB modules have been disabled. As these two modules are infrastructure specific, these logs may not be useful for end-users. However, developers can enable logging for these modules using debug commands.

  • You can configure log levels per BD, per module, or per group.

  • Do not keep logging enabled for longer duration in a scaled setup.

  • The following log levels are present in the system. Any level above the level Warning indicates that you should perform logging with caution as a scaled environment may cause a system instability.

If your system becomes unstable, you can remove the logging configuration using the delete log command in configuration mode.

  • All log levels lower than the log level specified are logged. For example, if the specified log level is "Warning", then all logs that come before "Warning" (Emergency, Alert, Critical, Error, Warning) are logged.

  • When you set the log-level to "None", that means log has been disables for the specific module, group, or global.

Syslog

Syslog is generated by an API based logging mechanism provided by Linux. Some of the open source libraries present in RBFS use Syslog as a logging mechanism.

Host operating system and RBFS Linux container use syslog for logging. Syslog can also be exported to Graylog server.

Syslog messages can also be exported to CtrlD and these messages are forwarded to the defined log management servers. Currently, RBFS supports exporting syslog from the Linux and ONL system facilities such as auth, authpriv, daemon, and kern to Graylog

For information about Syslog configuration, see section System Logging Configuration.

TSDB (Prometheus) Alert Messages

Prometheus is the systems and service monitoring application, which can be deployed in RBFS, to collect and process metrics. In RBFS, alert messages generated from Prometheus are forwarded to CtrlD and these messages, from CtrlD, can be exported to the configured log management servers.

Logging in ONL

In RBFS, there are daemons, which are not part of RBFS container, but run on the ONL host. RBFS provides logging for these daemons. The following are the daemons which reside on the ONL host:

hostconfd
CtrlD
ApiGwD

CtrlD and ApiGwD Logs

CtrlD logging provide log messages of events related to business, elements, ZTP, and security. ApiGwD logs contain details about who accessed the API and how they accessed it. hostconfd provides rest APIs to interact with container and ONL linux services.

ApiGwD`and `CtrlD send different log messages about status changes or progress of processes to the configured GELF endpoint.

Supported Platforms

Not all features are necessarily supported on each hardware platform. Refer to the Platform Guide for the features and the sub-features that are or are not supported by each platform.