Logging
Log Files
Log Files on the Host OS
On the host OS, i.e. ONL in case of hardware switches, navigate to the /var/log/ folder and list the available log files:
supervisor@spine1:/var/log$ ls -l total 3064 <...> -rw-r--r-- 1 root root 2242 Nov 4 11:13 rtbrick-apigwd.log -rw-r--r-- 1 root root 458617 Nov 4 14:27 rtbrick-ctrld.log -rw-r--r-- 1 root root 831 Nov 4 11:13 rtbrick-hostconfd.log -rw-r--r-- 1 root root 73509 Nov 4 14:27 rtbrick-hostnetconfd.log -rw-r--r-- 1 root root 668 Nov 4 11:13 rtbrick-lxcd.log -rw-r----- 1 root adm 286356 Nov 4 14:27 syslog <...>
Inspect the log file of interest using one of the standard Linux tools like cat, more, or less. For example:
supervisor@spine1:/var/log$ more rtbrick-ctrld.log
Log Files in the LXC
In the RBFS container, navigate to the /var/log/ folder and list the available log files. There will be one active log file per daemon or service, and up to ten compressed (.gz) log files created during log file rotation:
supervisor@spine1:/var/log$ ls -l total 2376 <...> -rw-r--r-- 1 root root 48268 Nov 4 15:51 rtbrick-alertmanager-service-out.log -rw-r--r-- 1 root root 3451 Oct 30 17:51 rtbrick-alertmanager-service-out.log.1.gz -rw-r--r-- 1 root root 2509 Oct 25 00:02 rtbrick-alertmanager-service-out.log.2.gz -rw-r--r-- 1 root root 0 Nov 1 01:17 rtbrick-bgp.appd.1-service-out.log -rw-r--r-- 1 root root 3333 Oct 30 17:51 rtbrick-bgp.appd.1-service-out.log.1.gz -rw-r--r-- 1 root root 3192 Oct 19 13:17 rtbrick-bgp.appd.1-service-out.log.2.gz <...>
Please note log files for brick daemons include only basic logs referring to the operation of the daemon itself. The actual applications like routing or service protocols use log tables described in section 5.2. In contrast, Prometheus and Alertmanager are no BDs and use log files as the primary means of logging. Inspect the log file of interest using one of the standard Linux tools like cat, more, or less. For example:
supervisor@spine1:/var/log$ more rtbrick-prometheus-service-out.log
BDS Logging
Enabling BDS Logging
BDS logging is enabled per BD, per module, and per group by configuration. By default, BDS logging is enabled for all daemons and all modules with level Error. Verify the log status of the module and optionally per daemon. The following output reflects a default configuration:
supervisor@spine2: op> show log status module bgp bd bgp.iod.1
Module log status:
bgp:
bgp.iod.1:
Level: error, Plugin: None
Log group status:
Group Level Plugin Rate limit
config error None 10
general error None 10
generic error None 10
<...>
For troubleshooting an application, enable a more detailed log level. First example, enable and verify logging with level Info for the bgp.iod and all groups of the BGP module:
supervisor@spine2: cfg> set log bd bgp.iod.1 module bgp level info
supervisor@spine2: cfg> commit
supervisor@spine2: op> show log bgp.iod.1 status
<...>
supervisor@spine2: op> show log status module bgp bd bgp.iod.1
Module log status:
bgp:
bgp.iod.1:
Level: info, Plugin: None
Log group status:
Group Level Plugin Rate limit
config info None 10
general info None 10
generic info None 10
<...>
Second example, enable and verify logging with level Debug specifically for the PPP and PPPoE group of logs:
supervisor@leaf2: cfg> set log bd pppoed.1 module pppoed group ppp level debug
supervisor@leaf2: cfg> set log bd pppoed.1 module pppoed group pppoe level debug
supervisor@leaf2: cfg> commit
supervisor@leaf2: cfg>
supervisor@leaf2: op> show log status module pppoe
Module log status:
pppoe:
pppoed.1:
Level: error, Plugin: None
Log group status:
Group Level Plugin Rate limit
config error None 10
dhcpv6 error None 10
general error None 10
generic error None 10
ppp debug None 10
pppoe debug None 10
subscriber error None 10
For a complete and detailed description of the logging configuration, please refer to the RBFS Logging Guide.
Inspecting Log Tables
BDS log tables are created on demand by the daemons if logging is enabled and there is an event to log. Use the 'show log table' command to look into the log tables:
supervisor@leaf2: op> show log table bgp.logs [ Info ] <2021-07-16T06:57:14.434358+0000> BGP peer fe80::7825:1dff:fe60:202, source fe80::7828:3bff:fe60:101, hostname spine1, instance default reset, reason Notification received [ Info ] <2021-07-16T06:57:14.435583+0000> BGP FSM change, peer fe80::7825:1dff:fe60:202, source fe80::7828:3bff:fe60:101, hostname -, instance default changed state from Established to Idle, reason Cease, Sub-Code: Admin reset [ Info ] <2021-07-16T06:58:01.033211+0000> Created table(default.bgp.rib-in.ipv6.unicast.fe80::7825:1dff:fe60:202.fe80::7828:3bff:fe60:101) [ Info ] <2021-07-16T06:58:01.033290+0000> Created table(default.bgp.rib-in.ipv6.labeled-unicast.fe80::7825:1dff:fe60:202.fe80::7828:3bff:fe60:101) [ Info ] <2021-07-16T06:58:01.033347+0000> Created table(default.bgp.rib-in.ipv4.vpn-unicast.fe80::7825:1dff:fe60:202.fe80::7828:3bff:fe60:101) [ Info ] <2021-07-16T06:58:01.033401+0000> Created table(default.bgp.rib-in.ipv6.vpn-unicast.fe80::7825:1dff:fe60:202.fe80::7828:3bff:fe60:101) [ Info ] <2021-07-16T06:58:01.344927+0000> BGP FSM change, peer fe80::7825:1dff:fe60:202, source fe80::7828:3bff:fe60:101, hostname spine1, instance default changed state from Idle to Established, reason None [ Info ] <2021-07-16T06:58:03.471937+0000> BGP message received from peer fe80::7825:1dff:fe60:202, source fe80::7828:3bff:fe60:101, hostname spine1, instance default decode failed. Message type 2 <...>
The 'show log table' command supports various filter options. You can filter on the level, module, or a regular expression. Example:
supervisor@spine2: op> show log table bgp.logs filter level error supervisor@spine2: op>
For a complete and detailed description of the filter options, please refer to the RBFS Logging Guide.