ZTP in a Nutshell

This section describes the ZTP process in a Nutshell, figure 1 illustrates the process at a high level.

The process is split into two main parts:

  • ONIE image discovery and Installation.

    • ONIE uses DHCP to discover the IP address along with the image download URL based on the provided DHCP options. For download, ONIE allows different ways to pull an image from the repository. In this ZTP process, HTTP is used to pull the image because ONIE conveys the serial number as the HTTP header. This serial number allows the image repository to identify the switch and select the appropriate image. A long with the serial number onie also send the `onie-operation that allows to distinguish between an os-install and onie-update, and select the correct image for either NOS install oder FW upgrade.

    • See the ONIE image discovery for further information (/ONIE/)

  • CTRLD configuration discovery and application.

    • CTRLD sends DHCPINFORM to ask for all options needed for config discovery.

    • The configurations are downloaded from the management server (HTTPD) and applied.

The ZTP Process
Figure 1. The ZTP Process

Figure 2. depicts the relationship between the fabric, the DHCP server and the Management Server.

The Management Server Architecture
Figure 2. The Management Server Architecture

DHCPD

Because of its low set of requirements, the default DHCP server shipped with ubuntu, isc-dhcp, is used to run the DHCP service.

The following code shows an example configuration of a DHCP server and hardware box (dhcp.conf).

dhcp.conf
authoritative;
default-lease-time 600;
max-lease-time 72----

# This is only needed if the version is lower than 4.4
option loader-pathprefix code 210 = text;


subnet 10.0.0.0 netmask 255.255.255.0 {
  range 10.0.0.200 10.0.0.250;
  option routers 10.0.0.138;
  option domain-name-servers 10.0.0.210;
  option domain-name "local";
  host LEAF01 {
    # Identify client by MAC address.
    hardware ethernet 48:65:ee:11:da:85;
    # Identify client by serial number
    option dhcp-client-identifier "\000WLC1C27L00003P2";
    fixed-address 10.0.0.250;
    option host-name LEAF01;
    # Set DHCP option 114 (default-url) to set the installer image URL.
    # ONIE loads the installer image from the specified URL.
    option default-url "http://managementserver/ztp/image";
    # Set DHCP option 210 (path prefix) to set the configuration base URL.
    # CTRLD loads all configuration files from this base URL.
    option loader-pathprefix "http://managementserver";
  }
}

Most of the used options are already predefined in the ISC-DHCP server please find a Reference under /ISCKB/ , the loader-pathprefix is defined since DHCP 4.4, so if you use an older one please define it as described above.

HTTPD (Management Server)

The HTTP service is responsible to provide the NOS installer, and the configuration files. Therefore, a self implemented golang HTTP Server is used, which reads the ONIE_SERIAL_NUMBER and ONIE-OPERATION HTTP header and maps them to the NOS/FW installer image download path, and maps the serial number to the ZTP configuration files (explanation for config files see the chapter below).

The ONIE-OPERATION header can have the following values:

  • install nos: os-install

  • update firmware: onie-update

The sections below describe the installation and configuration of the server.

ZTP installation

For the installation you can choose between the following two options:

ZTP installation with debian package

  • make sure that you have added the rtrbick repository to your apt.sources list and updated the cache

  • make sure that port 80 is not yet used on your machine

  • install the package rtbrick-fabric-ztp

  • the package will install a systemd service named rtbrick-fabric-ztp

  • make sure that the service is running with sudo systemctl status rtbrick-fabric-ztp

  • the default location used for the ZTP config files is /var/rtbrick/ztp/configs/ where you need to copy your config files

If needed, overriding server settings can be done like this:

  • edit the service configuration file /etc/systemd/system/rtbrick-fabric-ztp.service and add parameters to the ExecStart commmand

  • parameter --addr: the listen address of the server, default 0.0.0.0:80

  • parameter --requestTimeout: the request timeout server in seconds, default 600, must possibly be increased dependent on connection speed and image file sizes

  • parameter --filePath: the location for the ZTP config files, default /var/rtbrick/ztp/configs/

ZTP installation as docker container

  • make sure that you have access to the rtbrick docker registry

  • make sure that port 80 is not yet used on your machine

  • create a compose file docker-compose.yml with the content

version: '3.3'
services:
  ztp:
    image: 'docker.rtbrick.com/rbms-fabric-ztp:latest'
    container_name: rbms-fabric-ztp
    restart: unless-stopped
    ports:
      - '80:80'
    volumes:
      - './configs:/var/rtbrick/ztp/configs'

  • the compose setup uses a bind mound for the ZTP config folder and therefore the docker-compose.yml must be in the same location together with the ./configs folder for the ZTP config files (explanation for config files see the chapter below)

  • if needed, adapt the compose file for a different image version, port binding or different configs folder location

  • start the container with docker-compose up -d

ZTP configuration

The HTTP service matches the ONIE-SERIAL-NUMBER header to the configuration files. Therefore, for each supported serial number a <serial_number>.json file must exist in the configs folder. This files contains settings for locations of all additional config files which have to be delivered for the specific device, as well as settings for the NOS installer image or the FW update image.

Example sample.json file (for a serial number "sample"):

/var/rtbrick/ztp/configs/sample.json
{
  "description": "192.168.202.116",
  "ctrld": "ctrld.json",
  "ctrldrbac": "ctrldrbac.json",
  "startup": "sample_startup.json",
  "element": "sample_element.json",
  "accessjwks": "sample_accessjwks.json",
  "apigwd": "sample_apigwd.json",
  "tls": "sample_tls.pem",
  "image": "http://pkg.rtbrick.net/_/images/latest/rtbrick-onl-installer/rtbrick-onl-installer-accessleaf-qmx-20.4.0-g8daily.20200415051734+Bmaster.C059a09ea",
  "update_image": "http://pgk.rtbrick.net/firmwares/onie-firmware-x86_64-ufispace_s9600_32x_ufispace_s9600_64x-r0_v0.3.0.updater"
}

Image location configuration

For the config entries "image" and "update_image" you have three possibilities:

  • redirect URL: config value must start with http, the server will redirect the request to download the image from an URL, example "http://pkg.rtbrick.net/_/images/latest/rtbrick-onl-installer/rtbrick-onl-installer-accessleaf-qmx-20.4.0-g8daily.20200415051734+Bmaster.C059a09ea"

  • absolute file location: config value must start with /, can point to any file on the local disk, example "/usr/share/images/rtbrick-onl-installer.img"

  • relative file location: config value must be a filename and not start with /, points to any file in the <ztppath>/configs/images/ folder, example "rtbrick-onl-installer.img"

ZTP API usage

For usage of the HTTP API please refer to the openapi documentation

CTRLD

CTRLD acts as a post-ZTP demon, it runs after the image is activated. This demon is responsible for configuring the switch properly.

There are various configurations which CTRLD can load from a management server and apply it to the system.

  • CTRLD config: the base configuration for CTRLD. There the RBMS and Graylog can be specified, but also the authentication and authorization mechanism can be controlled.

  • Element config: Each LXC container can be configured for CTRLD. So the element config is the link between the element name and the container. By default, the element name is equal to the container name, but to configure differently, this can be specified also in the element configuration. Also the ZTP process for the element can be disabled via this configuration.

  • CTRLD rbac policy: the Role Based Access Control (RBAC) policy of CTRLD is specified in this configuration file.

  • Startup Config: The RBFS switch configuration.

  • TLS pem file: For APIGWD: This file is an X509 public/private key file in PEM format specified in the RFC7468.

  • Access JWKS file: For APIGWD: JSON Web Key Set (JKWS) is described in the RFC 7517.

Trigger the ZTP process

The ZTP process in CTRLD is triggered for a specific container (LXC) on the switch. This can be triggered in the following ways.

  • By the switch (RBFS lxc container) itself by sending the operational state up to CTRLD.

  • By sending a REST request to trigger the ZTP process to CTRLD (/api/v1/ctrld/ztp/_run).

If the ZTP Process for the specified container is enabled (element.config), the process starts.

Trigger the reinstall

The reinstall of a switch can be triggered by sending a POST request to CTRLD (/api/v1/ctrld/system/_reinstall)

Trigger the firmware update

The firmware update of a switch can be triggered by sending a POST request to CTRLD (/api/v1/ctrld/system/_update)

Management Server URL Discovery

CTRLD has to discover the management server URL in order to download the configuration from the management server. Therefore, a management interface is defined which allows to send an DHCPINFORM request to the DHCP server.

The request contains the DHCP option 61 that conveys the client identifier. The client identifier is either omitted or contains the serial number. The serial number is gathered from the onie file system information file /lib/platform-config/current/onl/onie-info.json. If that does not result in a valuable result the following command is executed dmidecode -s system-serial-number (see /RFC2131/ and /RFC2132/ for further information).

There are at least two DHCP options requested, DHCP option 54 that conveys the IP address of the DHCP server (see /RFC2132/ for further information), and DHCP option 210 that conveys the path prefix for all configuration files (see /RFC5071/ for further information).

If the DHCP option 210 is not returned, CTRLD attempts to read the configurations from the IP address of the ZTP server. Otherwise, CTRLD attempts to read the configurations from the base URL specified in DHCP option 210.

Request the configurations

The request to the Management server contains the following HTTP headers:

  • ONIE-SERIAL-NUMBER: This serial number is either the onie serial number or empty string.

  • CONTAINER-NAME: Container that triggered the ZTP process.

  • ELEMENT-NAME: Element name that triggered the ZTP process.

  • HOST-NAME: Host name of the device that triggered the ZTP process.

All this information can be used to select the right configurations for the container. This also allows the use of ZTP Configuration Process for virtual environments.

The requested URL:

  • CTRLD Config: <management server url>/ztp/config/ctrld

  • Element Config: <management server url>/ztp/config/element

  • CTRLD rbac policy: <management server url>/ztp/config/ctrldrbac

  • Startup Config: <management server url>/ztp/config/startup

  • TLS pem file: <management server url>/ztp/config/tls

  • Access JWKS file: <management server url>/ztp/config/accessjwks

If one of the files is not found the process still goes forward.

Business Events

During the ZTP Process log messages are sent to the configured ztp graylog endpoint.

Please consolidate the switch api documentation for more details.

Overall Process Flow

The 2 figures below show the CTRLD ZTP process flow.

.CTRLD ZTP process flow (Part 1/2)
Figure 3. CTRLD ZTP process flow (Part 1/2)
.CTRLD ZTP process flow (Part 2/2)
Figure 4. CTRLD ZTP process flow (Part 2/2)