Zero Touch Provisioning

Overview

Zero Touch Provisioning (ZTP) automates the tasks of installing software images. It is a method for setting up and configuring devices automatically. ZTP installs or upgrades the RBFS software image on your hardware platforms without any manual intervention.

ZTP automatically provisions routers newly installed in the network and it is very useful in deploying routers in a large-scale environment as it eliminates much of the manual intervention. ZTP is also used to automate the software upgrade process and help with a high level of network automation.

ZTP Workflow

A new hardware platform comes pre-installed with the ONIE (Open Network Installation Environment). ONIE is an open-source installation environment that acts as an enhanced boot loader utilizing capabilities in a Linux or BusyBox environment. ONIE allows users and channel partners to install the Network Operating System as part of provisioning.

ONIE requires a management LAN to obtain the configuration and software image information through the management interface. ONIE can access only the management interface. It starts a Dynamic Host Configuration Protocol (DHCP) based discovery process to obtain basic configuration information, such as the management IP address and the URL of the image to install on the bare-metal switch.

Then ONIE pulls the image and boots it.

Even after ONIE boots the image, the switch is not configured. This leads to questions about how to configure the switch.

The RtBrick images come with some pre-installed daemons. The pre-installed Control Daemon (CtrlD) is responsible for the management of the switch, and takes over after the image is activated.

The Control Daemon is responsible for configuring the switch. To do this, the hardware platform must be connected to the DHCP server and the management server through a management LAN.

The management server is responsible for providing the image binaries and the configuration of each device.

In the ZTP, ONIE performs the role of discovering, downloading and activating the image from the image registry.

In essence, the following is the high-level workflow of ZTP process:

ONIE performs the following tasks:

DHCP discovery
Image download
Image activation

Control Daemon performs the following tasks:

DHCP discovery
Switch configuration

ONIE allows to automate the firmware update. The image request to the management server is slightly different, and the management server needs to provide the firmware update image that the device vendor provides.

This section provides information about the NOS installation and firmware (FW) update.

ZTP Process

This section provides information about ZTP process. Figure. 1 illustrates the ZTP process at a high level.

The ZTP process is divided into two main parts:

Software Image Discovery and Installation

The ONIE in the device uses information that you have defined on the Dynamic Host Configuration Protocol (DHCP) server to locate the IP address and image download URL.

ONIE uses different ways to pull the image from the repository for downloading. In the ZTP process, HTTP is used to pull the image because ONIE conveys the serial number as the HTTP header. This serial number allows the image registry to identify the switch and select the appropriate image.

Along with the serial number, ONIE also sends the onie-operation that allows to distinguish between an os-install and onie-update, and select the correct image for either NOS install or firmware upgrade.

See the ONIE image discovery for further information (/ONIE/)
CtrlD configuration discovery and application.
CtrlD sends DHCPINFORM to request all options required for configuration discovery.
The configurations are downloaded from the management server (httpd) and applied.

Figure 1. The ZTP Process

Figure 2. depicts the relationship between the fabric, the DHCP server, and the management server.

Figure 2. The Management Server Architecture

DHCP Service

Because of its low set of requirements, the default DHCP server shipped with ubuntu, isc-dhcp, is used to run the DHCP service.

The following code shows an example configuration of a DHCP server and hardware box (dhcp.conf).

dhcp.conf

authoritative;
default-lease-time 600;
max-lease-time 72----

# This is only needed if the version is lower than 4.4
option loader-pathprefix code 210 = text;


subnet 10.0.0.0 netmask 255.255.255.0 {
  range 10.0.0.200 10.0.0.250;
  option routers 10.0.0.138;
  option domain-name-servers 10.0.0.210;
  option domain-name "local";
  host LEAF01 {
    # Identify client by MAC address.
    hardware ethernet 48:65:ee:11:da:85;
    # Identify client by serial number
    option dhcp-client-identifier "\000WLC1C27L00003P2";
    fixed-address 10.0.0.250;
    option host-name LEAF01;
    # Set DHCP option 114 (default-url) to set the installer image URL.
    # ONIE loads the installer image from the specified URL.
    option default-url "http://managementserver/ztp/image";
    # Set DHCP option 210 (path prefix) to set the configuration base URL.
    # CTRLD loads all configuration files from this base URL.
    option loader-pathprefix "http://managementserver";
  }
}

Most of the used options are already predefined in the ISC-DHCP server. You can see the reference under /ISCKB/, the loader-pathprefix is defined since DHCP 4.4, so if you use an older one, define it as described above.

HTTP Service (Management Server)

The HTTP daemon (httpd) is responsible for providing the NOS installer and the configuration files.

Therefore, a self-implemented Golang HTTP server is used, which reads the ONIE_SERIAL_NUMBER and ONIE-OPERATION HTTP header and maps them to the NOS/FW installer image download path, and maps the serial number to the ZTP configuration files. For more details about the configuration files, see the following section.

The ONIE-OPERATION header can have the following values:

install nos: os-install
update firmware: onie-update

The following sections provide information about the installation and configuration of the server.

ZTP installation

For the installation, you can choose any one of the following two options:

ZTP Installation with the Debian Package

You must perform the following steps for ZTP installation using the Debian package.

Ensure that you have added the rtrbick repository to your apt.sources list and updated the cache.
Ensure that the port 80 is available and not in use on your device.
Install the package rtbrick-fabric-ztp.
The package installs a systemd service named rtbrick-fabric-ztp.
Ensure that the service is running with sudo systemctl status rtbrick-fabric-ztp.
The default location for the ZTP configuration files is /var/rtbrick/ztp/configs/ where you need to copy your configuration files.

If you want to override server settings, perform the following:

Edit the service configuration file /etc/systemd/system/rtbrick-fabric-ztp.service and add parameters to the ExecStart command.
Parameter --addr: the listen address of the server, default is 0.0.0.0:80.
Parameter --requestTimeout: the request timeout server in seconds, default is 600, must possibly be increased depending on the connection speed and image file sizes.
Parameter --filePath: the location for the ZTP configuration files, the default location is /var/rtbrick/ztp/configs/.

ZTP Installation as Docker Container

You must perform the following steps for ZTP installation as a docker container.

Ensure that you have access to the rtbrick docker registry.
Ensure that the port 80 is available and not in use on your device.
Create a compose file docker-compose.yml. The following is a sample compose file.

version: '3.3'
services:
  ztp:
    image: 'docker.rtbrick.com/rbms-fabric-ztp:latest'
    container_name: rbms-fabric-ztp
    restart: unless-stopped
    ports:
      - '80:80'
    volumes:
      - './configs:/var/rtbrick/ztp/configs'

The compose setup uses a 'bind mound' method for the ZTP configuration folder. Therefore, the docker-compose.yml must be placed in the same location together with the ./configs folder for the ZTP configurations. To know the details of the configuration files, see the following sections.
If required, adapt the compose file for a different image version, port binding or different configuration folder location.
Start the container using the docker-compose up -d command.

ZTP configuration

The HTTP service matches the ONIE-SERIAL-NUMBER header to the configuration files. Therefore, the configuration folder should contain a JSON file for the serial number (<serial_number>.json) for each supported serial number.

This file contains settings for locations of all additional configuration files that have to be delivered for the specific device and settings for the NOS installer image and the firmware update image.

Example sample.json file for a serial number 'sample':

/var/rtbrick/ztp/configs/sample.json

{
  "description": "192.168.202.116",
  "ctrld": "ctrld.json",
  "ctrldrbac": "ctrldrbac.json",
  "startup": "sample_startup.json",
  "accessjwks": "sample_accessjwks.json",
  "apigwd": "sample_apigwd.json",
  "tls": "sample_tls.pem",
  "image": "http://pkg.rtbrick.net/_/images/latest/rtbrick-onl-installer/rtbrick-onl-installer-accessleaf-qmx-20.4.0-g8daily.20200415051734+Bmaster.C059a09ea",
  "update_image": "http://pgk.rtbrick.net/firmwares/onie-firmware-x86_64-ufispace_s9600_32x_ufispace_s9600_64x-r0_v0.3.0.updater"
}

Image Location Configuration

For the configuration entries "image" and "update_image" you have three possibilities:

Redirect URL: Configuration value must start with http, the server redirects the request to download the image from the URL. For example, "http://pkg.rtbrick.net/_/images/latest/rtbrick-onl-installer/rtbrick-onl-installer-accessleaf-qmx-20.4.0-g8daily.20200415051734+Bmaster.C059a09ea"
Absolute File Location: config value must start with /, can point to any file on the local disk, example /usr/share/images/rtbrick-onl-installer.img.
Relative File Location: config value must be a filename and not start with /, points to any file in the <ztppath>/configs/images/ folder, example "rtbrick-onl-installer.img"

ZTP APIs

For information about ZTP REST APIs, refer to the ZTP Management Server API.

Control Daemon

Once the RBFS image is activated by ONIE, Control Daemon (CtrlD) is responsible for executing the remaining tasks and configuring the switch. CtrlD acts as a post-ZTP demon, it runs after the image is activated.

There are various configuration files that CtrlD can load from a management server and apply to the system.

CtrlD config: This is the base configuration for CtrlD. There the RBMS and Graylog can be specified, but also the authentication and authorization mechanism can be controlled.
CtrlD rbac policy: The Role Based Access Control (RBAC) policy of CtrlD is defined in this configuration file.
Startup Config: This is the file for RBFS switch configuration.
TLS pem file: This file is intended for the API Gateway (ApiGwD). The file is an X509 public/private key file in PEM format defined in the RFC7468.
Access JWKS file: This file is intended for the ApiGwD. The JSON Web Key Set (JKWS) is described in the RFC 7517.

Trigger the ZTP process

The ZTP process in CtrlD is triggered for a specific container (LXC) on the switch. This can be triggered in the following ways.

By the switch (RBFS Linux container) itself by sending the operational state up to CtrlD.
By sending a REST request to trigger the ZTP process to CtrlD (`/api/v1/ctrld/ztp/_run').

You can also enable or disable ZTP in the switch configuration. By default, ZTP is enabled.

To disable ZTP, enter the following command:

set system host ztp-enable false

RestConf Snippet:

{
  "ietf-restconf:data": {
    "rtbrick-config:system": {
      "host": {
        "ztp-enable": "false"
      }
    }
  }
}

If load-last-config is set to false, the configuration is ephemeral. The default value of ztp-enable is true. As a consequence of this, by default, after every reboot ZTP gets triggered even if you set ztp-enable to false. So to disable ZTP you need to set ztp-enable to false and load-last-config to true.

Trigger the reinstall

The reinstall of a switch can be triggered by sending a POST request to CtrlD (/api/v1/ctrld/system/_reinstall)

Trigger Firmware Update

The firmware update of a switch can be triggered by sending a POST request to CtrlD (/api/v1/ctrld/system/_update)

Management Server URL Discovery

CtrlD has to discover the management server URL to download the configuration files from the management server. Therefore, a management interface, that allows sending an DHCPINFORM request to the DHCP server, is defined.

The request contains the DHCP option 61 that conveys the client identifier. The client identifier is either omitted or contains the serial number. The serial number is gathered from the ONIE file system information file /lib/platform-config/current/onl/onie-info.json. If that does not result in a valuable result the following command is executed dmidecode -s system-serial-number (see /RFC2131/ and /RFC2132/ for further information).

There are at least two DHCP options requested, DHCP option 54 that conveys the IP address of the DHCP server (see /RFC2132/ for further information), and DHCP option 210 that conveys the path prefix for all configuration files (see /RFC5071/ for further information).

If the DHCP option 210 is not returned, CtrlD attempts to read the configurations from the IP address of the ZTP server. Otherwise, CtrlD attempts to read the configurations from the base URL specified in DHCP option 210.

Request configurations

The request to the Management server contains the following HTTP headers:

ONIE-SERIAL-NUMBER: This serial number is either the onie serial number or empty string.
CONTAINER-NAME: Container that triggered the ZTP process.
ELEMENT-NAME: Element name that triggered the ZTP process.
HOST-NAME: Host name of the device that triggered the ZTP process.

All this information can be used to select the right configurations for the container. This also allows the use of ZTP Configuration Process for virtual environments.

The requested URL:

CtrlD Config: <management server url>/ztp/config/ctrld
CtrlD rbac policy: <management server url>/ztp/config/ctrldrbac
Startup Config: <management server url>/ztp/config/startup
TLS pem file: <management server url>/ztp/config/tls
Access JWKS file: <management server url>/ztp/config/accessjwks

If any of the file is not found, the process still goes forward.

Business Events

During the ZTP Process log messages are sent to the configured ztp graylog endpoint.

For more information, see the switch API documentation.

Overall Process Flow

The following two figures show the CtrlD ZTP process flow.

Figure 3. CTRLD ZTP process flow (Part 1/2)

Figure 4. CTRLD ZTP process flow (Part 2/2)

References

References
/ONIE/	Open Network Installation Environment Image Discovery
/RFC2131/	RFC2131 - Dynamic Host Configuration Protocol
/RFC2132/	RFC2132 - DHCP Options and BOOTP Vendor Extensions https://tools.ietf.org/html/rfc2132
/RFC5071/	RFC5071 - Dynamic Host Configuration Protocol Options Used by PXELINUX
/ISCKB/	ISC Default DHCP Options