A national programmable infrastructure to experiment with next-generation networks

Published on Tue March 31, 2020

We have set up the first multi-domain P4-programmable network in the Netherlands, consisting of six different sites interconnected by SURF's optical network. The purpose of the network is to experiment with novel inter-domain network functions and emerging internet architectures that fulfil the security, stability, transparency and autonomy requirements of future networked applications such as cooperative driving, intelligent transport systems and remote surgery. The testbed is part of the 2STiC programme, and in this blog we provide a brief overview of its set-up, examples of how we have used it and our plans for the coming six months.

New security, stability and transparency requirements

Some time ago, we published a long read about the 2STiC (pronounced “to stick”), our research programme aimed to increase the security, stability and transparency in inter-network communications and enable societies to stay in control of their network infrastructures. We're pursuing that objective by developing, experimenting with and contributing to new network functions and emerging internet architectures, such as SCION, RINA and NDN. We build on programmable networks, the equipment for which is becoming commercially available and can thus act as a further enabler for prototyping and testing these emerging internet technologies without compromising on speed or scalability.

The 2STiC partners envision that new types of internet will complement and co-exist with the current Internet while providing the increased security, stability, and transparency for critical applications such as intelligent transport systems, healthcare, water management and energy grids. Our long-term objective is to establish a centre of expertise in the field of trusted and resilient internets, thus helping the Dutch and European networking communities to play a leading role in this exciting field.

2STiC is a collaborative programme involving AMS-IX, NLnet Labs, SIDN Labs, SURF, Technical University of Delft, University of Amsterdam and University of Twente.

2STiC testbed

When we started 2STiC last year, we first built the 2STiC testbed: a national programmable network interconnecting the 2STiC partners. With this programmable network we can experiment with new internet infrastructures, open programmable network equipment, new network functions such as path control and verification, and network slicing across administrative domains.

For programmability we use P4, which is a domain-specific programming language that allows network developers to program the data plane of a packet forwarding device. Amongst research networks, there is increasing interest in evaluating P4 devices. In this blog we will elaborate two of the use cases we explored: supporting protocol design with hardware implementations and path tracking.

P4 enables us to implement any (internetworking) protocol on the hardware in the testbed, as long as it aligns with the physical layer of the Ethernet frame. That makes it very suitable for research on new network functions and (non-IP) protocols, such as SCION and RINA. The usage of programmable hardware switches and interfaces augments research using network simulators and software switches, because it offers the possibility of verifying results on hardware, for example by showing that a protocol can run on physical switches.

Figure 1 shows the 2STiC testbed, which currently connects six of the seven 2STiC consortium partners. We use a star-shaped network centred on the SURF location in Amsterdam, which allows us to configure different network topologies.

Each node contains a P4 programmable switch with a Barefoot Tofino ASIC, which can be seen in Figure 2. We can expand the nodes with additional hardware, from a simple server with a programmable Netronome SmartNIC to a complete research group testbed.

We also connected the 2STiC testbed to the i-P4EN initiative (Figure 3), which interconnects P4 programmable testbeds in other countries, such as Canada, Taiwan and the USA. We expect that the i-P4EN initiative will attract use cases from data-intensive science and network research institutions around the world. The 2STiC testbed is thus part of a worldwide initiative to support distributed network research (including on 100Gbit/s paths) and provide a development environment for advanced empirical experiments at global scale.

In the following sections, we discuss how we have used the 2STiC testbed so far: to implement the SCION protocol on P4 hardware and to track path the paths that data takes through an IP-based internet.

Use case #1: Implementing SCION in P4

SIDN Labs is implementing SCION in P4 (see the blog on our research on SCION). SCION (Scalability, Control, and Isolation on Next-Generation Networks) is an internet architecture developed by the Network Security group at ETH Zurich in Switzerland and their spin-off company Anapaya.

SCION is intended to deliver more stable and secure internet connectivity by isolating trust in so-called Isolation Domains (ISDs), which are collections of autonomous systems. At the same time, SCION users will benefit from more control over and insight into the inter-domain routes their traffic takes (i.e. which autonomous systems the traffic passes through). We decided to start with SCION as it has an actively maintained software implementation, an active community and a public international testbed.

We started our implementation with the basic forwarding functionality for the open-source P4 simulator simple switch. Currently, we’re working to implement the SCION protocol for the Barefoot Tofino ASICs that we use in the 2STiC testbed. That will allow people to run SCION on the testbed and evaluate its performance when running on hardware. We feed our experiences back to Barefoot and the SCION team at ETH Zurich and Anapaya. Based on the experience gained so far, we have made several recommendations for changes to the SCION protocol to make it easier and more efficient to implement in hardware.

At the moment, we are also setting up a BGP-free connection from SIDN Labs to SCIONLab, which is the international SCION testbed. The purpose of the BGP-free connection is to enable native experimentation with SCION’s inter-domain routing functions, without dependency on or influence by current routing technologies. In the future, we expect we can also use this connection to hook up the 2STiC testbed to SCIONLab.

Use case #2: Path tracking

Certain new technologies (such as SCION) are intended to determine (as part of their routing choices) how the traffic is flowing or has flowed through the network. That implies tracking the route. Using the software switch, with BMv2, INT (In-band Network Telemetry) and the Netronome SmartNICs, the University of Amsterdam verified whether the traffic was routed as expected/required or took an unintended path.

Two different chunks of information relating to a packet’s path are recorded: node recording and forwarding state logging.

Node recording: at each node we append the Node Identifier to the IPv6 extension header. The complete path a packet took is extracted from the IPv6 extension header in the last node of the path. The process can be seen in Figure 4, where the data in the IPv6 extension header at the last node contains the three Node Identifiers: A, B and C.

Forwarding state logging: For this scenario we assume that the routing information for each node is stored and available. At each node, the information about the routing table version (used to make the forwarding decision) is included in the IPv6 extension header. The complete path can be reconstructed based on the node where the packet entered a network.

In Figure 5 the Node Identifier field identifies the entry point in the network; the version field identifies the routing table version at the entry point in the network (which is “V1” in the version field). The trackable field is used to flag version changes along the path (which is shown by the “1” after exiting node C, as the routing table changes from V1 to V2).

Experience gained

As the use cases outlined above illustrate, we have already gained valuable experience by using the 2STiC testbed. Programmable network equipment allows us to implement new protocols and run them at high speeds. Implementing novel network technologies in hardware is not a trivial task though: protocols are often not designed with high-speed hardware implementation in mind. At the same time, programming high-speed hardware requires additional skills, as the underlying principles and algorithms are quite different compared to generic software programming. The 2STiC consortium tries to bridge the gap between hardware and protocol developers and has been able to feed all its experience back to the respective developers, leading to several improvements that we'll trial again on the 2STiC testbed.

The monitoring tools have increased understanding of how INT can be implemented and the experience gained has been fed back to the OPSAWG/IPPM groups and header extensions discussions in the IETF. We will be able to draw on the experience gained when we look at other internet protocols. Furthermore, the results of the in-band monitoring scenarios have been presented in a poster at the Internet Measurement Conference 2019.

Future work

In the coming months we will be working on various items, including the following:

Upgrade of 2STiC testbed using optical circuits: The 2STiC testbed is currently built around 100Gbit/s Ethernet-based circuits (PBB-TE). To become less dependent on the transport protocols at a lower layer, SURF will replace the PBB-TE with 200Gbit/s optical circuits. As well as increased speed, the future 2STiC testbed will provide full transparency to protocols like IP and Ethernet.

Interconnecting SCION networks: The interconnections provided by an IXP (Internet eXchange Point) increase redundancy and availability for interdomain routes, which can be further extended by SCION. So we plan to involve the AMS-IX community in the use of SCION (or other novel technologies) on the AMS-IX interconnection platform.

Contact us

If you are interested in using or joining the 2STiC testbed, please contact us at testbed (at) 2stic (dot) nl.