This section collects information on the client implementations of the ALTO protocol and their
features/extensions.
| Publisher | Available Features | ||||||||||||||||||
| Name | Source | Language | License | Maintained by | NM | CM | ECS | EPS | MC | CC | SSE | UP | PV | PM | CDNi | TIPS | HTTP | ||
| Paper (2013) | NEC Europe Ltd., Heidelberg, Germany | Yes | Yes | ||||||||||||||||
| CBCD | Paper (2012) | Network Technology Lab, Korea Telecom | Yes | Yes | |||||||||||||||
| Paper (2012) | Alcatel-Lucent Bell Labs | Yes | Yes | ||||||||||||||||
| Paper (2017)Paper (2018) | Technical University of Denmark | Yes | Yes | ||||||||||||||||
| Sextant | Paper (2021) | Tongji-Yale Lab & Telefonica | Yes | Yes | |||||||||||||||
| Paper (2020) | University of Minho | Yes | Yes | Yes | |||||||||||||||
This section collects information on the server implementations of the ALTO protocol and their
features/extensions.
| Publisher | Available Features | ||||||||||||||||||
| Name | Source | Language | License | Maintained by | NM | CM | ECS | EPS | MC | CC | SSE | UP | PV | PM | CDNi | TIPS | HTTP | ||
| Paper (2013) | NEC Europe Ltd., Heidelberg, Germany | Yes | Yes | ||||||||||||||||
| CBCD | Paper (2012) | Network Technology Lab, Korea Telecom | Yes | Yes | |||||||||||||||
| Paper (2012) | Alcatel-Lucent Bell Labs | Yes | Yes | ||||||||||||||||
| Paper (2017)Paper (2018) | Technical University of Denmark | Yes | Yes | ||||||||||||||||
| Sextant | Paper (2021) | Tongji-Yale Lab & Telefonica | Yes | Yes | |||||||||||||||
| Paper (2020) | University of Minho | Yes | Yes | Yes | |||||||||||||||
This section collects information on applications/libraries that are developed on top of the ALTO
protocol.
| Publisher | Used Features | |||||||||||||||||
| Name | Source | Language | License | Maintained by | Description | NM | CM | ECS | EPS | MC | CC | SSE | UP | PV | PM | CDNi | TIPS | HTTP |
| Unicorn | Homepage | Python & Java | MIT | OpenALTO Community | ALTO is used for multi-domain resource orchestration | Yes | Yes | |||||||||||
| Paper (2013) | VPN optimization based on ALTO information | Yes | Yes | |||||||||||||||
| Sextant | Paper (2021) | Tongji-Yale Lab & Telefonica | CDN selection. Forked from the ODL ALTO project. | Yes | Yes | |||||||||||||
| Paper (2017) | Technical University of Denmark | Video streaming with ALTO. Propose new routing metrics. | Yes | Yes | ||||||||||||||
| Paper (2018) | Edge caching optimization with ALTO. | Yes | Yes | |||||||||||||||
| Paper (2020) | University of Minho | P2P file sharing using ALTO. | Yes | Yes | ||||||||||||||
| Paper (2021) | University of Minho | P2P file sharing. Multi-domain setting. | Yes | Yes | ||||||||||||||
| Paper (2018) | NEC | P2P live streaming. | Yes | Yes | ||||||||||||||
| Paper (2017) | CDN optimization | |||||||||||||||||
| FlowDirector? | Paper (2019) | Benocs GmbH | Collaborative traffic engineering | Yes | Yes | |||||||||||||
The deployment of ALTO in Telefonica has been primarily motivated by the need for automating the process of network topology retrieval and generating an up-to-date topological information that can be consumed by different applications. With such a topological view, it is possible to optimize the delivery of traffic within the Telefonica network, with some clear benefits: improvement of the quality of experience perceived by the end user, more efficient usage of IP network resources, etc.
Currently, media streaming distribution is the main application consuming ALTO's network topology information. Telefonica develops its own Content Delivery Network (CDN) solution in order to provide IPTV and Video On Demand services to its base of customers, either from Telefonica video services (under the brands Movistar+ and Movistar Play) or from third parties. The video delivery is performed using HTTP Adaptive Streaming (HAS) protocols, then serving both internal and external customers to Telefonica in an Over-The-Top (OTT) fashion.
The Telefonica CDN (TCDN) implements state-of-the-art Request Routing Logic (RRL) which considers multiple information sources in order to maximize both QoE and video delivery efficiency. Some RRL inputs are streamer health status and load level, cache hit ratio maximization, content popularity, etc, but also and importantly, the network topology (PIDs and cost matrix). With all this information, the TCDN takes the final decision on selecting the more appropriate delivery point for a given content requested by a certain end user, coupling network and application information.
It is also here where ALTO plays an essential role by providing an automated way of retrieving the network topology. Before the integration of ALTO in TCDN, the network topology has been generated and provisioned manually. This is problematic not only because it is a process prone to errors, but also because the topological information (i.e., the allocation of the IP prefixes of the end-users requesting a given content) becomes outdated throughout time.
To overcome such limitations, ALTO is being integrated with TCDN. Two kinds of PIDs are generated: (1) the PIDs associated to customer’s IP prefixes which are the consumers of video streaming, and (2) the PIDs associated to the connection of CDN streamers to the network, representing the potential sources of TCDN traffic.
For the selection of the more convenient streamer in each case, the RRL takes into consideration the lowest cost between the PIDs of the CDN streamers and the ones of the customers. The association of IP prefixes to PIDs generates the ALTO network map, which is obtained by means of BGP, and the hop count among PIDs generates the ALTO cost map, which is created parsing BGP-LS information. For this purpose, ALTO connects with a number of Route Reflectors of the Telefonica’s backbone, using exaBGP as BGP speaker.
-
BGP :
+------+ +-----------+ session : +------+
| | | ALTO | ----->| RR_1 |
| TCDN | | +-------+| / : +------+
| |<---->| | BGP ||<--- :
| RRL | | |speaker||<-- :
| | | +-------+| \ : +------+
+------+ +-----------+ ------>| RR_2 |
BGP-LS : +------+
session :
: Telefonica
: backbone
:
The PoC is currently in place. Before the PoC, several tests were performed in the lab and pre-production network. Currently ALTO is deployed in the production network of Telefonica Spain. At this stage, only a part of the topology can be retrieved because of a bug in one of the network vendors, which affects the activation of BGP-LS in all the network. This is expected to be solved during Q2 2023. Even with partial information, the integration of ALTO is resulting valuable for validating the capability of updating network topology changes automatically.
After the PoC in Spain, Telefonica is planning to extend the same solution to other countries where Telefonica is present (Brazil, etc). As part of the evolution of the solution, Telefonica is working on incorporating richer performance metrics than the number of hops, as described in draft-ietf-alto-performance-metrics.
The progress of this PoC has been presented to both ALTO and MOPS WGs during IETF 114 [ALTO, slides 19 to 22], [MOPS] and IETF 115 [ALTO],[MOPS].
The expriments (e.g., ATLAS, CMS, LHCb) at CERN produce a large amount of data, which need to be distributed and processed globally. The networking infrastructure supporting the data distribution is realized by a L3VPN network called [LHCONE], which consists of more than 600 distributed storage systems, distributed across 170 data centers in 40 countries. Building on its success, the networking infrastructure is also being used by Belle II, Pierre Auger Observatory, NOvA, XENON, or JUNO. In 2022, just the aggregated outgoing traffic on LHCONE from CERN to its ten largest connected data centres reached 457 Petabytes of data.
The transport control of LHCONE is realized by two software control systems called Rucio (https://rucio.cern.ch/) and FTS (https://fts.web.cern.ch/fts/), where Rucio is the data orchestrator, which provides services such as selecting data sources and destinations given application-layer policies (e.g., replication rules); and FTS is the data transport scheduler, which determines when and at what rate a transfer received from a higher-layer (e.g., Rucio) is dispatched. As a large-scale, multi-domain, shared, often overcommited infrastructure, LHCONE requires efficient, flexible transport orchestration and scheduling.
The deployment of ALTO at LHCONE involves both integration with FTS and integration with Rucio.
The objective of FTS/TCN is to generalize the standard FTS Optimizer (https://fts3-docs.web.cern.ch/fts3-docs/docs/optimizer/optimizer.html), to (1) improve its efficiency and (2) provide flexible resource control capabilities in FTS. FTS/TCN realizes a new control scheme called flexible, strong resource control on top of minimal, universal, weak control knobs. It also can be considered as a hybrid control architecture, which leverages the underlying, fully distributed TCP congestion control to achieve fast time-scale efficiency and (congestion) robustness, and realizes global efficiency and flexible resource control using the coordination FTS controller.
Src Dst
\ /
+-------+ +------+ +----------+
| - --> | -- |------|---|- --> |
| AS1 | | AS2 | | AS3 |
+-------+ +------+ +----------+
| | /____________|___/
| \ / |
| -- / ----| /
+----------+ +--------------+ +------------+
| Resource | | Resource Use | | Scheduler: |
| Spec | | Mapping(ALTO)| | Concurrency|
+----------+ +--------------+ +------------+
The figure above shows the overall implementation architecture, which consists of 3 components: resource specification, resource mapping, and scheduling:
The ALTO+Rucio deployment complements the FTS/TCN. Specifically, it uses ALTO to provide a uniform interface to implement a core function of Rucio: replica sorting when conducting replica selection. The implementation consists of two components: (1) ALTO metric specification, and (2) Rucio sorting integration.
ALTO metric specification, with an example such as below:
ALTO IRD:
"as_hopcount" : {
"resource_type" : "path-vector",
"resource_id" : "cern-pv",
"prop-name" : "as_path",
"prop_transformer": "tolist | len",
"aggr_transfomer" : "sum"
}
"delay_ow": {
"resource_type" : "cost-map",
"resource_id" : "delay-ow",
"dependent_network_map": "default-networkmap"
}
Rucio sorting integration, using the commandline as:
Rucio integration:
rucio list-file-replicas
--sort='alto;stmt="By as_hopcount,delay_ow"'
The Flow Director (FD) is a production CDN-ISP collaboration system developed by Benocs. Its development started in 2013, and the system went live in 2017 and has been in production since then. The work won CoNEXT Best Paper and also the IETF/IRTF Applied Networking Research Prize 2020. Additional details of its development history can be found at the ACM CoNEXT 2019 paper: Steering Hyper-Giants’ Traffic at Scale and the [Flow Director IETF 112 slides],
A Flow Director for an ISP provides the following 3 functions: (1) it collects data to determine the state of the ISP’s network; using the collected data, it computes forwarding path and optional inventory and performance data; (2) it then computes the best ingress location for each customer prefix; and (3) it communicates with a cooperating (hyper-giant) CDN using automated, near real-time via ALTO, and out-of-band via BGP.
1. Data Collection (Southbound). Flow Director interacts with routing protocols including IGP (IS-IS, OSPF, IBGP) and EGP (BGP) protocols to collect network links and routers information. It also collects flow information from Netflow, to determine ingress/egress points. Through network monitoring including SNMP, Flow Director collects network utilization, bandwidth, and latency.
2. Best Ingress Computation (Core Engine). Flow Director computes best ingress points both for the ISP and for the CDN, with specified ISP and CDN objectives. An example of ISP objective is to reduce long-haul traffic, while an example of CDN objective is to reduce latency. For robustness, Flow Director's suggestions can be ignored by a CDN.
3. Communication with CDN (Northbound). Flow Director has implemented multiple northbound interafces. In particular, it provides the Base ALTO protocol (RFC7285) with all the provided features, including IRD, Network Map (NM),filtered-NM, Cost Map (CM), filtered-CM, Endpoint Cost Service (ECS), and Endpoint Property Service (EPS). Note that endpoints in the Flow Director are not IP addresses but IP subnets (CIDR). Flow Director also implements ALTO SSE. NM and CM are deployed in production.
An example of a large ISP that Flow Director supports includes:
>900 Routers
>760k IPv4 Prefixes, among which
>12k IGP Prefixes
>750k BGP Prefixes, among which
>170k IBGP Prefixes
>580k EBGP Prefixes
>20k IPv4 Ingress Prefixes, among which
>950 Ingress Points
~30% of public IPv4 Address Space
The ALTO information computed includes:
Network Map, with
>250k Prefixes
>1700 PIDs, among which
>750 Internal PIDs
>950 External PIDs
∼ 15 OnNet PIDs
With average map sizes:
Map: ∼ 6 MB ( ∼ 1 MB compressed)
SSE Patch: ∼ 1.7 MB ( ∼ 282 KB compressed)
Cost Map (Custom), with
>1.3M PID pairs
Average Sizes
Map: ∼ 47 MB ( ∼ 5.6 MB compressed)
SSE Patch: ∼ 37.5 MB ( ∼ 5 MB compressed)
Benefits for the ISP (Combined with network planning): 30% reduction long-haul traffic.
Benefits for CDN (Distance as a proxy for latency): 40% reduction.
The National Research Platform (NRP) is a network that interconnects educational and scientific institutions and computing facilities in the US, supported in part by awards from the National Science Foundation (https://nationalresearchplatform.org/). NRP’s primary computation, storage, and network resource is a distributed cluster of ~300 K8S nodes called Nautilus.
The NRP/ALTO deployment is overlayed on top of GradientGraph (G2), which is leveraged to pull real-time topology, routing, and flow information to compose the various ALTO objects (for instance, the ALTO Cost Map). The architecture is shown in the next diagram:
+-------------------------------------+
| ALTO |
+-----------------^-------------------+
|
+-----------------+-------------------+
| GradientGraph |
+----^------------^-------------^-----+
| | |
+----+----+ +----+----+ +-----+-----+
| K8S | | sFlow | | Perfsonar |
+----^----+ +----^----+ +-----^-----+
| | |
+----+------------+-------------+-----+
| Network |
+-------------------------------------+
In the next sections we describe each of the components.
GradientGraph (G2) is a technology developed by Qualcomm Technologies, Inc., that implements bottleneck structure theory ([1], [2], [3]) to make fast computation of optimized network reconfigurations. In this specific deployment, G2 ingests topology, flow and routing information from K8S, sFlow, and Perfsonar, respectively, to compute the bottleneck structure of the network and make operational recommendations. It also exposes a REST API that is queried by the ALTO server to retrive the network state.
The K8S controller's REST API is queried by G2 to retrieve topology information, including a list of K8S hosts, a list of all the links part of the network, and their capacities.
G2 also queries the PerfSonar REST API service running in NRP to retrieve routing information. PerfSonar periodically runs traceroute between every pair of nodes (K8S hosts) in the K8S cluster to derive the paths. Then, G2 queries the PerfSonar traceroute results to get the path between any two given nodes. Using sFlow information, it can also know the pod that is associated with each node, which allows to compute the pod-to-pod traffic paths.
To learn the active flows in the network, G2 queries the Inmon sFlow TrafficSentinel's REST API. The Inmon sFlow collector runs inside the NRP network and has visibility into all the inter-pod traffic.
The ALTO server is based on OpenALTO (https://github.com/openalto/alto). A G2 agent is developed, which queries the G2 RESTful API to obtain topology, routing and flow information. The agent also loads a static configuration file that defines equivalent classes based on IP prefixes. The static file is based on a static file generated by NetSage. The process will be made automatic in future releases.
The ALTO server now provides an endpoint cost service with the path vector extension enabled.