Cluster-to-Cluster Applications

Cluster-to-Cluster Protocol Coordination : Cluster-to-Cluster Protocol Coordination Ketan Mayer-Patel University of North Carolina kmp@cs.unc.edu

In A Nutshell : In A Nutshell Cluster-to-Cluster Applications Collection of endpoints. Many streams of information. Complex interstream semantics and relationships. Coordination Protocol Aggregate measurements of network conditions. Enables distributed, coordinated, application-driven adaptation.

Cluster-to-Cluster Applications : Cluster-to-Cluster Applications Endpoints are collections of devices and computational resources. Application is distributed over this collection.

Examples : Examples Tele-immersion Distance learning “Smart” spaces Ubiquitous computing environments Distributed sensor arrays Virtual reality “caves”

Networking Characteristics : Networking Characteristics Endpoint Aggregation Point Aggregate Flow Bundle

Adaptation : Adaptation Complex inter-stream semantic relationships. Adaptation must incorporate application-level knowledge about the role of different streams. Static policies may not be adequate. Example: region of interest within a tele-immersion application.

Problem : Problem Transport-level protocols lack the ability to coordinate with peer streams. Compete instead of cooperate. Particularly true when end-to-end path is not shared. Metaphor: Frayed Rope

Networking Requirements : Networking Requirements Peer awareness Individual streams should know about network usage for the aggregate. Global measurements of network conditions. All streams given a coordinated view of current conditions. Preserve end-to-end semantics.

Application-level Approach : Application-level Approach Multiplex all the streams together. Demultiplex on the other side. Drawbacks: Breaks end-to-end semantics of individual flows. Particularly complicated if flows have different reliability requirements. Centralizes adaptation. Would like to support “composable” applications. Introduces delay. Crossing the kernel/application boundary is expensive.

Congestion Manager : Congestion Manager Separates congestion from individual transport- and application- level protocols. Uses information from individual streams to determine aggregate available bandwidth. Coordinates usage of available bandwidth among different flows.

Close, but... : Close, but... Works when all of the end-to-end paths are the same. Coupled with a scheduler to provide coordination. Doesn’t provide individual flows with aggregate information. Architecture not well-suited for moving CM function to the aggregation point.

Coordination Protocol : Coordination Protocol Build coordinated transport-level protocols on top of a new protocol called “CP”. CP is charged with measuring network conditions for all associated flows. Delay, loss rate, available bandwidth, etc. This information is provided to the T-L protocol and the application. Allow the application to implement whatever adaptation policy it wants.

CP : CP Endpoint Aggregation Point Endpoint Aggregation Point Packet Path

Basic operation: Endpoint  AP : Basic operation: Endpoint  AP CP header identifies packet as part of a particular cluster application. IP Header CP Header Transport-level Header Packet Data Unused Unused Unused Protocol ID Flags C-to-C App ID Flow ID V

Basic operation: AP  AP : Basic operation: AP  AP CP header processed Bandwidth accounting. Some fields are overwritten with new information that will be used to estimate current network conditions. Available bandwidth and loss rate is reported. IP Header CP Header Transport-level Header Packet Data Timestamp Echo Timestamp Bandwidth Available Protocol ID Flags C-to-C App ID Flow ID V Seq No. Echo Delay Loss Rate

Basic operation: AP  Endpoint : Basic operation: AP  Endpoint Network conditions assessed and fields are overwritten with report information. Network delay, delay variance, available bandwidth. Aggregate bandwidth used, number of cluster flows. IP Header CP Header Transport-level Header Packet Data Round Trip Time Aggregate Bandwidth Used Bandwidth Available Protocol ID Flags C-to-C App ID Flow ID V RTT Variance No. of Flows Loss Rate

Features : Features Leverages equation-based congestion control. RTT measured using first available packet of any flow within the application. Loss rate measured across all flows. Does not require changes to interior routers. AP’s are simply accounting. Keeps the forwarding path relatively fast. Allows application complete control over actual adaptation policy. All state is soft.

T-L Requirements : T-L Requirements Has to provide application with a way to attenuate CP report. Example: C-TCP built on top of CP needs to know how much of advertised available bandwidth it should use. Variety of approaches: Application-level callback. Modified socket interface.

Congestion Control : Congestion Control A1 A2 A3 B1 B2 B3 AP AP 1 2 3 4 5 6 7 8 CC Algorithm TFRC RAP Binomial Packet Events Avail. BW Available BW reported in packets on reverse path. Packet Event: RTT Seq. No. Packet Size Arrival Time Lost?

Measuring RTT : Measuring RTT Timestamp-based mechanism: CP header contains a timestamp. Last timestamp heard from other AP is echoed in a timestamp echo. Delay between last timestamp received and current echo is also reported. Allows AP to update RTT estimate without keeping any per packet state. Key feature: uses next available packet from any of the associated flows.

Measuring Loss : Measuring Loss Gaps in sequence number in CP header. Not responsible for reporting specific losses. T-L operation for reliable protocols. All packets from any associated flow are multiplexed into a single seq. space. Allows for aggregate measures of loss across the entire application. Seq. number space can be kept fairly small.

How much feedback? : How much feedback? T-L protocol will generate packets based on its own feedback needs. Application can set up “empty” packet streams solely for feedback. AP Generated Feedback Design easily modified to generate feedback at AP if no feedback packet is available. Not clear that we actually want to do this.

Application Architectures : Application Architectures Distributed control. Each endpoint determines what portion of reported available bandwidth is used independently. Depends on independent decisions aggregating correctly by design. Examples: Static proportions. Functions of available bandwidth that sum to 1.0 at all points. Distribution among flows in an aggregate is arbitrary as long as sum of bandwidth used is equal to available bandwidth calculation.

Application Architectures : Application Architectures Centralized control. May have a single process on one of the endpoints that translates current conditions. Application-level control mechanism from the controller to all of the processes involved.

Policing the Aggregate : Policing the Aggregate What happens if the sum of the traffic is greater than the available bandwidth reported? One solution: guarantee that this won’t occur. CM does this by modulating sends with a scheduler. Hard to do in our distributed architecture. Our approach: don’t do anything.

Multishare Problem : Multishare Problem Conforming aggregates are limited to one “flowshare”. This is actually more of a policy issue. Simply producing k * available bandwidth will break the CC algorithm. The algorithm will only work if the source responds to detected congestion in the prescribed manner.

TFRC Multishare Problem : TFRC Multishare Problem

Bandwidth-Filtered Loss Detection : Bandwidth-Filtered Loss Detection Create a “virtual” packet event stream which is sampled from the actual packet event stream. Sample stochastically. Use the ratio between available bandwidth and actual bandwidth to determine sampling probability.

Congestion Control : Congestion Control A1 A2 A3 B1 B2 B3 AP AP 1 2 3 4 5 6 7 8 CC Algorithm TFRC RAP Binomial Packet Events Avail. BW Available BW reported in packets on reverse path. Packet Event: RTT Seq. No. Packet Size Arrival Time Lost?

TFRC w/ BFLD : TFRC w/ BFLD