Bloomberg: Achieving a World-Class Campus Network
How Bloomberg achieved its ambitious project to transform a 1.1 million square foot site into a technology-rich workspace.
Location: London, UK
Sector: Software & Media
Completion Date: 2018
• Network Architecture enables Smart Sustainability, helping to achieve a BREEAM Rating of 98.5%
• Technology Vision, Strategy, Design, Implementation, and Optimisation for over £70m worth of technology
• Intelligent, interconnected systems provide Bloomberg with detailed performance and space utilisation analytics, aiding decision making and allowing Bloomberg to constantly monitor and improve the efficiency of the building
• The future-proofed infrastructure allows Bloomberg to incorporate state-of-the-art innovations and systems onto the existing corporate network
• Commitment and attention to every detail of the staff experience throughout the space provide Bloomberg with a platform to attract and retain the very best talent to the organisation
PTS Consulting were engaged by Bloomberg L.P. to assist in their IT strategy execution, procurement, and implementation management relating to Bloomberg’s relocation to a new UK HQ in the City of London. The entire new Bloomberg London site comprised of two distinct buildings occupying approximately 1,000,000 square feet. These buildings were physically interconnected at various levels and shared the same basement area facilities.
As part of the relocation engagement, Bloomberg sought PTS assistance with establishing a network infrastructure to support all IT and building services within the new UK HQ site, providing end users and guests with the optimum combination of quality, service level and cost. An aspiration that PTS understood from the very beginning of the project was to incorporate the latest IT Network design philosophies and deliver (in a timely manner) innovative solutions offering flexibility and leading-edge technical design.
“PTS ensured that Bloomberg’s requirements were met throughout the lifecycle of the project and helped ensure the delivery of a world-class, leading-edge campus network incorporating the latest in innovative network design philosophy, delivered both to schedule and within budget constraints.”
Bloomberg’s position as a primary global financial software, data, and media company meant that stability and performance of the network was key. The network needed to be highly available but at the same time flexible enough to scale where required and provide robust security features throughout.
Network segmentation was a key concern for Bloomberg with the network design having to support the ability to deploy multiple networks, while ensuring that each network deployed remained performant, highly available and secure.
Bloomberg were very aware of the issues inherent with traditional layer 2 based architectures where loops in the network could cause slow convergence, wasted bandwidth (through uplink blocking) or sub-optimal traffic flows. For these reasons, Bloomberg were very keen to explore a modern network design where layer 3 was brought right to ‘the edge’.
A further challenge for Bloomberg was the aspiration to create an ‘intelligent building’ where the vast majority of building systems and building management systems would reside on the IP network. Along with the obvious challenge of needing to deploy a resilient, performant and secure building network there was also the very tangible issue of needing to secure the building network from the rest of the campus network. There was also a requirement to facilitate various applications that relied on protocols that predicated hosts co-existed within the same (layer 2) broadcast domain (either due to fixed IP subnet requirements or a requirement to send/receive broadcasts for discovery purposes).
PTS worked with the Bloomberg network architecture team and their chosen network integrator to design a campus network based on a ‘leaf-spine’ architecture. Leaf-spine is a network architecture design that has been gaining significant traction in modern data centre environments and the vision for Bloomberg was to translate the known advantages of this design into the campus network.
Within a leaf-spine architecture every access switch connecting host devices (leaf layer) is connected to each of the core switches (spine layer) in a full-mesh, highly redundant topology (with the spine switches therefore forming the backbone of the network). As each leaf switch is connected to every spine switch, the spine switches themselves are never inter-connected. In a leaf-spine architecture, traffic sourced from any host in the network is always the exact same number of network hops away from any other host in the network. Put simply, in the Bloomberg network design most network traffic would be just two network hops from source to destination; specifically, one hop from the source leaf to the spine layer and one hop from the spine leaf to the destination leaf (with the only exception being when two hosts reside on the same leaf switch). Knowing that hosts on the network are equidistant keeps latency at a predictable level and provides for deterministic traffic flows.
Layer 3 IP Fabric
The network design chosen for Bloomberg was further enhanced through the use of a Layer 3 IP fabric deployed on top of the spine-leaf architecture. By extending Layer 3 to the edge, the network design removed traditional VLAN/MAC constraints, reduced fault/broadcast domains, solved the issue of sub-optimal traffic flows (common when using traditional layer 2 loop avoidance mechanisms) and allowed for ECMP (equal cost multi-path routing) to be leveraged via load balancing flows across the multiple spine nodes (with each leaf node having multiple paths of equal ‘cost’ to the spine layer). Path selection was effectively random (a load-balancing algorithm configurable based on defined L3/L4 information for added granularity) to ensure traffic load was evenly distributed among the spine switches. In this way, if any of the spines were to fail, it would only slightly degrade performance rather than cause a significant outage preventing traffic from forwarding correctly.
The process for expanding capacity in a leaf-spine network is straightforward. Additional leaf switches can be added to provide extra host port connectivity by simply ensuring the new leaf is connected to every existing spine switch. Alternatively, if oversubscription becomes an issue then an additional spine switch can be added and links extended down to every existing leaf switch (effectively increasing the available bandwidth while reducing oversubscription).
For security, the use of a layer 3 IP fabric allowed for network virtualisation techniques to be used such as VRF-Lite (Virtual Routing and Forwarding). The use of VRF-Life in the Bloomberg design ensured that multiple secure virtual networks could be deployed using the same physical infrastructure.
The use of VRF-Lite in the Bloomberg design effectively created multiple logical spine-leaf networks, each one completely isolated from the others. This meant multiple security domains could be created that maximised the technology investment while allowing for network segmentation at layer 3 and also at layer 2 (via traditional VLANs). In the Bloomberg design, Inter-VRF traffic would only be possible through best-of-breed next generation security enforcement points deployed strategically within the Bloomberg network.
For applications where stretched layer 2 was still a requirement (such as the intelligent building systems), the solution provided this through the use of VXLAN (Virtual Extensible LAN), a Layer 2 overlay/virtualisation scheme where layer 2 traffic is encapsulated using MAC Address-in-User Datagram Protocol (MAC-in-UDP) for transport over the Layer 3 IP fabric.
The use of VXLAN ensured that layer 2 domains could be stretched across the campus wherever they needed to go in order to facilitate the requirements of some applications that demanded the same IP subnet or broadcast domain was available across all hosts they managed.
Finally, for hosts that required specific redundancy at the leaf layer, the solution provided this through the deployment of MC-LAG (Multi-Chassis Link Aggregation Group), allowing resilient dual-homing of hosts at the leaf layer where two physical Leaf switches would appear as a single logical switch.
PTS assisted Bloomberg with the deployment of a spine-leaf campus network with a layer 3 IP fabric at Bloomberg’s new UK HQ site which allowed Bloomberg to meet their requirements of a highly available, performant and scalable network architecture. The network was designed to allow the flexibility of creating multiple security domains through the use of virtualised networking which helped Bloomberg with segregating network traffic and enforcing security policy across their new site.
PTS assisted Bloomberg throughout the lifecycle of the project, including the design phase, procurement, and implementation management and helped to create a network that contained:
• Approx. 25,000 network ports for the Campus network
• Approx. 2,000 network ports for the Wi-Fi network
• Approx. 1,300 network ports for the out-of-band network
• Approx. 4,000 network ports for the Media control network
The overarching architecture provided a platform for Bloomberg to deploy various different and isolated virtual networks within each of the network areas defined above, for example providing the ability for corporate network traffic and building management traffic to co-exist on the same campus network switches (while maintaining logical isolation and segmentation from a security standpoint).
PTS ensured that Bloomberg’s requirements were met throughout the lifecycle of the project and helped ensure the delivery of a world-class, leading-edge campus network incorporating the latest in innovative network design philosophy, delivered both to schedule and within budget constraints.
• RIBA Sterling Prize
• AV Awards Corporate Enterprise project of the Year
• BCO Best of Best Award
• BCO Corporate Workplace of the year
• BCO National Award