Cormant has been tracking the Interop NET’s physical infrastructure for over 5 years providing full documentation of all equipment, cabled connections, power usage, and more.
Cables and cable management can be fairly one-dimensional. Together they provide an organized system of “pipes” for data to flow through and are limited only by a few, simple technical factors. There are, however, some environments which provide tough and interesting trials. Under the strain of rapid change or diverse requirements, these environments test the strength of any technology product which steps up to the challenge. The InteropNET (Interop’s publicly accessible, ISP grade network) is possibly the greatest of them.
Cormant has been tracking the InteropNET’s physical infrastructure for over 5 years providing full documentation of all equipment, cabled connections, power usage, and more. In a normal data center the process for capturing that type of information is controlled and driven by strict process. The InteropNET is anything but. Racking, cabling, configuring, testing, and deploying everything happens over the course of 4 total weeks. Two are spent staging in a warehouse and two others are spent in Las Vegas setting all the pieces in place, with the smallest show encompassing 24 racks, 356 pieces of equipment, and over 2,000 connections! Many aspects of the network will change continuously throughout those weeks and proper documentation helps everyone understand the exact changes.
InteropNET’s Hot Stage
The initial warehouse set up stage, better known as “Hot Stage,” is always full of vigor and new beginnings. Vendors, who supply all equipment and significant labor, arrive on site to attend to their part of the network. In the two week timeframe, all equipment will be racked, cabled, and configured. At this stage Cormant’s role is somewhat like being the “technology janitor” – we follow technicians from rack to rack documenting changes made along a number of data points. This includes equipment make/model, size, location, orientation, available ports, and associated cabled connections. We also document every connection between equipment and CDUs providing a full understanding of dependency on each CDU. These records are initially used for power testing, power tracking, inter-rack connection tracking, and various capacities. Simultaneously, in Cormant-CS we document all exhibiting vendors’ pre-ordered data connections. This gives the show information on the number of cables needed for the 360,000 square foot convention facility. At the end of two weeks, almost as quickly as it began, all equipment is powered off and the racks are packed into trucks. Click to read a new Cormant-CS user’s account of documenting IT infrastructure at Hot Stage.
Interop Expo
With renewed energy from a short break, all participants plus a crew of volunteers arrive at the Las Vegas show floor to put the pieces back together. Racks are carefully wheeled off the trucks to their final locations. Some are sent off to the expo show floor, others to the conference spaces, and a final few to the NOC. Cormant-CS’s documentation is extremely valuable during this stage because it holds detailed, point-to-point information on how the racks need to be reconnected. Having this record set saves valuable troubleshooting time by restoring the physical connections to their exact previous positions. The task could take two to three times as long without this documentation. Once the racks are reconnected electricians begin running patch panels with pre-terminated horizontal cables which are placed in the ceiling high above. From these ceiling-mounted patch panels, cables are dropped into each booth (or group of booths) and connected back to a distribution rack (similar to an IDF).
at http://motobayashi.net
Now the real work begins. Trading “technology janitor” hats for “technology plumber” hats, Cormant technicians and a handful of volunteers move from booth to booth to booth documenting and testing the data “pipes” along the way. At each stop the full path, from drop cable to NOC, is verified, recorded, and tested. Results from three cable tests (physical, logical, and WAN) are documented as well. An established process is followed during documentation and testing to keep results consistently accurate. In the end, each cable’s pathway and condition is known from the core router to distribution switch to patch panel to booth, highly important for determining the source of issues and making quick changes once the show is underway.
Executing this work on over 600 vendor booths can be quite a challenge. There are multiple stages where things could go wrong, but don’t thanks to Cormant-CS’s design. First, consider the workforce. Most of the individuals carrying out the work are volunteers who have approximately one hour of training before heading into the field. A consistent, easy-to-use interface helps make the most of that hour. Second, consider the infrastructure. Most cabling equipment, especially passive pieces, were crowded into wood crates for transportation. While care is taken to avoid or minimize damage, it is inevitable – cables get crimped, ports and port pins get crushed, and leads get torn from punch downs. These issues require accurate tracking so electricians get enough detail to re-run cables successfully then notify the Cormant team to retest. Third and last (but definitely not least!), consider tracking passive infrastructure you cannot reach or even see. A lot of setup and care goes into documenting and tracing all patched and horizontal connections which get suspended in the ceiling.
Mere hours after setup ends, the Interop expo floor opens. Vendors and staff flood in and start consuming massive amounts of data. This is when all the work put into Cormant-CS pays off. If any user has an issue on the show floor our software can provide end-to-end physical routes, test results, and a full work/action history. Volunteers manning the Interop Help Desk can easily look up any of this information using our web interface. Technicians armed with a full understanding of the cable pathway are dispatched to fix issues all over the show.
Conclusion
Cormant-CS’s value is proven year after year during Interop. Our records are instantly available to troubleshoot all reported problems and maintain solid uptime during the show. Whether executing moves, adds, and changes in a data center, or rebuilding an entire mobile network like InteropNET’s, downtime is downtime and infrastructure must be properly documented for time-saving, efficient processes in any environment.
Want more? View our InteropNet Case Study
By: Stuart Hallin
Senior Technical Consultant