The world of open source has been a goldmine for organisations looking to create innovative new software and services. The ethos behind it has extended into the world of networking and management and is exemplified by vendors such as Nagios, standards like OpenStack and vSwitch technologies. Remote network and out of band management has also been boosted by open source by promoting interoperability and innovative solutions.
Open source fits neatly with the notion of self-sufficiency. In fact, some IT managers effectively build their own remote management solutions using often recycled Cisco routers with WIC cards to build out basic remote management. Some have taken things even further and extended this “roll your own” or DIY philosophy to custom embedded application.
Some use generic embedded computer boards, open source and additional components in an attempt to bridge the networking world to physical infrastructure. Although technically possible, there are a number of issues that should be considered before trying to build a Frankenstein out-of-band remote management platform.
Building A Remote Console Server
Firstly, it’s not that easy to find a system board with all the right interfaces and connectors, DIMMs, flash sockets, cable headers and so forth, all in optimum positions. Then there’s the tendency nowadays to fit missing interfaces via a USB or other dongle. Already it’s looking like a high-school project. Next you need to find an enclosure which may have connectors and power in poor locations and leak RF. You’ll also need to find an approved power supply.
The potential for significant pain now begins. Multiple rounds of expensive regulatory compliance testing (EMC, safety, and carrier) may be needed for your combination of components. You run the real risk that you may not be able to deploy the final solution due to design issues with components which you don’t control and cannot change. In some cases, third party data centre owners will not allow an uncertified item into a facility.
Beyond that you need to select and tune the embedded OS, storage model, additional subsystems, device drivers and finally applications, to create a firmware image. You have to stress test both the hardware and firmware in real-world environments.
The appliance needs to be ultra-reliable for critical installations that must stay operational when core managed devices are down. This is rarely the case with “roll-your-own” appliances which may not be equipped with watchdog circuitry and support firmware. You also need to monitor the embedded OS for security and other significant patches.
An easier method is to reuse an existing router as a base platform but this also has its own issues. We work with organisations every week that are decommissioning legacy Cisco routers with WIC cards in favour of dedicated Infrastructure Management appliances. Some in part due to the fact this Cisco solution is end of life and has no future support. Others push this old equipment out the door to meet security standards outlined by NIST for FIPS 140-2 compliance.
The FIPS 140-2 standard is an information technology security accreditation program for cryptographic modules produced by private sector vendors who seek to have their products certified for use in government departments and regulated industries such as financial and health-care institutions that collect, store, transfer, share and disseminate sensitive but unclassified (SBU) information.
Organisations that transmit or share data from these industries will endure IT audits to ensure FIPS 140-2 compliance. Gaining compliance for a home brew device is almost impossible without a huge and on-going investment in development and compliance testing.
Adding Out Of Band Connectivity
Another issue is the connectivity between the appliance and a viable out-of-band network. The logical method is wireless and the 3G and 4G LTE cellular network is proving a more compelling alternative to PSTN and DSL for out-of-band access for remote network provisioning, maintenance and repair. From our own sales data, this is becoming a clear preference for customers as for over a year our cellular-enabled remote management solutions had surpassed dial-up.
Reasons for this include speed of provisioning and ease of deployment, which is down to the mobile nature of the solution. But it’s also about the bottom line – in the era of tablets and smartphones, mobile data has never been cheaper. Mobile networks also fit neatly into the roll your own methodology but unfortunately adding 3G access to either a repurposed router or a home brew device is not a simple task.
Either way, adding cellular connectivity should follow a few best practice guidelines. One is the notion of using the right type of SIM as all these modules and associated price plans are not equal. For example, a Machine to Machine (M2M) SIM is a type of service offered by carriers that is designed for data applications and not voice calls. M2M SIM’s are commonly used in tracking applications and digital signage but remote infrastructure management is a rapidly growing area.
However, M2M SIMs may not be offered by your preferred carrier, or the incremental pricing model may not make sense for your usage pattern – particularly where an always-up cellular connection is preferred, activities like continuous Nagios or SNMP monitoring can push data use into the 100s of MB per month.
On the other hand, commodity “SIM only” or “bring your own device” plans are readily available with generous blocks of data for as little as £5/month in the UK. With the boom of unlocked Nexus and i-devices, the USA which is the last hold-out of the contract locked device is rapidly catching on too.
These dirt cheap SIMs are suitable for remote management, as long as you’re aware of a couple of caveats. The first is that you’ll probably be NATed by the carrier. That is, your remote management appliance won’t be assigned a public, routable IP address that you can browse or SSH to for remote access. This is not a big issue as you can contact your carrier and request an APN that assigns a routable IP address be activated for your SIM. This may require moving to a “business grade” service and cost a bit more.
Also, If your SIM has a public IP address then you need to ensure that your bought or built solution has a stateful firewall that can lock down remote access to trusted source IP address ranges, and operates a default-deny policy for any WAN facing, unencrypted services. As with any public service, always use strong passwords, or consider disabling password authentication entirely and using SSH key auth instead.
Solving Top Of Rack Issues
Another consideration in a buy and build discussion is how to deal with the increasingly common data centre designs that include “Top-of-Rack” switches to terminate fibre links to the rack and deliver Infiniband and 10 GigE links locally. This solution keeps the bulky copper cabling inside the rack and ensures the infrastructure is future proofed, sustaining transitions to 40G and 100G in the future. Terminating fibre connections only at the rack can cause a complete lack of visibility if the problem is the top-of-rack switch.
A neat solution is deploying a console management solution to gain remote visibility into the rack during an outage. This implementation requires a copper link that terminates at a core switch delivering a PoE connection to the PoE powered appliance. This provides a cost effective way to gain complete visibility into racks that utilise top-of-rack designs and extend both Ethernet and serial console connectivity without relying on power from the rack.
Not utilising the console port on a top of rack switch is a gamble, considering the value it provides with instant triage and problem remediation within seconds of a network issue. Implementing smart console management solutions ensures complete visibility and remote access when it is needed most.
The above “Top of Rack” example provides a great example of why the build versus buy argument is significantly swayed in favour of prebuilt but “Open” remote management solutions are starting to become dominant in the marketplace. The combination of purpose built hardware, software, 3G/4G connectivity and flexible port combinations are hard to match with a roll your own solution.
Throw in FIPS certification and continual software updates for operating systems and emerging standards and the long term value of a bought appliance for organisations that require mission critical operation are clear. For organisation that want to try to build their own then it is still a technically viable option but the costs across the short and longer terms are higher and while the on-going level of risk is higher.