How to Avoid Your Next 12-Month Science Project

While most customers immediately understand how the magic of Oracle’s Hybrid Columnar Compression, intelligent storage servers and flash memory make Exadata uniquely powerful against home-grown database systems, some people think that Exalogic is nothing more than a bunch of x86 servers, a storage appliance and an InfiniBand (IB) network, built into a single rack.

After all, isn’t this exactly what the High Performance Computing (HPC) world has been doing for decades?

On the surface, this may be true. And some people tried exactly that: They tried to put together their own version of Exalogic, but then they discover there’s a lot more to building a system than buying hardware and assembling it together. IT is not Ikea.

Why is that so? Could it be there’s more going on behind the scenes than merely putting together a bunch of servers, a storage array and an InfiniBand network into a rack? Let’s explore some of the special sauce that makes Exalogic unique and un-copyable, so you can save yourself from your next 6- to 12-month science project that distracts you from doing real work that adds value to your company.

Engineering Systems is Hard Work!

The backbone of Exalogic is its InfiniBand network: 4 times better bandwidth than even 10 Gigabit Ethernet, and only about a tenth of its latency. What a potential for increased scalability and throughput across the middleware and database layers!

But InfiniBand is a beast that needs to be tamed: It is true that Exalogic uses a standard, open-source Open Fabrics Enterprise Distribution (OFED) InfiniBand driver stack. Unfortunately, this software has been developed by the HPC community with fastest speed in mind (which is good) but, despite the name, not many other enterprise-class requirements are included (which is less good).

Here are some of the improvements that Oracle’s InfiniBand development team had to add to the OFED stack to make it enterprise-ready, simply because typical HPC users didn’t have the need to implement them:

• More than 100 bug fixes in the pieces that were not related to the Message Passing Interface Protocol (MPI), which is the protocol that HPC users use most of the time, but which is less useful in the enterprise.

• Performance optimizations and tuning across the whole IB stack: From Switches, Host Channel Adapters (HCAs) and drivers to low-level protocols, middleware and applications. Yes, even the standard HPC IB stack could be improved in terms of performance.

• Ethernet over IB (EoIB): Exalogic uses InfiniBand internally to reach high performance, but it needs to play nicely with datacenters around it. That’s why Oracle added Ethernet over InfiniBand technology to it that allows for creating many virtual 10GBE adapters inside Exalogic’s nodes that are aggregated and connected to Exalogic’s IB gateway switches. While this is an open standard, it’s up to the vendor to implement it. In this case, Oracle integrated the EoIB stack with Oracle’s own IB to 10GBE gateway switches, and made it fully virtualized from the beginning. This means that Exalogic customers can completely rewire their server infrastructure inside the rack without having to physically pull or plug a single cable - a must-have for every cloud deployment. Anybody who wants to match this level of integration would need to add an InfiniBand switch development team to their project. Or just buy Oracle’s gateway switches, which are conveniently shipped with a whole server infrastructure attached!

• IPv6 support for InfiniBand’s Sockets Direct Protocol (SDP), Reliable Datagram Sockets (RDS), TCP/IP over IB (IPoIB) and EoIB protocols. Because no IPv6 = not very enterprise-class.

• HA capability for SDP. High Availability is not a big requirement for HPC, but for enterprise-class application servers it is. Every node in Exalogic’s InfiniBand network is connected twice for redundancy. If any cable or port or HCA fails, there’s always a replacement link ready to take over. This requires extra magic at the protocol level to work. So in addition to Weblogic’s failover capabilities, Oracle implemented IB automatic path migration at the SDP level to avoid unnecessary failover operations at the middleware level.

• Security, for example spoof-protection. Another feature that is less important for traditional users of InfiniBand, but very important for enterprise customers.

• InfiniBand Partitioning and Quality-of-Service (QoS): One of the first questions we get from customers about Exalogic is: “How can we implement multi-tenancy?” The answer is to partition your IB network, which effectively creates many networks that work independently and that are protected at the lowest networking layer possible. In addition to that, QoS allows administrators to prioritize traffic flow in multi-tenancy environments so they can keep their service levels where it matters most.

• Resilient IB Fabric Management: InfiniBand is a self-managing network, so a lot of the magic lies in coming up with the right topology and in teaching the subnet manager how to properly discover and manage the network. Oracle’s Infiniband switches come with pre-integrated, highly available fabric management with seamless integration into Oracle Enterprise Manager Ops Center.

In short: Oracle elevated the OFED InfiniBand stack into an enterprise-class networking infrastructure. Many years and multiple teams of manpower went into the above improvements - this is something you can only get from Oracle, because no other InfiniBand vendor can give you these features across the whole stack!

Exabus: Because it’s not About the Size of Your Network, it’s How You Use it!

So let’s assume that you somehow were able to get your hands on an enterprise-class IB driver stack. Or maybe you don’t care and are just happy with the standard OFED one? Anyway, the next step is to actually leverage that InfiniBand performance. Here are the choices:

1. Use traditional TCP/IP on top of the InfiniBand stack,

2. Develop your own integration between your middleware and the lower-level (but faster) InfiniBand protocols.

While more bandwidth is always a good thing, it’s actually the low latency that enables superior performance for your applications when running on any networking infrastructure: The lower the latency, the faster the response travels through the network and the more transactions you can close per second.

The reason why InfiniBand is such a low latency technology is that it gets rid of most if not all of your traditional networking protocol stack: Data is literally beamed from one region of RAM in one server into another region of RAM in another server with no kernel/drivers/UDP/TCP or other networking stack overhead involved!

Which makes option 1 a no-go: Adding TCP/IP on top of InfiniBand is like adding training wheels to your racing bike. It may be ok in the beginning and for development, but it’s not quite the performance IB was meant to deliver.

Which only leaves option 2: Integrating your middleware with fast, low-level InfiniBand protocols. And this is what Exalogic’s “Exabus” technology is all about. Here are a few Exabus features that help applications leverage the performance of InfiniBand in Exalogic:

• RDMA and SDP integration at the JDBC driver level (SDP), for Oracle Weblogic (SDP), Oracle Coherence (RDMA), Oracle Tuxedo (RDMA) and the new Oracle Traffic Director (RDMA) on Exalogic. Using these protocols, middleware can communicate a lot faster with each other and the Oracle database than by using standard networking protocols,

• Seamless Integration of Ethernet over InfiniBand from Exalogic’s Gateway switches into the OS,

• Oracle Weblogic optimizations for handling massive amounts of parallel transactions. Because if you have an 8-lane Autobahn, you also need to improve your ramps so you can feed it with many cars in parallel.

• Integration of Weblogic with Oracle Exadata for faster performance, optimized session management and failover.

As you see, “Exabus” is Oracle’s word for describing all the InfiniBand enhancements Oracle put into Exalogic: OFED stack enhancements, protocols for faster IB access, and InfiniBand support and optimizations at the virtualization and middleware level. All working together to deliver the full potential of InfiniBand performance.

Who else has 100% control over their middleware so they can develop their own low-level protocol integration with InfiniBand? Even if you take an open source approach, you’re looking at years of development work to create, test and support a whole new networking technology in your middleware!

The Extras: Less Hassle, More Productivity, Faster Time to Market

And then there are the other advantages of Engineered Systems that are true for Exalogic the same as they are for every other Engineered System:

• One simple purchasing process: No headaches due to endless RFPs and no “Will X work with Y?” uncertainties.

• Everything has been engineered together: All kinds of bugs and problems have been already fixed at the design level that would have only manifested themselves after you have built the system from scratch.

• Everything is built, tested and integrated at the factory level. Less integration pain for you, faster time to market.

• Every Exalogic machine world-wide is identical to Oracle’s own machines in the lab: Instant replication of any problems you may encounter, faster time to resolution.

• Simplified patching, management and operations.

• One throat to choke: Imagine finger-pointing hell for systems that have been put together using several different vendors. Oracle’s Engineered Systems have a single phone number that customers can call to get their problems solved.

For more business-centric values, read The Business Value of Engineered Systems.

Conclusion: Buy Exalogic, or get ready for a 6-12 Month Science Project

And here’s the reason why it’s not easy to “build your own Exalogic”: There’s a lot of work required to make such a system fly. In fact, anybody who is starting to “just put together a bunch of servers and an InfiniBand network” is really looking at a 6-12 month science project.

And the outcome is likely to not be very enterprise-class. And it won’t have Exalogic’s performance either.

Because building an Engineered System is literally rocket science: It takes a lot of time, effort, resources and many iterations of design/test/analyze/fix to build such a system. That’s why InfiniBand has been reserved for HPC scientists for such a long time.

And only Oracle can bring the power of InfiniBand in an enterprise-class, ready-to use, pre-integrated version to customers, without the develop/integrate/support pain.

For more details, check the new Exalogic overview white paper (no link, page no longer exists) which was updated only recently.

P.S.: Thanks to my colleagues Ola, Paul, Don and Andy for helping me put together this article!