Closing the Storage Gap. OpenStack Summit Vancouver 20150
OpenStack is one of the most exciting open source projects these days. Starting next week, I will attend several OpenStack events around the globe: the OpenStack Summit in Vancouver, the OpenStack CEE Day in Budapest, the OpenStack DACH Day in Berlin just to name a few. OpenStack Foundation invited me to present a very cool project for OpenStack: the Software-Defined Storage platform openATTIC and the closing of the storage gap.
What do I mean by “the storage gap”? Let´s start with software-defined storage. When you´re just about to run your own cloud, of course you want to use smart software-defined storage. Why? There are a lot of good reasons. One is that you want to have the elastic performances, the scalability that is way easier when applying an SDS approach. You also want to have multi-tenant service operations and you want to combine a lot of your storage approaches in one integrated system.
Benefits of smart SDS in cloud environments
The most crucial thing however are open APIs. If you look at legacy storage systems they usually do not have an API helping you to integrate the processes you need when deploying clouds to the storage. But OpenStack gives you the choice. There are a lot of APIs you can use like Cinder. This gives you the possibility to choose your underlying systems without having to accept vendor lock in. OpenStack gives you the freedom to decide if you want to go into legacy support or choose a commodity approach or an SDS approach (of course you can combine both).
Open source storage – the smarter choice
In terms of open source storage there are many tools available. I think that the real challenge is to pick the appropriate tools and combine them in a proper way so that they work together on an enterprise-grade level. The goal is to not have hassle with administrating these tools.
Data growth will be gigantic in the next couple of years. Analysts are talking about 40% to 50% growth rate per year. The question usually is: can your IT budgets cope with this growth? So you will find yourself wondering whether there are possibilities to make it cheaper than with your legacy approach. Yes, there are! Open source gives you the choice. With open source you have the needed technologies – and they are much cheaper.
No vendor lock-in
The next big advantage when you decide to go open source is that you have no vendor lock-in. If you look at the storage approach commercial vendors have, you will see that they drive huge margins by bundling things – and locking you in! This means that you have no choice about whether to buy an API, modules or functionalities – you just have to purchase all of them. Open source is different. It gives you the choice about which feature or module you need. And it’s not only about cost saving it’s also about adding a lot of functionality if you need it.
One of the things that are quite important is automation that is abstracting software from hardware. Open source gives you a lot of functionality which you can actually use with commodity hardware. This way, you can build it up on the software-defined level to achieve enterprise-grade storage.
Framework architecture: SDS & open source
If you look at this framework picture, you see on the left side that a lot of services and tools that are now available come from the field of open source. But if you want to use them on an enterprise-grade level there is still much to do because it´s a lot of work to combine these services and tools. And believe me, there are thousands more! You need to combine them in a way that everything works tightly integrated and smoothly. If you look at the red block of course you need a common API, too.
One the one hand, this API provides the entire graphic user interfaces. On the other hand, the API helps you with your cloud approach by generating all the functionality of underlying services and tools. If this is done in a smart way you can address them all via the API. That’s the major advantage of this approach.
Use cases for open SDS in an OpenStack environment
Now I would like to mention some use cases for an open software-defined storage approach in an OpenStack environment. There are basically four of them: First, I’m going to talk about how to manage your commodity storage nodes in an OpenStack environment. The next is automated provisioning of resources through an open API. The third one is how to provide your open storage infrastructure with a high-availability approach. And the fourth case is about monitoring your storage configuration without having too much manual work.
#1 Manage your commodity storage nodes with an intuitive GUI
If you use commodity storage it’s quite easy because you get cheap hardware, can collect a lot of add-on tools and can use linux as standard platform. But then you end up in a situation where you have to scale on a large level and want to apply thousands or ten thousands of servers but lack petabytes for storage. This means that there is a lot of work to do from the administrator’s perspective.
So, you need something what we call a storage operating system. It is way more than a pure linux platform and helps you performing these tasks. At the top you see some screen charts showing things we have already done to facilitate the job (I will get back later on this).
What you really need is a single point of administration for all these tools. When you are working with 30, 40 or even 50 different tools on the underlying platform you need software that helps you to administrate everything easily. Then you have the choice to compete all the legacy vendors and providing the nice features you need and you are used to when you run your systems with the major solutions.
#2 Automate provisioning of resources through an open API not covered by Cinder API
The second one is about the automation of provisioning processes. Take OpenStack’s Cinder module. At the Cinder-API itself there is not much functionality. You can create a volume or a snap-shot but that is basically it.
If you look at the storage you use and its functionality, you see a lot of things you need to have when you run an enterprise-grade cloud providing mirroring, high availability, replication and consistent snap shots of applications like data bases. However, they are currently not supported by Cinder. This means that you have to find other ways. If you use Cinder on the volume control to get the target there is still a lot of work to do like high availability, mirroring, clustering – all these things are not done by Cinder.
So you need one more thing to take care of this. You do not want to do it manually so you think of automating it. Again, an open API is very important here. If you have an open API in your storage system it helps you to implement these processes.
#3 Provide your storage infrastructure with built-in high availability and redundancy
I have already talked about high availability. If you look at Cinder in its native form you don`t see much of high availability. Of course it depends on the type of cellar for your cloud. But high availability can be important even on the storage level if you own an enterprise-grade cloud.
High availability as you might know depends on your set-up, your policy, the know-how of your employees and your distribution. If you want to go on a local or just a metro high availability scenario that you do synchronous or asynchronous of course you have the amount of cluster-nodes that really help you to pick up the right tools for this.
If you look at the linux platform scenario there are a lot of tools available like DRBD, Corosync or Pacemaker. These tools bring you a lot of value in high availability environments. There is one problem, however: you need to configure each of them individually. Of course, this is not what you want to do when scaling up to a large scenario.
#4 Monitor instantly every storage configuration item without manual tasks
The next thing when running large clouds is reliable storage monitoring. I bet you want to watch the volumes filling up and see if there are problems around that. And you surely want to take a look at them.
If you take OpenStack and Cinder you see that there is no automated way to do that right now. Also, you need to add the functionality as well. As we have seen, many tools come from the field of linux or open source. For example, there is Nagios that helps you to monitor tools and processes. If you just want to look at the available open source frameworks you need to combine a lot of things to really reach an enterprise-grade level (for example for service-level management and reporting).
openATTIC as Software-Defined Storage Platform
So, what does the storage platform openATTIC have to do with this? Well, what we basically have done so far with openATTIC is to combine different open source tools into one framework that is open itself. The resulting software-defined storage platform helps you to build enterprise-grade functionality for different cloud platforms on 100 % open source.
The openATTIC platform provides unified storage for the unified storage approaches you already have. It supports different protocols and practically every standard, a smart software-defined storage layer which is fully accessible by an open API, monitoring, high availability application, snapshotting, and cloud storage with a specific API and object functionality. There is a community as well as an enterprise version available. We also offer services on top of that in case you need more.
As you have seen, Cinder currently does not offer enough to run enterprise-grade scenarios. The good news is that there are a lot of things that you can add on top of an existing open source framework. You just need to do it in a very clever way. If you combine the different tools you will be able to provide a high-end software-defined storage platform on top of open source that works perfectly for cloud set-ups and OpenStack. And this has been done already in our openATTIC.
Take a look at openATTIC at openattic.org. I hope you like it and maybe feel inspired to contribute to the project. There are a lot of things you can add, a lot of things which you can modify for your own project. But of course communities can only live if there are a lot of people contributing so that´s why I would be quite happy if you do so. If you need service, support or consulting visit www.openATTIC.com where you find the commercial version.
The Storage Gap at upcoming OpenStack events
If you want to know more about openATTIC and it can help you to better run your OpenStack visit my talks at the next OpenStack events:
– OpenStack Summit Vancouver, May 18-22
– vBrownBag TechTalks at OpenStack Summit Vancouver, May 20
– OpenStack CEE Day, June 8 in Budapest
– OpenStack DACH Day, June 11 in Berlin
Found this article interesting? Maybe you´ll like one of these, too: