BlueXP is now NetApp Console
Monitor and run hybrid cloud data services
Thanks for uh thanks for coming. So yeah, as we said, we're talking about uh Postgress on Azure NetApp files. So I work for InstaCluster, which was bought by NetApp about 18 months ago. Uh we run open- source databases, streaming solutions uh and workflow engines in the cloud on your premises, wherever you want them run, fully managed. So we'll kick off with the confidentiality notice that I'm sure you've seen 20 times now. Says don't take it outside of NetApp. says that product managers road maps are more akin to dreams than forward-looking financial statements that you can rely on. So anything we talk about in the road map here doesn't have any specific dates on it. So let's set the stage. Postgress SQL running in the cloud can be a complicated beast to wrangle uh and often doesn't keep up with your on-prem version. So if you're used to running Postgress in your own data center, uh you sometimes can't get the performance if you try to move that straight to the cloud. Using Insta Instacluster Postgress on ANF, you've got a fully managed solution uh that will keep up with any on-prem environment and give you really quite blazing performance. As we go through, we're going to talk about how to create a cluster through the install, how to connect to it, my favorite bit, which is the benchmark results, uh and a bit of the road map that we're going to look at going forward. So what is Instacluster? So just a little bit of familiarity for those who haven't heard of us since the uh since the acquisition. So we run open source databases in the cloud. They're fully open source. There is no lock in. We're not going to trick you with a little bit of open core so you have to stay with us forever. We're going to give you the best service in the business so that you want to stay with us forever. Uh we're used to design designing and solving all of the complexities of uh running these open source systems either in on premises or in the cloud. Uh it's fully supported. You've got a 24/7 helpline. You've got monitoring, you've got backups, all the things that you need to make your software work for you and uh reliably. Uh and we've got probably more experience than most in the business. We've got over 100 million node hours that we've run, over seven pabytes of data that is currently under management. Uh and we even have open source committers to a lot of these projects. So we're really deep in the guts of anything that we run. We know it back to front.So let's look at how we would create a Postgress cluster. So this is I was going to do a live demo. They said in Vegas Wi-Fi, that's a terrible idea. So, here's some screenshots of what would be a live demo. So, you're welcome. You log in. This is the InstaCluster console. Uh you can do this with a free trial if you want to go through these steps your own. Uh once you've got good Wi-Fi, uh hit the create cluster button and you're given all the options. This is all the different products we run. So, Cassandra, CFKA, Open Search, Reddus, uh you know, just about anything you could need for abig data workload. But obviously, we're going to talk about Postgress today.here. today here. You can run it on any of your major cloud providers, AWS, Azure or GCP or we can wire it into your data center and we can deploy it directly into your data center. You tell us how to make VMs and where you want the backup stored, we can create it directly for you exactly where you already live. Uh there's a few options down there. You can run private network cluster, you can run it in PCI compliance mode if uh if that level of security is important to you. And then you get to your Postgress options. So once you've got Postgress, we support all the latest versions. That's 154 there. 16 should be coming out any day now on the platform. Uh for a test cluster, you may want to add your IP directly to uh the directly to the firewall so that you can connect to it as soon as it's spun up. Uh choose your replication mode. So depending on how strict your data uh replication requirements are, synchronous, asynchronous, synchronous mode, strict, you can add nodes. Uh so you've got more and more layers of backup. And then you can uh chuck some enterprise add-ons in there. So at the moment we've got PG bouncer. So that's a connection a thread pooling or a connection pooling add-on so that you can connect manyclients to your Postgress. Usually it starts to suffer over a certain number. Then we choose where we want to put it. So we're going to just chuck it in central US. Uh with a data center it's a two node. We recommend two nodes as a minimum for Postgress. So you've got redundancy. Should your primary node fail, you can fail over to a secondary. If you need greater redundancy than that, you can put three, four, five, as many as you're willing to pay for kind of thing. Um, and we've chosen the node size. So, we've got node sizes dedicated to ANF. So, that's what we're choosing in the middle there is that these ANF nodes are backed by ANF storage, which gives us the results that I'm about to show you. Once you've done that, you get a last little confirmation. You hit I agree to the terms and conditions. And about 10 minutes later, before I could even finish this presentation, we would have an operational Postgress cluster uh that we could connect to. So, you can do that with Amazon RDS. I'm sure you can do that in many other places. The InstaCluster advantage is thelevel of support that you receive. So we've wired these up with all the monitoring you could possibly want. So we do synthetic reads. We do synthetic rights. Our SLAs's guarantee read write access, not just that the server will be up and something may be talking there. Um all of this is consumable through a Prometheus API if that if you want to bring it into your single pane of glass or you log into our console. There's, you know, dozens and dozens of different metrics that you could look through in there. We're also monitoring them with our support team. So our support team's getting alerts if anything's looking wrong. If your ride ahead logs filling up. If you're getting delays between the primary and the uh secondary, we are monitoring that 24 by7. We can go in there, we can fix it. Sometimes before you even notice, you'll just get a nice email saying one of your nodes fell over. Don't worry, we replaced it. All the data is safe. Nothing has gone wrong. Uh and if you discover something's wrong, we're wearing a Zenes ticket or an email away. Uh ready to support you 24x7, 365 days a year. Very easy to connect. Once again, in my live demo that I was going to do, uh, we just give you the connection info. So, you get a user to log in. We give you some connection strings if you want to connect to C++ with Java directly through the command line. Just copy and paste down there straight into a console. Uh, and you've got yourself a Postgress server and then you can just start running SQL commands, same as any Postgress server you could want. Um, and it all works. It's all ready within 10 minutes. So the exciting bit of this talk is the results that we're getting out of running ANF uh instead of standard sort of cloud commodity hardware storage. So it took a little while to get these results but you can see here the bottom two lines are ourtraditional offering uh versus a competitor offering and sort of unsurprisingly those lines are basically on top of that each other. When you run open source software on commodity hardware you tend to get the prettiest result pretty much the same results. You can look in the rewrite test, the mind-blowing result up there, 4.5,000 transactions per second. So, this is the same core count, the same memory count, the same everything. All we've changed is the underlying storage and we are over doubling our readwrite transactions. So, this took a bit of fiddling like when we first built this product, we sort of saturated clients equals cores and there's some tricks to getting Postgress to work nice with ANF. There's tricks for getting ANF to work nice with Postgress. So, we did a bit of tuning in the background and like we almost couldn't believe it when we started getting these results. So I had to get theteam to run them again cuz we got such like slightly disappointing results in the early days and such amazing results were before we were ready to publish that it was too good. So it gets even better. We look at a readon workload. So if you've got a really readheavy database where most of what you're doing is grabbing data looking up customer orders etc. Uh we have over quadrupled the speed that we can get out of again standard Postgress fully open-source hardware in the cloud just by backing it with ANF. uh this is the kind of results we can get. So that is it's a bit washed out there but it is 17 and a half thousand trans transactions per second which is pretty great. Not great enough. We had to go back one more time. Uh and that's this is the same graph again those two lower lines and then we decided you know what's the most amount of uh what's the most amount of George Curran's money we can spend on extremely expensive hardware. So we got a 32 core ultra machine. We backed it with ANF ultra storage. This isn't generally available. This is sort of something we built privately for ourselves. We got 74,000 transactions per second. The only time I've ever seen a Postgress do better than that was at about 80,000 transactions per second. It was running 96 cores on a bare metal machine with custom drivers installed. So getting this results at the click of a button. Uh this is why we're pretty excited and this is why I'm on stage talking to you here.What does that translate to? So if you look at the dollar per transactions per second, this is 75% cheaper than what you could get with a competitor offering. So because you get so much more speed, you can afford to downscale your instance for a given workload uh and save considerable money by matching your storage and your compute to your workload. So this is what we can deliver uh just with the basic ANF features of it's reallyfast. But we're not done. Anyone who is working, you know, anyone who's here at NetApp, I hope knows that NetApp Storage does a whole lot more than just be very fast. So, what we're looking to do is take a lot of advantage of the snap mirror uh and snapshot functionality in ANF. And this is what our road map looks like. So, we're looking for ultraast backups. At the moment with PG back rest, it can take literally hours to generate a backup using the standard Postgress method of doing a backup. And during that time, one of your cores is being chewed up by creating one of the backups, shipping it off site, slowing down your workload. Using al using ANF snapshots, we can just take a snapshot every 24 hours or every 12 hours or however often you need your backups taken, ship it off with snap mirror and that data is safe and protected and your Postgress workload barely even knows anything that's happened. Should the worst happen and your data get base gets encrypted or you drop the wrong table or just the hardware fails, we can bring that back up in the same times it create takes to create a new server 10 minutes plug in the old storage away we go. Database forking is one that's pretty exciting to me. So I was a data scientist in my past life and one of the things I would always love to do is like can I just get credentials to themain database that runs our whole company and can I just like run some queries and the SIS admins would look at me like I'm crazy. They're like no obviously we're not going to give you access to our core database. you will blow it up immediately. I'm like, yeah, it probably would blow it up really immediately. What you can do and what we're going to make a button on the console do you click a button, you get a live fork. So, we'll take a snapshot, we'll pull it over here, we'll bring up a new Postgress instance, point it at that data, and then it's completely isolated. It doesn't touch the old database at all. You can give it to your data scientists, you can give it to your analysts, you could put it in your CI/CD pipeline to do tricky tests if you need to test on live data. Uh, and then that data scientist can accidentally drop tables to their heart's content. uh and no one inuh transaction world needs to be any the wiser. You could even do this daily, right? We can do this all via API. So you could at midnight destroy that server, create a new one from a fresh snapshot, away we go. Your data scientists wake up in the morning and they've got a fresh copy of the data no more than an hour or two stale and they can look at exactly what's going on live, play with it to their heart's content, run massive queries at it that would, you know, that are going to bring Postgress to its knees uh and still not impact your production workload whatsoever. So that one I'm very excited about delivering. We're also looking at a storage level data recover disaster recovery. So Postgress typically if you want to do disaster recovery you'll chuck another Postgress server off in another rack somewhere and the two Postgress servers will talk to each other. But then obviously you need compute you need storage you need in networking to connect. You need allof the stuff that goes with the Postgress server. With the snap mirror functionality we can just mirror the storage across to a new region. And then when the disaster happens you call our tech support. You say look the whole region's fallen over. We've lost everything because I don't know if someone unplugged thepower to our data center. We've got a copy up in the cloud. 10 minutes we plug the Postgress into it, put it into that data store and you're back up and running, you know, veryquickly. It's not the highest resiliency datadisaster recovery, but it's much cheaper than running, you know, redundant compute instance in another region that you'd barely use 99.9% of the time and just needs that disaster recovery. So, all of this is on our road map. All of this is pretty exciting. Uh, and all of this is enabled by ANF, which is, you know, why we're all here at NetUP. So, give you one last takeaways. It's very easy to create one of these. You can do it via the console as we walk through. You can do it via our API or we've got a Terraform client. Simple to pull into your existing workload. We can run it on uh yeah, so we run it in Azure. It's a 75% reduction in cost per transaction per second. So, it's very valuable choosing the right storage for your compute. Uh, and the road map going forward on this product is very exciting. So, I'm looking forward to getting home and getting stuck back into that related sessions. We've already missed some of these, I'm afraid. Ben here gave the uh leveraging ONAP for open source. It was great yesterday. We've got my SQL with more for a lot less. Uh, and we've got an open source database workshop. If you're interested in any of the other Instacluster products or how that all works, uh, one of our members here can take you through that. We've also got a booth just over there uh, with Instacluster. So, if you want to come and chat to us about any more detail, uh, we're there all week. stay connected. Hopefully, you're on the Discord or Facebook or Twitter already. Uh, and if you want to contact me, I'm Chris Carter, product manager for Postgress. My email's just there. Drop us a line anytime you like. Thank you.
Instaclustr for PostgreSQL® on Azure NetApp Files gives customers the benefit of blazingly fast IOPS of Azure NetApp Files storage to drive the highest possible levels of PostgreSQL performance—backed with the rock-solid reliability and [...]