(and 10 things you can do now to control storage costs)
But it’s challenging to examine and really comprehend how this data growth applies to you, your job, and your budget. To start with, 60% of that data will be created by organizations like yours, and at least half of all that data is expected to reside in the public cloud.
Cloud consumption comes at a cost, and sometimes that cost shows up, unexpected and uninvited, at the bottom of your cloud bill. In this guide, we’re chiefly focusing on the public cloud. But we’re ultimately concerned with your infrastructure as a whole and the variables you need to examine closely to control escalating cloud storage costs.
Often, your data architecture is to blame for cost overruns. But with a little more strategic planning, you can easily tackle what might feel like an overwhelming situation. By reading this step-by-step guide, you’re already on the right path.
Ready? Let’s talk about the five warning signs that you may have a cloud spending problem. Then we’ll give you what you really came for—ten ways you can address out-of-control cloud costs right now.
Long answer long:
Before the cloud, your organization enjoyed a world in which capital expenditures (capex) bought ample storage that existed within a fixed data center ecosystem. Multiple data types coexisted on similar hardware, making the allocation of storage—regardless of whether you really needed it or not—a reasonably simple task. The IOPS were glorious and abundant; the security, airtight like a Vegas casino. You probably had a team of storage administrators on the payroll to attend to your every storage whim.
But then something broke down, and the corridors of bureaucracy and its cascade of red tape hindered quick recovery. You dreamed of the day when your storage would work harder for you, like an overeager intern poring over a stack of spreadsheets.
Let's talk about the five warning signs that you may have a cloud spending problem, and then we'll give you what you really came for—10 ways you can immediately address out-of-control cloud costs, right now. Like, today.
Then your dream became some version of reality—you made it to the cloud. And today, you’ve exchanged the complexity of data center management for an easy monthly operating expense (opex) that offers virtually infinite scalability and flexibility. Sounds great, right? But as any astute accounting major can attest, that opex can sneak up on you at the end of the month, when your profit margin resembles a slowly disappearing oasis. Paying for storage is now based on multiple factors that involve separate storage types, tiers, protocols, and costs.
Inevitably, unexpected costs will bite you in your bottom line. The culprit? Any combination of a lack of tiering, legacy virtual machines (VMs) that aren’t deprovisioned, and unnecessary snapshots, to name a few. Alas, simply living the cloud dream is not enough.
Whether you’re seeking to optimize your organization’s cloud storage, or just searching for ways to cost effectively expand your cloud storage use, you’re going to be constrained by your organization’s insatiable need to lower the cost of cloud storage. Sound familiar?
You’re still reading, so it’s safe to assume that you have a sense that your organization is overspending on cloud. If you have one or more of the following symptoms, a cloud spending problem is a safe diagnosis.
A typical enterprise maintains multiple accounts with multiple cloud providers. Because the public cloud is designed to empower anyone in an organization to set up an account and allocate resources, those with spending discretion often do so without any centralized notification of account creation. Shadow IT accounts pop up everywhere. Each one can be traced back to the lack of central control, and there never seems to be a single person who knows what accounts exist and what’s in use. The typical result is a finger-pointing match rather than a productive discussion to help you place your finger on the real issue.
Your organization’s monthly cloud bill probably spans a dizzying number of pages that contain thousands of lines and reference many service names, instance types, and regions, none of which are necessarily self-explanatory. Sure, storage volume pricing may seem straightforward at first glance, but look at your latest bill and try to comprehend—let alone explain—all the additional charges and how they relate to specific business initiatives. It’s often next to impossible. Time to pull out that magic cloud cost decoder ring.
Warning sign number 3 often goes back to the anonymous overspender. You have a blind spot when it comes to who is swiping the credit card—and what’s behind these spending sprees. Building on the second warning sign, organizations with multiple teams, projects, and budgets often don’t take the time to map their cloud storage and service consumption to projects and/or cost centers. The result is regular recurring spending that no one can explain. And that’s a drag.
The beauty of the cloud is that you buy whatever storage (and other cloud resources) you need…on demand. Although this pay-as-you-go (PAYGO) method gives your organization the easiest means to curate additional storage, the reality is that more often than not, you don’t need what you just purchased. Oh the irony.
That confounding mess of a bill we just mentioned? It shows that there’s no managed procurement process and reveals a complete lack of blended purchasing. Take the example of purchasing AWS On-Demand Instances. Even though On-Demand lets you spin up a VM at will, using AWS Reserved Instances instead gives you a discount of up to 75%. Taking it one step further, Spot instances (where you pay for a spare Elastic Compute Cloud instance that meets your needs) can save you as much as 90% over On-Demand.
See? If you‘re indiscriminately paying On-Demand prices, you can safely assume that there’s no spending strategy or usage plan—just a bunch of random purchases made by individuals acting much like your friendly neighborhood shadow IT. They just expect the organization to pay the overages. And they’re right, you will pay, but at what cost to the business?
Breaking news: Using data can be as costly as allocating it in the first place. There are two issues here. First up: How efficiently is your storage being used? You don’t want to pay for completely unused storage, and you definitely don‘t want pay for the most expensive tier of storage offered for data that could be stored on a less costly one. Costs can drop from $.20-$.30/GB all the way down to $.02/GB per month in “colder” storage tiers.
Second player at bat: Egressing data from the cloud (or among clouds). This is one of those pricing components that you probably fully comprehend in concept, but you don’t accurately know what actual monthly cost it translates into. This ties back to aligning costs with initiatives—the forensics work. Someone needs to trace data that’s egressed and determine whether the constant egress makes sense. For example, an unsuspecting developer might be simply copying test data to their on-premises endpoint to speed up the testing of some code. Doing this regularly adds up to huge fees, and it’s often completely avoidable. Someone needs to be on top of reviewing, questioning, and adjusting best practices around these charges.
All organizations, regardless of size, can easily find themselves in these cloud storage quandaries. The overarching challenge is twofold
First, everyone is using the cloud as if it exists exclusively to meet their needs, and no one is concerned about the resulting cost to the business.
Second, there’s a visibility challenge; it’s difficult to tie spending back to business needs and the user who’s responsible. And without visibility comes a lack of accountability. Every time. The unfortunate result is that organizations like yours end up spending far more than they should.
The bad news is that you have a cloud storage spending problem. The good news is that the first (and often hardest) step is admitting that you have a problem.
There are practical steps you can take literally today to begin fixing the problem.
The following 10 tips, although they’re largely IT-centric, can help the owners of each moving part of cloud storage to review their usage and reduce their storage footprint, enabling the organization to rein in spending.
When your VMs are terminated, normally only the root volume associated with the VM is automatically deleted. The additional storage volumes remain intact and incur storage costs— in some cases by design, to avoid accidental deletion. An easy way to reduce cloud costs is to find and delete unattached volumes. This one’s a no-brainer—if you’re not using the storage, get rid of it. But, as with any untethered resources, that involves some detective work to identify who owns the storage and have them attest to its need to exist or not. Who knew that data could be so existential?
"HOT" storage tiers (which usually holds frequently accessed data) can be priced as much as 5 times higher than their "cold" tier counterparts.
Every public cloud provider offers a number of storage tiers. And yet everyone seems to choose the fastest (and most expensive) level, with little or no regard for cost. If it costs more, it must be better, right? Not necessarily.
The GB/month tier price is generally based on how often and how quickly you need to access your data. “Hot” storage tiers (which usually hold frequently accessed data that requires low latency, high performance and throughput, and high availability) can be priced as much as 5 times higher than their “cold” tier counterparts (where infrequently accessed data, such as backups and archives, should reside).
Consider both your performance and cost requirements when assessing storage tiers, balancing your needs with attention to budgetary targets. You can always move it to another tier later.
There’s no easier way to waste money on cloud storage than to create a storage volume that is never used appropriately. Cloud providers don’t let you shrink storage volumes. NetApp recommends that you first go through the process of identifying oversized volumes, then create a new volume with the space you actually need, migrate the existing data, and delete the oversized volume. In the future, simply implement better storage need assessments for the volume creation process.
Cloud providers also offer performance tiers to meet your throughput needs. You can reduce storage costs by monitoring the actual read-write access of a given volume; if throughput is low, then downgrade it to a lower performance tier. This reduces the storage IOPS to be better aligned with the workload using it, while reducing cost.
When people have the option to replicate data anywhere, they tend to get panicky and choose a far-away location. But do you really need your data in the United Kingdom to protect against a loss of data due to, say, a hurricane in the United States? The answer, of course, is no, unless it’s a really, really big hurricane. These sorts of decisions have a material impact on cost; for example, redundancy across different geographies can be twice as expensive as local redundancy. It’s important to plan your redundancy requirements wisely, taking a page from business impact analysis and risk assessments to determine what’s actually needed.
Mental math: By choosing the right storage tier based on capacity and throughput, and jumping to a higher storage tier only at peak usage times, you can save up to 70% on your monthly cloud bill.
The snapshot is a staple of any virtual machine recovery strategy. Multiple snapshots give IT organizations the ability to restore to a specific point in time, based on a specific disaster recovery scenario. The last thing you want is to delete something that the owner of the workload needs. But when you have hundreds of VMs, each creating snapshots daily without deleting the previous day’s snapshot, your cloud storage costs grow exponentially. You must establish a strategy around snapshot expiration. Fortunately, most cloud providers have some form of snapshot lifecycle policy to automate deletions, eliminating the need to rely on any one individual.
Moving data costs money. That’s a fact. But those costs are not all created equal. In the cloud, the cost of transferring data depends on the location of the source and the destination cloud servers. Inbound traffic is normally free (or close enough). But when data is transferred outside the cloud provider’s network (aka data egress), it gets expensive fast. And remember, as far as the data owner is concerned, the transfer is all about “I need it done,” not “it’s cost effective to do it this way.” To address this issue, encourage users to store data as close to where it’s actually used as possible, to eliminate the need to move it somewhere else. Also consider compressing and deduplicating data before egress, using incremental synchronization to save on transfer costs. Finally, if you can delete data, or move it to an archive tier, do so with zeal.
When data moves across regions, countries, or availability zones, cloud providers get to charge you more. These data transfers can be part of an application’s architecture, used by DevOps to maintain test data, or they can be a part of a redundancy strategy. So data transfer needs to be both purposeful and sanctioned. Make it your goal to host necessary data as geographically close to its userbase as possible. Over the long run, consider rearchitecting solutions to minimize the path your data needs to travel.
Pricing for both storage and data transfers often includes additional cost tiers based on consumption. If you reach specified levels in the cloud provider’s pricing table—as indicated by terms such as “greater than ”—you are likely to be able to negotiate a better price. Keep in mind that higher discounts apply only to stored data that meets the pricing requirement. Finally, you may be locked into a multiyear contract. You should try to stay within the constraints of that contract, while looking for ways to negotiate costs down based on increased usage overtime.
Companies can save hundreds of thousands of dollars a quarter, sometimes even $100,000 a month simply by considering how they architect their cloud-based architecture.
Spencer Hamons, NetApp Professional Services
Some workloads require users to upload files. In this scenario, interrupted uploads sometimes result in partial objects that linger in cloud storage as unusable data that costs you real money. Depending on size, incomplete uploads can add up to tons of waste because admins tend to be afraid to delete or move anything (see step 6). The best course of action is to back up incomplete uploads and then delete them.
It’s imperative that you do something now.
Although these 10 steps can materially reduce cloud storage costs, organizations need a way to gain more visibility into what storage has been purchased, allocated, used, and wasted across the entire organization.
What’s needed is a way to empower team members who are concerned with what’s being spent and how well their cloud storage works. Embracing reporting, calculators, and other tools offered by cloud providers is a solid first step toward visibility and centralized management of your cloud spending.
When you’ve outgrown those basics, you’ll need to look at solutions designed to assess, collect, aggregate, analyze, and report on your cloud spending—including storage. Read more about that here.
To keep the savings sustainable, use policy, process, and additional technology to rein in your organization’s out-of-control cloud spending.If you’d like to speak to us directly about what you just read, feel free to contact us.
NetApp cloud data services are available on the biggest clouds and integrated with leading technology partners and orchestration tools.
“The work we are doing together to support our customers’ need for hybrid cloud infrastructure has been breakthrough work.”
– Satya Nadella, CEO Microsoft
“It’s great to have our engineers—and the synergies of our two companies—going all the time.”
– Bill Vass, VP of Engineering AWS
Find the latest blog posts about our products emerging technologies and NetApp culture.
Stay current on the latest NetApp product and company news, insights, and innovations.
Browse upcoming live, online, and on-demand events featured on the NetApp events calendar.
Our strong supplier relationships are grounded in the highest levels of trust and integrity.