Storage becoming synonymous with software

The reality is that the cagey storage providers are becoming software companies now.

Evolution always accelerates in response to worldwide cataclysmic events; so it comes as no surprise that the last two years have brought sweeping change to the data storage sector as well as the world at large.

Forward-leaning enterprises are moving to cloud-native architectures, with applications designed and developed specifically for the cloud. They are also shifting to DevOps to increase the cadence of their development-to-operations model. And of course, virtualization continues to advance, enabling what was once physical hardware to morph into virtual, software-defined infrastructures.

While 2019 was about getting data to the cloud, and 2020 was more about shifting millions to work-from-home mindsets, 2021 is about setting the data free. The challenges of accessing data, locked in storage siloes, have become all too acute. With workforces and applications becoming distributed, people are literally anywhere, and their data is still siloed. Having to navigate proprietary infrastructures and being forced to access data only through the point it was ingested makes using data a slow, expensive process for distributed users – one that typically requires specialized storage expertise. Meanwhile, other trends, including mounting ransomware attacks, network outages, and the growing need for the global collaboration of remote workers, make solving this problem even more urgent. Now, not only is on-premises storage often insufficient, relying on one cloud is, too.

By 2022, data storage vendors will have to support cloud initiatives that their customers have already adopted, and create solutions that take full advantage of the cloud and its ability to provide ease of accessibility, flexible services, and scalability. And most of this is software-defined innovation. The reality is that the cagey storage providers are there already – actually are becoming software companies now.

The line between storage and software will continue to blur, to the point that storage evolves into mostly next-generation data-management software, leveraging generic physical or virtualized resources to operate. This is the point where open standards are key – storage solutions in the form of data-management platforms must interoperate and play nicely with everything else in the ecosystem.

If Dave Buss’, CEO, OpenDrives, prediction for 2022, storage will finally become synonymous with software, is even close to being in the right zip code and storage does finally become synonymous with software in 2022, this change in perception will have a lasting impact on the storage industry.

Perhaps more importantly, though, nearly all companies that rely on data movement will reap the benefits of a software-defined and open storage ecosystem.

The market for data storage is growing at a CAGR of 32.3 percent from 2021 to 2026 and forecast to reach USD 4.2 billion by 2026. Data storage comprises of primary storage, often referred to as main memory and secondary or auxiliary storage. In cloud data, the order of level in which the data is stored in a memory card, HDD or SSD storage, is generally adopted by consumers. Cloud storage is preferred by enterprises.

The expansion of Internet of Things (IoT) has added numerous new sources of Big Data into the data-management background, and is one of the major trends; AI, ML, IoT, software-defined storage, object storage, and hyper-convergence infrastructure storage, anticipated to increase scalability will help the data storage in impacting businesses, while most of the enterprises are also embracing NAS solutions that allow for simple and reliable access to data.

IDC predicts that 55 percent of data created will be stored in the cloud and 65 percent of enterprises will be using cloud storage by 2025, the migration of enterprise production workloads to the hybrid cloud is the biggest factor now, and over the next five years.

Subscription services are enjoying strong growth in the data storage market, providing opportunities for partners to differentiate through services and technology that enables customers rather than holding them back. And 43 percent of organizations expect consumption of as-a-service models to increase by 2024.

Unstructured data, such as video, different forms of imagery, and other large files, is a significant growth factor. File and object storage is also expected to grow as the primary means of storing unstructured data. Large-scale analytics, backup, social media, and rich content are all driving unprecedented demand for scale-out NAS and object storage, where data is growing at a rate of nearly 40 percent annually.

The adoption of all-flash file and object storage is an undercurrent to this rapid growth in data, where organizations are now buying five to 10 times more flash-based file and object storage than in the past.

In terms of areas for future growth, Rob Tomlin, vice-president for the UK channel at Dell Technologies, singles out SMEs, which make up more than 70 percent of its dealings. The majority are at a crossroads because they need to re-evaluate their IT infrastructure.

“Many businesses will be looking to the public cloud hyperscalers as an easy option to meet their infrastructure needs,” he says, but “an exclusively public cloud operation is not the best route for businesses looking to make the best use of the ongoing data explosion.”

“Speaking with customers, the need to quickly scale is clear,” Tomlin remarks. “A hybrid cloud strategy offers the combined advantages of on-premise, private and public cloud deployments, promising security, protection, and availability.”

Hybrid storage will make a comeback in 2022. The era of single media-storage systems, like all-flash arrays, is coming to an end. Storage system vendors will need to make sure their storage software can intelligently move data between two or more types of storage, be it storage class memory, NVMe flash, SAS SSDs, hard disk drives, or cloud storage. They will also need to address the legitimate concerns of data tiering and minimize the weaknesses of each tier. Moving data seamlessly between these tiers, as it ages, is critical to establishing a perfect balance across all workloads.

The primary challenge with tiering is inconsistent performance, either because the data is on the wrong tier or because the data-tiering activity itself impacts performance. The decreasing cost of storage class memory and flash means that IT can oversize higher-performing tiers to ensure operational data is always on these tiers. To address the potential performance impact of data-tiering, vendors need to move a percentage of the least active data to a lower tier in advance of subsequent inbound writes. This pre-warming activity will smooth out the IO spike, and deliver consistent performance.

Vendors also need to invest in development efforts to take advantage of the innovations in hard disk drives. As the densities of these drives exceed 20TB, their cost advantage increases to 10x over flash media. A hybrid storage will become the dominant storage type because it will enable IT to meet a wide range of performance needs in a single storage system, while also driving down the CapEx and OpEx cost of storage for the next decade.

Cybersecurity is already one of the biggest issues any business faces, and the shift to hybrid working has created new problems for IT security departments to worry about. Outside of the protective umbrella of office networks, there’s a real risk of data breaches and devices have become more susceptible to malware attacks. This is against a background where ransomware is becoming a major threat that can entirely cripple a business.

To meet these challenges, zero-trust authentication security models are growing in use. Quicker responses to zero-day attacks help limit the fallout from security incidents, and solutions based on AI/machine learning could see increased deployment to enable tracing leaks faster.

VPN use will no doubt grow to support staff working from home, along with mandates on encrypted storage, particularly when hybrid work means critical data is constantly shifting between offices and external locations. Hardware-encrypted devices like USB are certain to remain best practice for endpoint cybersecurity, as the encryption process is kept separate from the rest of the machine.

Enterprise spending on storage systems accelerated in 2021, with total storage capacity shipped rising 13.8 percent year-over-year (YoY) to 88.7 exabytes. According to IDC, the global market revenue for enterprise external OEM storage systems grew 9.7 percent YoY to USD 6.9 billion during the second quarter of 2021 (Q2 2021). Total external OEM storage capacity shipped was up 27.9 percent YoY to 22.1 exabytes during the quarter.

Dell Technologies was the largest external enterprise storage systems supplier during the quarter, Q2 2021, accounting for 26.8 percent of worldwide revenue. HPE/H3C was the second-largest supplier during the quarter with 10.9 percent of revenue. NetApp and Huawei tied for third place with 9.9 percent and 8.9 percent of total revenues, respectively. Three vendors, Hitachi, IBM, and Pure Storage tied for fifth place in the market with shares of 4.9 percent, 4.7 percent, and 4.1 percent, respectively.

Gartner pegs the global external controller-based (ECB) storage revenue at USD 23.4 billion in 2025, up from USD 21.1 billion in 2020. This will be buoyed by expansive demand in Greater China, and solid growth in unstructured data. And APAC is expected to account for 36 percent of the market’s growth. The increased demand for data centers from the banking and finance players, hyperscalers, and technology firms will drive the enterprise external OEM-storage systems market growth in APAC.

Overall demand for storage technology of all types continues to grow, as the total amount of stored data increases. This storage-capacity demand will drive demand for all types of storage technology.

Magnetic storage tape has not been the recommended destination for the initial backup copy of data for quite some time, and the question is whether LTO-9, the latest tape open standard, and other market dynamics will change that.

The LTO Consortium seems to know that tapes that keep getting faster are the main reason many organizations have moved away from it as a primary backup medium. While LTO-9 is 50 percent larger (18 TB versus 12 TB, uncompressed), it is only 11 percent faster (400 Mbps versus 360 Mbps). Compare this with the 20-percent speed increase of LTO-8 over LTO-7 and the 87-percent increase of LTO-7 over LTO-6. It is clear that the consortium knows it needs to increase tape’s capacity without making it much faster.

LTO-9’s speed of 400 Mbps (uncompressed) and more than 1000 Mbps (compressed) is still way faster than any incremental backup. However, tape vendors seem to be trying to solve the issues that caused the mismatch in speed between backup sources and tape.

One downside to tape is that, unlike disk and cloud backups, the recovery environment cannot simply be launched directly from the backup; that requires random access that tape cannot provide. So, a recovery from tape will likely take much longer than a recovery from a replicated disk system, but at least it will not get infected by ransomware.

Given that time to restore data is a business-critical metric, it makes sense to use disk or cloud or both as the primary DR mechanism. But also consider using tape to create an air-gapped copy to specifically protect you from ransomware and think of it as the doomsday copy. Even the slowest recovery from tape is better than a deleted copy on disk.

HDD use got a boost in 2020 and early 2021 due to the purchase of computers, enabling people to work remotely. Forbes estimates that desktop HDD shipments were 6.9 percent higher in 2021 than in 2020, and mobile HDD shipments were 3.0 percent higher. However, with consumer SSDs selling for less than 5x the price of HDDs (in USD/TB), and with many users satisfied with 1–2 TB of local storage, SSDs have been gaining popularity. As a consequence, the majority of PCs sold today and those sold in the future, will be using SSDs rather than HDDs. HDDs continued to decline in shipments for high-performance enterprise applications as well as consumer applications (down in 2021 compared to 2020 by 11.5 percent and 46 percent respectively).

Although declining for many applications, high-capacity nearline HDDs for secondary storage in data centers increased by about 50 percent in 2021, and should continue to grow in 2022 and beyond. Magnetic tape is used for cold storage and archives and uses less energy to store data than HDDs. Both technologies will continue to be important to store the world’s data in the future.

SSDs are now primary storage in enterprise and data center applications. NVMe is the primary SSD interface with NVMe-oF and an important element in storage networking. NAND and DRAM scaling continues to provide higher capacity at lower prices. CXL is poised to transform far-memory and OMI could offer advantages for near-memory. MRAM and RRAM are enabling new embedded products.

NAND flash memory has become the dominant primary storage in data centers and enterprises and is the storage used in consumer devices, such as smart phones and tablets, and now dominates as storage in PCs. Active data now lives on SSDs and with colder data kept on HDDs or magnetic tape, SSD companies are trying to put pressure on HDDs for secondary storage. They are primarily doing this with very-high layer-count NAND flash as well as quad-level cell (QLC) flash that stored four bits per flash memory cell.

Much of what will come to be will involve open-source storage, which has increasingly made its way into mainstream enterprise storage applications as more and more organizations discover the benefits to their operations.

Storage innovation is thriving as delivery models become more nuanced. Technology and service providers must spot the timing and potential impact of the technologies to attain competitive advantage.