The interesting thing about EMC's new flash storage technologies named "Lightning" and "Thunder" is that they likely make all large-scale databases obsolete from a performance perspective. That was the big thought I walked away with this week when I got the briefing on these advanced PCIe-based flash technologies.
Let's talk about the end of an era where large databases, and undoubtedly a lot of storage-intensive applications, will be made obsolete largely because magnetic media is being moved increasingly to more of a long-term storage role and, due to its historic performance limitations, away from real-time access.
Databases and Data-Intensive Applications
At one time magnetic media was thought to be incredibly fast, particularly when compared to magnetic tape, which replaced paper tape and punch cards. However, like all-things magnetic, media is being first enhanced and then replaced by flash for improved real-time performance and it will have a non-trivial impact on existing data-intensive applications, particularly databases.
You see, and I hadn't thought about this myself, but data-intensive applications have been optimized against the limitations of magnetic storage with advancements in indexing, caching and predictive fetching that are designed to get around the physical limitations of a magnetic arm that has to physically sweep a storage media to find a bit of data. Databases in particular compete against each other for speed of access and they largely get that speed by finding creative ways to overcome the limitations of magnetic media.
What if you remove those limitations? This would be like a race that focused much of its efforts on building cars that handled high-speed corners as being changes to a track that had none. Every car would need to be redesigned against a new performance template and conditions. The cars that won in corners would likely be non-competitive on the new track (compare to a dragster to an F1 car, for instance). The elimination of magnetic media as a problem in much the same way should eventually make virtually all data-intensive applications obsolete, particularly databases.
However, initially, once again using that F1 car example, it will make them all faster, but as the vendors learn the new performance limitations, competition will force them to rebuild and this should provide a huge opportunity for smaller firms, which don't have to rewrite legacy code, to advance on the bigger entrenched players.
Thunder and Lightning
These are EMC project names and Project Lightning is now known as VFCache, which is a card-based solution that goes into servers to increase I/O performance massively. Project Thunder is a high-speed flash appliance that goes into server farms as a massive high-speed cache. Both use SLC NAND flash, which has lower capacity than the more popular MLC flash, but it is far faster and has better endurance. The flash comes from Micron, a well-regarded supplier of flash technology.
Like all technologies in the enterprise class, expect testing to take much of this year and the first large-scale deployments, if the technology passes testing (which appears likely), to occur in 2013.
Impact on the market will depend on how aggressive the software suppliers are, but it seems unlikely that major providers will move to the potential of these offerings for several years. However, a challenging vendor or new entrant could be more aggressive and target early adopters, likely in partnership with EMC (because it would enhance the value of its new flash offerings), earlier. This could provide a unique opportunity for a company that wants to move against a major vendor like Oracle to create a competitive migration opportunity that might have not existed otherwise.
In addition, it is likely that EMC's competitors will see these products as a threat and move against them with their own similar or alternative offerings. This could further validate and accelerate this move to provide technology like this for the market and likely will accelerate the rate of change making 2015's storage environment vastly different than it otherwise would have been.
Wrapping Up: Storage Change Is in the Wind
Storage is an area that doesn't change often, but when it does it has a broad impact on the entire technology ecosystem. Aggressively applying an accelerating technology to this ecosystem will likely be seen, in hindsight, as one of the bigger changes this industry has had to adapt to because there was so much optimization around magnetic media, which now may be obsolete.
This will provide opportunities for new entries and risks for existing companies that can't or won't move quickly enough. In the end, and likely over the next five years, Project Lightning and Thunder will likely become one of the most disruptive catalysts of this decade.