Intel Unveils Distributed File System to Handle Big Data

    The problem with existing file systems is that they were never designed to deal with Big Data, which results in a lot of fragmentation that makes IT that much more difficult to manage.

    To address that issue, Intel today is releasing an enterprise edition of Lustre, an open source distributed parallel file system that masks the complexities associated with accessing files in parallel.

    Slide Show

    Four Steps to a Big Data Strategy

    According to Brent Gorda, general manager of the High Performance Data Division for Intel, Lustre supersedes the file system that comes with an operating system, which was never designed to handle massive files.

    Gorda says Lustre takes advantage of parallelization to support tens of thousands of client systems and tens of petabytes of storage at speeds of well over 1 terabyte per second. As a POSIX-compliant file system, Gorda says Lustre also automatically updates all files in parallel.

    In addition, Gorda notes that Lustre is compatible with multiple public cloud computing platforms, including Amazon Web Services, and comes with an adapter for Big Data environments based on the Intel Distribution for Apache Hadoop.

    In all the enthusiasm for Big Data, there is a tendency to overlook some of the more mundane storage management issues that can make these projects pretty complex. Of course, this is an issue that any High Performance Computing (HPC) environment has long since dealt with. Now it’s just time for everybody working with Big Data in the enterprise to come to terms with the same issue.

    Mike Vizard
    Mike Vizard
    Michael Vizard is a seasoned IT journalist, with nearly 30 years of experience writing and editing about enterprise IT issues. He is a contributor to publications including Programmableweb, IT Business Edge, CIOinsight and UBM Tech. He formerly was editorial director for Ziff-Davis Enterprise, where he launched the company’s custom content division, and has also served as editor in chief for CRN and InfoWorld. He also has held editorial positions at PC Week, Computerworld and Digital Review.

    Get the Free Newsletter!

    Subscribe to Daily Tech Insider for top news, trends, and analysis.

    Latest Articles