More

    Airbyte Adds Kit to Simplify Building Connectors for Data Integration

    Airbyte has added a connector development kit to an open source data integration platform that cuts the time to add a data source down to less than two hours.

    Previously, IT teams would have had to employ a REST application programming interface (API) that would have required two days of effort, says Airbyte CEO Michel Tricot. “Our next goal is to get it down to 15 minutes,” he adds.

    Ultimately, the goal is to democratize data engineering with an open source platform that Airbyte will make additional commercial capabilities available for next year. The company recently raised $5.2 million in seed funding to help achieve that goal.

    The Airbyte Connector Development Kit (CDK) also standardizes the way connectors are built, maintained, and scaled for an extract, transform and load (ETL) tool that is already employed by more than 600 organizations, including Safegraph, Dribbble, Mercato, GraniteRock, Agridigital, and Cart.com. 

    Also read: Reliance on Open Source Software Set to Increase

    The Builder Community

    Organizations have embraced Airbyte because they don’t have to wait for a provider of commercial ETL tools to create connectors for various data sources. Instead, the community collaboratively builds and supports the connectors it deems most critical. The community has, thus far, certified 50 connectors, each of which is encapsulated in a Docker container that makes it possible to deploy them anywhere.

    Those connectors make it simpler to move and consolidate data from different sources that range from data warehouses, data lakes, or databases running in on-premises IT environments or in public clouds. Tools such as Airbyte are used by data engineers to set up data pipelines between these platforms as part of a digital business transformation initiative that might, for example, require data to be regularly shared across multiple platforms. Those requirements have led some organizations to define a set of best data operations (DataOps) best practices to automate the movement of data at scale across a highly distributed cloud environment using one of several data fabric platforms that have recently emerged.

    Also read: Rush to AI Exposes Need for More Robust DataOps Processes

    Developing DataOps

    ETL tools have been around in many forms for decades. Historically, organizations employed these tools to programmatically move data into a backup and recovery platform. With the rise of the cloud, IT teams began using these same tools to move data into a public cloud. Since then, DataOps processes have started to emerge that has created increased demand for data engineers that have the engineering skills required to build data pipelines. It’s not clear, however, to what degree the construction of data pipelines might one day be automated. Tools for building and managing data pipelines using graphical tools that are accessible to the average IT administrator have started to emerge.

    In the meantime, the number of platforms that have varied types of data that need to be integrated with other platforms continues to exponentially expand. In the short term, connecting all those data sources will require a data engineer, which is why the function for now remains one of the hottest job markets in the IT sector. The degree to which that will continue to be the case once machine learning algorithms are employed more within various classes of data management platforms to automate processes remains to be seen.

    Read next: AIOps Trends & Benefits for 2021

    Mike Vizard
    Mike Vizard
    Michael Vizard is a seasoned IT journalist, with nearly 30 years of experience writing and editing about enterprise IT issues. He is a contributor to publications including Programmableweb, IT Business Edge, CIOinsight and UBM Tech. He formerly was editorial director for Ziff-Davis Enterprise, where he launched the company’s custom content division, and has also served as editor in chief for CRN and InfoWorld. He also has held editorial positions at PC Week, Computerworld and Digital Review.

    Latest Articles