How Open-Source Data Integration Will Make Your Company Future Proof cover

How Open-Source Data Integration Will Make Your Company Future Proof

David Molot profile image

by David Molot

Jan 5th 2022

The Landscape

undraw_Selected_options_re_vtjd.png

The data integration space is filled with companies that help businesses move billions of rows of data everyday. You can look at companies like Fivetran, Talend, and Jitterbit and think to yourself that the problem of data integration is solved. The current incumbents are huge players in the space and are still growing (for example, Fivetran just raised $565 million). They build and maintain thousands of integrations for customers around the world, and are multi-billion dollar companies.

But then you take a look at the open-source data integration space, which while not new, has been evolving immensely in the past few years. Just look at some of the companies that have popped up - Meltano, which recently spun out of GitLab and raised a $5 million seed round. The question is why are they growing in what seems like an already-saturated market?

Why is open-source data integration exploding?

So if the problem of data integration is already solved, why are there new companies popping up over the past couple years to compete with companies like Fivetran? Let’s provide a little context.

If we zoom out and look at the SaaS industry from a macro perspective, the amount of new SaaS tools that businesses use every year is increasing, whether it is in the engineering, HR, sales, or the accounting department. For most businesses, this means there is another tool where essential company data is stored. Furthermore, this means that they will need to integrate with those tools in order to get the relevant data out of them.

Now let’s bring it back to integration providers. When a company signs up to use a tool like Fivetran, they are then locked into the integrations that Fivetran supports. While Fivetran does support thousands of connectors, being locked in can make it difficult to support every customers needs. Let’s walk through an example.

The experience

Closed-source integration provider

undraw_access_denied_re_awnf.png

Let’s say that your team began using a new tool and you now need to bring data that sits inside that tool into your database. Your team already uses Fivetran to integrate with all the other tools you use, so you would like use Fivetran to orchestrate pulling the data out of your new tool. You go to Fivetran prepared to set up the new integration, but they don’t currently support the integration you want!

While there are some workarounds, you would really love for this tool to be plug and play with Fivetran. You email your account manager at Fivetran and ask if Fivetran will be building support for it and they say that they will add it to their backlog. For now, you figure out how to use the work around and duct tape together your pipeline, while you wait for Fivetran to get back to you (which could take weeks, if not months).

Open-source integration provider

undraw_Happy_news_re_tsbd.png

Now let’s see what the experience would be like with an open-source integration provider.

Once you realize you want to bring in data from a new tool, you check if the open-source integration platform (such as Meltano) currently has a supported integration for the tool. If there is none, you can head to the SDK that the open-source tool provides to build the integration. You dig in and create the connector in a couple hours using the SDK, with all of the endpoints you need.

You can then plug the integration you built into your data pipeline, and it is natively supported! No waiting on someone to log your ticket, put it in with the product team, and finally prioritize building that integration.

The benefits

Let’s go ahead and break down the specific benefits of the open-source side of integrations.

Customization

With the open-source model of integrations, having a connector fit your needs is much easier. Let’s say there is a connector that doesn’t support pulling all the data you need. You simply submit a code change, and voila - the connector now supports the endpoints and data you want!

Coverage

With the open-source model, the long tail of connectors can be much more easily covered. There is no need to now have to convince a tool like Fivetran that building a new connector will provide a significant ROI. Instead, you can just go ahead and build it yourself, and get the help of a larger community to maintain the connector.

Debugging

As you may already know, the hardest part of integrations is not building the integration. It is maintaining the integration for the rest of its life. When you utilize open-source integrations, you have an entire community using the same integration and submitting updates to the code-base. So if you run into any issues with a connector, you can simply submit an issue and pull request and the whole community benefits.

This makes it much easier to keep a large library of integrations up to a high standard. Unlike a closed-source provider, you wouldn’t need to wait for their team to prioritize bug fixes against a long backlog – you have full control to make changes yourself.

Beyond the connectors

The open-source data integration space is incredible, and is changing the landscape for the future of data integration. It enables anyone to leverage a robust and production-level integration in a record amount of time, all without sacrificing the customizability of the connectors.

hotglue is a proud participant of the open-source data integration community, and contribute all our connectors back to the open-source community. For our customers, our open-source architecture is a strong reason for choosing hotglue to power their native SaaS integrations.

If you are interested in harnessing the power of open-source data integration for your SaaS product, check out hotglue.