Prospects coming from competing integrations have been trained to expect data to sync on a schedule. We built Unific to be near-real-time without relying on a sync schedule. (By near-real-time we mean the sync might take between 1 second to 3 minutes after events happen in the Magento store.) 

How did we do it? We get this question frequently and I’ve long wanted to make the answer a blog post! Note that this article is going to get a little technical, but it’s a complex topic and those of you who are Magento developers or admins should find this useful.

Webhooks enable near-real-time synchronization

Unific relies on Webhooks instead of Polling at a scheduled time. The Unific Connector Magento 2 extension is made to detect certain events in the shopping cart. For each event the Unific connector will send a message to Unific’s backend servers. This message is called a Webhook. (Here’s a great article on how webhooks work.) 

Webhooks are faster at synchronizing data

Webhooks happen when an event occurs. For example, if a new order was placed in your store, the order/create webhook will be ‘fired’. That means the Unific backend will receive the webhook as soon as Magento sends it. Unific will then sync the data in that webhook to HubSpot. 

Compared to syncing data on a schedule, syncing by event is much faster since there’s not a forced waiting period.

Webhooks are more efficient than Polling

Webhooks require a much simpler and scalable setup for the integration that’s syncing data from Magento 2 to HubSpot. Polling has challenges that can be overcome, but it takes a bigger effort to do correctly and is not as efficient as webhooks. Lack of efficiency gets in the way of speed!

Most integrations that use polling are forced to ask your Magento store’s API for new or updated data with a single server or single process.

That’s because using multiple servers would result in an unpredictable number of API calls being made to Magento. An unpredictable number of API calls also results in an unpredictable number of queries being made in the same instance of time to your Magento database. 

That unpredictability can cause Magento 2 to crash and not receive orders. That ends up harming your whole business. In a polling architecture, you would have to invest in a much more powerful Magento 2 database to handle the load on the API to prevent crashes.

By contrast, webhooks are much better because Magento only queries data for a single object which changed (and other related joined tables) on which an event was triggered (ie. order/update). Webhooks end up making the job of the Magento 2 database easier because when an event happens, the ID of the order, or customer (for example) is already known because the webhook creation process comes right after an event is detected inside of Magento. Thus the webhook creation process is querying for a known ID (which hopefully has a database table index already). This type of query is really easy for the database return results for. 

Conversely, polling will result in a query that returns multiple rows of the same object type (ie. orders) in order to check whether a new order was received or updated in a date or time range. That query can be very expensive for the database to run if a lot of activity occurred in that time period.

Also the event that creates data for the webhook is triggered by the user themselves (for example, as they are placing an order during their user session). Thus multiple webhooks are being created at the same time, and if you have optimized Magento’s ability to handle many many user sessions at once, you have also optimized a faster sync because the user session is the trigger for the webhook.

Most likely, you have not optimized your Magento 2 database for queries run by the Magento 2 API. You most likely have optimized your Magento 2 database to handle the traffic that is coming in regularly. 

I know that explanation was long, but for this reason, your sync will be faster with webhooks rather than a polling sync.

I thought Magento 2 doesn’t support webhooks. What are you talking about?

At Unific, we learned the above advantages and disadvantages of webhooks and we consciously decided to invest in creating webhooks with our Unific Connector for Magento 2 extension instead of relying on polling. 

Webhooks are the right solution to the problem of syncing real time data (which ecommerce data is) to HubSpot. Other platforms we support like Shopify and WooCommerce already use webhooks and thus we thought it was a wise investment to build our own for Magento 2.

Dealing with spikes of activity and why polling is a downward spiral

Increasing the Magento server’s database CPU and RAM works for the baseline of activity you already know about - but what happens when there are spikes of activity? Perhaps a spike is caused by a coupon code being posted on Retail Me Not (we’ll discuss preventing coupon fraud in a future article, but know that our Dynamic Coupon Code Generator creates unique expiring coupon codes to prevent that).

During a surge in activity polling architecture will have a slower sync time because a single process making a single API call to Magento 2 returns a limited set of data (pre-designed), and there will need to be many API calls to completely synchronize the new incoming data. A polling architecture can’t easily scale up to start making more API calls when there is more data to sync. Net, new data to be synced will stack up and it will be hard to automate a timely marketing response in a polling architecture. 

Unific’s event streaming architecture is built to deal with spikes

As discussed above, the Unific Connector Magento 2 extension has been built to send webhooks for every relevant event in Magento 2 to Unific’s API. When a webhook is received by Unific, Unific will automatically queue up the webhooks into a message queue. If that message queue is experiencing a spike, Unific’s systems will automatically increase the resources needed to process messages in that queue. The result of this is that sync times are only slightly impacted by spikes. 

The easier scaling ability of webhooks and Unific’s event streaming architecture ensures a faster sync time than polling.

(If you are unfamiliar with message queues read this article: How Stream Processing Makes Your Event-Driven Architecture Better).

Retries

One of the biggest advantages to webhooks and event-based syncing is the ability to retry. If webhooks, for example, are not received by Unific, then the Unific Connector for Magento 2 extension will try re-sending them. The main point being -- since webhooks are event-based -- Unific will know specifically which webhooks (events) failed by their status and did not sync, and we can re-trigger them at any point in time. It’s best if this is automated, but if they will forever fail, it’s also good to mark them as failed and try them again after whatever problem prevents them from working is fixed.

With polling, retries are certainly possible and recommended, but practically it’s very difficult to to mark certain specific API calls to retry for a certain date period. If that’s not done, the next best option is to retry more frequently which can needlessly expend resources.

Are there disadvantages to webhooks for syncing data from Magento 2 to HubSpot?

The biggest disadvantage to webhooks is that since they are asynchronous, it’s possible an event with identical data might be sent many times per second. For example, there’s not much difference between an order/create and order/update webhook. Technically, both of these are fired at almost exactly the same time. If timing is exactly right, it can result in duplicate deals being created in HubSpot or duplicate customers, or products etc.

The receiving integration (in this case Unific) will have to de-duplicate the webhooks and at scale! If a website is experiencing a lot of traffic (maybe one of their really good discounts got put on Retail Me Not and is causing a big spike in orders, abandoned carts and customers), Unific will start to receive all the webhooks in near real time and increase the chances that the same exact webhooks are sent at the same time.

Fortunately, Unific is built to handle this and performs extremely well at ensuring duplicates are not created. The deduplication process can add a very small amount of delay to the whole sync, but it’s usually less than 1 second.

Summary

In this post, we went into some deep technical details about webhooks, the disadvantages of polling, and how an event-streaming architecture contributes to data sync speed between Magento 2 and HubSpot. 

Be sure to do your research before choosing which integration and speed is necessary for your business. If you have any questions about integrating your Magento 2 store with HubSpot, we’re standing by to help. You can use the chat feature on our website, or use our meeting booking tool to schedule a casual discussion with our team below.