I was answering a client’s question about Unific’s Magento 2 to HubSpot integration this morning and realized that we could do a better job of explaining how the integration works and why it works that way. While moving data from one platform to another may seem like a very simple problem to solve in this day and age, there’s a lot more to it!
With over eight years of experience helping ecommerce merchants use our Magento 1 extension, and now our Magento 2 extension, I’ve learned that the following three differentiating factors matter most:
- Is the data sync based on polling or webhooks and event-streaming architecture?
- How an integration manages Spikes, Magento Server CPU Load and Database Load
- Constant, proactive monitoring of sync status and health
Through the rest of this article we’ll examine these points in detail to help you better understand how Magento 2 to HubSpot integrations work, and why they are not all created equally.
Is the data sync based on polling or webhooks and event-streaming architecture?
What is “polling” anyway?
The most common (and easiest) way to create an integration to sync orders to HubSpot with Magento is to “Poll”, in other words “Ask” the Magento 2 API at regular intervals:
“Are there any new orders?”
<< Wait 30 minutes >>
“How about now?”
<< Wait 30 minutes >>
“Are there any new orders?”
And so on …
The problem with this approach is that an ecommerce store potentially captures events that are happening seconds apart and an integration that syncs every 30 minutes will miss most of those events.
Think about this sequence of events:
- Polling integration asks Magento 2 API: “are there any new orders?”
- Magento 2 API says: “nope!”
- User adds a product to the cart
- User clicks ‘Checkout’
- User enters their address
- User enters their payment information
- User places an order
- Your team sees that the order was just paid
- 5 minutes go by
- Your team finds the items ordered and packs them
- 5 minutes go by
- Your team ships the order
- 20 minutes go by
- Polling integration asks Magento 2 API: “are there any new orders?”
- Magento 2 API says: “yes! Found one order!”
- Integration syncs that order to HubSpot 20 minutes later
If your distribution team is really on the ball, they might pack and ship the order 10 minutes after receiving a paid order.
However, in this example, if your integration is “Polling” the API every 30 minutes, that integration missed a total of six steps (steps 7 - 12).
Six is not that many you say?
In reality there are a LOT more events that fired in this sequence. Here are just some:
- checkout/create
- checkout/update
- customer/create
- customer/update
- order/create
- order/update
- category/create
- category/update
- checkout/create
- checkout/update
- customer/create
- customer/update
- order/create
- order/update
- product/create
- product/update
The update events themselves nest many minor events that occur. For example, if the order status in Magento changes from “Pending Payment” to “Payment Received”, that would be an order/update event.
Why is this important?
As you probably already know, HubSpot is a really powerful CRM and marketing automation platform. The key word for the example in this post is automation (which is provided by HubSpot Workflows.)
With workflows, you are able to create workflow enrollment triggers when any data changes.
For this, I will give an example of my own online store, Bugs for Growers. Bugs for Growers sells live beneficial organisms to organic gardeners and greenhouses for pest control.
That means we have to ship live organisms that require constant temperature regulation. If you order from us and won’t be home when the order arrives we request that you put a cooler with ice on your front porch. We then tell our shipping company to put the box we ship in the cooler on the doorstep on delivery.
Not all the products we sell require this step, so I need to check if a product that requires a cooler is purchased.
So what I want to accomplish is:
- When there is a new order
- And that customer bought in that order product x,
- Send an email to that customer with instructions on putting the cooler out
I want that email to be sent immediately, because as the customer is expecting an order confirmation from my store, they will also see the additional instructions email.
If there was a significant delay between order confirmation and the delivery instructions email, we run the risk of the customer not seeing it, not complying with the instructions and having a bad experience.
The Workflow:
- This workflow is triggered when the “Order Status” property on the Deal record is set to “Complete”. (equates to Magento Order Status is set to ‘Complete’)
- If the “Nematodes Product” was bought in this order, send the delivery instructions email discussed above to the Customer
- If the “Nematodes Product” was not bought, then do nothing
If there’s a 30 minute or more delay caused by the Magento integration extension, this notification will also be delayed by 30 minutes or more and the customer may not see it and we might have to send another box of the product for free if those organisms didn’t survive.
You might say, “this is a good example for your store but I don’t have this problem.”
True, your store’s use case might be different, however, in ecommerce the competition is only getting tougher and whoever is closer to the customer wins.
You invested in HubSpot so that you can get closer to the customer and use its powerful tools, but if your data does not sync to HubSpot in time, you won’t be able to use that power and your customer’s experience may suffer.
Just think! What can you do to make your customer’s experience as seamless as possible with the power of HubSpot workflows and a fast integration?
How Webhooks and Event Streaming Architecture are different from Polling
Unific doesn’t use “Polling” - instead, our Unific Connector Magento 2 Extension is built to listen to these events, which Magento 2 produces:
When each event “fires” the extension sends a notification to Unific’s backend servers called a “Webhook”. The backend servers then take each of these events, and organize them so that they are synced to HubSpot in the order they were received and as quickly as possible.
Unific is built on an “Event Streaming” architecture (for a more technical description, see this article) and differs from all the other extensions we know of.
The best advantage of webhooks is that they are faster than polling. They are sent every time specific events happen in the Magento store and are not relegated to a schedule. They also make retries very easy. In case a webhook is not received by Unific’s backend, our Magento extension will re-send the webhook again. We also don’t need to query the Magento database again to re-send the same webhooks. They are saved to be re-sent again.
Polling is confined to a schedule and is very inefficient both in speed and waste of resources.
Spikes, Magento Server CPU Load and Database Load
The more work Magento does the more CPU and database resources it will use. When your store has low traffic and activity, this is not a big issue, but if your store receives spikes in activity Magento has to do a lot more work to keep up.
Spikes could easily be created by marketing campaigns (expected), or one of your coupons gets shared onto Retail Me Not, or another coupon sharing site (unexpected). We have a solution for that coupon problem, but let’s stay on topic!
There are many scaling solutions that do deal with spikes, but let’s face it - a Magento extension used to send data to HubSpot should be the least of your, or your Magento developer’s worries! It should just work.
Since most extensions use Polling (as discussed above), those extensions alone can cause the website to go down if:
- an API call made to Magento returns too many results OR
- there is so much activity that the Magento store can’t respond to an API request and crashes
Most API’s on the internet have Rate Limits, but since Magento 2 is usually self-hosted, it does not have any rate limits set natively.
If your integration relies on Polling, it’s likely you will have to buy another Magento extension or service to manage rate limits. If you do buy one of those - be sure that they implemented rate limiting correctly so that the polling extension knows when to retry or how often to retry? Also, you’ll want to confirm your HubSpot will be updated with the most accurate data after the website crashed?
How Unific solves for this problem
The solution to this problem is to have Magento and it’s database do as little work as possible and also give your developer as much control as possible so that they can prevent crashes. Magento administrators and Magento developers love to be prepared because no one likes being woken up rudely by a 3 AM phone call!
With the webhooks solution discussed above there is no need to ask the Magento 2 API “Are there any new orders?” Instead, the Unific extension is generating an event for every new order as they are placed and the Magento database simply has to find the data for the current order that was placed. This is a small amount of additional work for Magento compared to querying the Magento database for “all orders in the last 30 minutes” - which is what Polling would do.
Also, these events that are being generated are not being sent to Unific backend servers immediately. Instead they are saved to a Message Queue.
See the “Message Queue” section of the Event Streaming article I linked previously for more on that concept.
Your developer will now probably ask - what consumes these queues?
The Unific Connector Magento 2 Extension creates cron jobs using Magento’s crontab which takes a number of messages off the queue and sends them to Unific in an asynchronous fashion.
This decouples the collection of data from the sending of data which allows the Magento administrator to control how much load the Magento 2 server and database take. This is a version of Rate Limiting that Unific puts into the Magento developer’s hands.
In the Unific Connector Magento 2 Extension we’ve created these settings:
- Message Batch Size:
- Defines the maximum number of messages that are loaded into the cron job process at any one time
- Default value is set to 10
- Allowable range is from 1 to 25
- Single Cron Run Message Limit:
- Defines the maximum number of messages that can be sent during one cron job run
- Default value is set to 100
- Allowable range is from 50 to 200
These settings allow your Magento developer and/or administrator to customize how much load the Unific Connector extension is putting on your Magento Store and balance that with the speed of the sync.
They can even opt to turn off the cron jobs (queue consumers) for a short while to reduce load on Magento in case of a spike. If the Magento developer/administrator does stop the cron jobs, all the Magento events just get queued up and saved up for sending later. When the cron jobs are turned back on they will just continue syncing events from where they stopped previously. Thus, even recovery is easier with this architecture.
Bottom line: the Magento developer/administrators know how the store they are managing is handling the baseline load as well as spikes. Thus they are in the best position to manage it. They don’t have to contact Unific to reduce the number of API calls (a common request with Polling solutions.)
They can instead just lower the usage of Magento crons temporarily if they need without asking Unific. That’s the beauty of a decoupled architecture!
Wait just a minute, aren’t cron jobs just the same as schedules? And you just said that schedules were bad!
Yes, you are right, cron jobs are just the same as schedules because cron jobs must be scheduled. However, we recommend the Magento administrator run Unific Connector’s cron jobs once per minute. So new webhooks are processed less than a minute after they are created. They are almost always sending the webhooks. That’s what gives them the near real time speed.
Constant, proactive monitoring of sync status and health
"If a tree falls in a forest and no one is around to hear it, does it make a sound?"
Once an integration works, you wouldn’t normally come back to check on it. The only indication you might have that it’s failing is that your data might not sync to HubSpot. However, that is still not a very good indicator. You would only notice if some major things were not happening, like if Contacts or Deals were not being created.
What if it was just one part of the integration, like maybe abandoned carts stopped syncing? It would not occur to you to check on that specifically.
Additionally HubSpot and it’s APIs can also go down or may be slow.
A different approach
We’ve built constant, proactive monitoring into Unific with automated systems that detect anomalies in the webhooks we are getting. We don’t just monitor order webhooks and syncing. We also monitor webhooks and sync for customers, abandoned carts and coupon code generators. Our monitoring systems will alert us if we are not getting webhooks, or if there are spikes for a particular customer and a number of other monitors.
We also monitor HubSpot and it’s API. If we are experiencing latency or failure, we’ll know and we can temporarily pause the sync for our customers until issues are resolved. An event streaming architecture automatically allows for retries and your data will just get synced when the services are back without asking Magento 2’s API again for that data.
Bonus #1: We’re monitoring the above items for more than 2,000 stores. That means you also get early warning in case one or more of our customers are seeing issues.
Bonus #2: You can always see Unific's sync status at: http://status.unific.com/.
Summary
In this post, we’ve laid out our top three differentiating factors when it comes to Magento 2 to HubSpot integrations. We went into some deep technical details about webhooks, the disadvantages of polling, the importance of scalability and reliability.
All of these items are critical to consider before you choose which extension/integration to use for your Magento store. As you know, Magento development and management are high cost and can include some risk.
If you have any questions about integrating your Magento 2 store with HubSpot, we’re standing by to help. You can use the chat feature on our website, or use our meeting booking tool to schedule a casual discussion with our team below.