1. Background: How product sync worked before
Previously, HotWax provided two separate jobs to sync your Shopify catalog into HotWax: one for newly created products and another for recently updated products. These were kept separate to maintain performance.
A newly created product in Shopify is defined as a new parent product with one or more variants. Adding variants to an existing parent product does not count as a new product in Shopify APIs.
To sync new variant details, updated images, or rich content for existing products, HotWax also needs to track product updates. However, the challenge lies in how Shopify marks a product as updated.
Each time an order is placed, Shopify updates the product’s inventory, marking it as an updated product. This creates significant noise, as most of these updates are inventory changes, not meaningful product updates. To handle this, HotWax had to parse every product update JSON to determine whether it contained real changes or not.
Because of this inefficiency, HotWax’s product update job could not run more than once every six hours. Each run would fetch thousands of products marked as updated by Shopify, using up valuable API rate limits, only for most of them to be overwritten in the OMS without any actual data changes. This resulted in unnecessary write operations and added load on the system.
If misconfigured, the update job could block the OMS bulk import queue from processing critical operations and even slow down order fulfillment.
2. The new architecture: Unified and efficient
The new product sync addresses performance bottlenecks at every stage
• Bulk query operations:
Uses Shopify’s bulk query APIs to fetch updated products. These APIs are designed for large-scale data extraction, ensuring that real-time API bandwidth remains available for transactional syncs.
• Diff calculator:
A dedicated diff calculator in the Shopify connector identifies which products have genuine updates and pinpoints exactly which fields changed since the last sync. This allows HotWax to process vastly more data in a fraction of the time.
3. How the new filtering process works
After converting the JSONL payload to standard JSON, the system stores each variant’s essential details in a ProductUpdateHistory log table. Each mapped field, such as price, tags, features, and identifiers, is stored both in its original form and as a SHA-256 hash. This ensures we can efficiently compare changes between syncs.
When a new batch of product updates is processed, the connector generates hashes for each field and compares them to the previously stored values. If the hashes differ, the connector flags that field as updated. This hash comparison approach is fast and resource‑efficient, it’s effectively a numeric comparison instead of a deep JSON diff.
The system also tracks variant changes through a variant set hash, a single hash representing all variant IDs for a given product. If the variant set hash changes, the system quickly identifies which variants were added or removed and records those details as structured diff data.
4. Fields tracked in variant diffs
To ensure complete visibility into changes, the following key fields are tracked when comparing variants:
All of these are stored in ProductUpdateHistory along with a hash comparison for each. A differenceMap JSON captures added, removed, or modified values, allowing the OMS to only process fields that have truly changed.
6. Results: faster, smarter, scalable
This design allows HotWax to skip unchanged products, process massive catalogs efficiently, and unify product creation and update jobs into a single, high-frequency sync in Job Manager. Retailers can now schedule this sync freely without risking API saturation or OMS delays.
For a deeper look at the architecture and technical implementation, see the full Shopify Product Sync engineering documentation.
