92% cost optimization for first-ever Active Content Engine

For a few years in a row, the client struggled with the burden of skyrocketing infrastructure charges caused by imperfect product architecture. The platform that was initially rushed into development to test hypotheses and attract early users unexpectedly evolved into a long-term commitment even though it wasn't ready for this.

Despite data mining being a big part of the product core, its functionality was built in a non-scalable manner. The absence of parallel processing, along with deep-first crawl and imbalanced web scraping, posed challenges to both product expansion and infrastructure enhancement.

With 50K users relying on stable access to real-time data daily, Syndigo is steadfast in its commitment to providing seamless services and top-notch user experience. Fully aware that even minor errors or interruptions could result in critical data loss, the client wanted to reduce the frequency of issues and downtime to an absolute minimum.

Tweaking the product in a way that saves an impressive chunk of the budget, the OpsWorks team reimagined the whole solution. Spider bots now swiftly complete tasks within 30 seconds before seamlessly transitioning to the next one. Built that way, they can handle interruptions easily, so unlock the ability to use SPOT instances and dynamic scaling. As a result, infrastructure spending fell from $1M/month to $80K/month.

With new core logic and Kubernetes adoption, the product gained the ability to scale quickly and efficiently. The number of spider bots is now dynamic and can be tweaked in response to the task volume, whether scaling up or down. This flexibility enhances the product's responsiveness to varying workloads and unlocks its growth potential.

Empowering the client to manage the product more strategically, OpsWorks has implemented automation for routine processes in both the production and testing phases. This approach unlocks possibilities for resource optimization and budget-saving initiatives within the team. Moreover, it significantly reduces time-to-market, improving customer satisfaction level.