1

Building a Query Microservice to Process Millions of Events per Minute

Google Cloud
Java
Node.js
JavaScript
Information Security

Description

A cybersecurity provider turned to Altoros to build a scalable data query microservice, ensure stable performance, and enable data visualization for its platform.

Brief results of the collaboration:

  • The company developed a data query microservice—supplying data for app behavior analysis, risk prioritization, microsegmentation, and anomaly detection—in just two months.
  • Thanks to the chosen architectural approach and tooling, the microservice is capable of processing 6.8 millions of events per minute without impact on performance. It also lays the foundation for future scaling.
  • A comprehensive UI enables users to visualise multiple analysis findings in a single chart.
  • The startup was able to deliver new features every 2–3 weeks. This allowed to get early feedback and test feature feasibility on the market before showcasing the product to investors. It is expected that series A funding will gather up to $100 million for further development.

The customer

Based in the USA, the customer operates at the intersection of data science and cybersecurity. The company has a flagship platform designed to prevent data leaks and security breaches by monitoring existing infrastructure, detecting anomalies, and accessing risks.

The need

A transition from monoliths to microservices, cloud, and software as a service created numerous connections between app components. The proliferation of API development only increases the number of connections prone to vulnerabilities. In an enterprise system, monitoring these connections, which start at dozens, is anything but a trivial task. Using its expertise in data science and cybersecurity, the customer built a prototype of a platform that monitors API behavior, data flows, and traffic patterns to identify / prioritize risks, as well as detect anomalies. Relying on Altoros, the startup wanted to build a data query microservice and develop a user interface for data visualization. With an upcoming funding round, there were tight deadlines for delivery.

The challenges

Under the project, the team at Altoros had to address the following issues:

  • As a data query microservice would aggregate massive massive arrays of data, it was important to ensure its uninterrupted, stable performance.
  • The changing risks—identified in infrastructure connections in real time—had to be instantly mapped on the dashboard and prioritized by severity.
  • The system monitored infrastructure connections across 300 metrics. For user convenience, multiple metrics visualized in a single chart must not overlap.

The solution

Developers at Altoros were engaged in front- and back-end development, as well as took on project management responsibilities. Our team was actively involved in creating a product roadmap, prioritizing backlog, and outlining recommendations for implementation. Following best practices of Agile development, engineers at Altoros were able to deliver new functionality every 2–3 weeks.

In two months, our developers created a data query microservice, which aggregates information for app behavior analysis, microsegmentation, risk prioritization, and anomaly detection. To find an optimal variant for processing massive arrays of data, the team at Altoros researched and tested a number of big data and analytics tools. As a result, the engineers implemented a pipeline based on Apache Kafka and Apache Druid. To ensure stable performance and fault tolerance, developers at Altoros built an event-driven architecture. Thanks to all this, the microservice is capable of ingesting over 6.8 million events per minute (at peak times, it can process up to 300,000 events per second).

Our developers delivered algorithms that identify risks and classify them as low, medium, and critical. This data is mapped on a dashboard with a 1–100 score assigned to each risk. The engineers also provided a possibility to monitor how the severity of risks changes over different time periods (30 minutes, 2 hours, and 24 hours).

To properly visualize hundreds of metrics, our team worked out a comprehensive UI architecture. In addition, engineers at Altoros made it possible to sort the analysis findings by 86 filters. This way, a user can visualize multiple insights in a single chart.

6.8M

events ingested per minute

2–3

weeks for feature delivery

300

metrics

The outcome

Partnering with Altoros, the customer delivered a data query microservice serving business-critical processes in its product in just two months. Thanks to the chosen architectural approach and tooling, it became possible to process 6.8 million events per minute (at peak times, it can process up to 3x more) without impact on performance. This lays the foundation for future scaling. The startup was able to deliver new features every 2–3 weeks, which provided an opportunity to get early feedback and test their feasibility on the market. This way, the startup was able to build up the functionality to showcase to investors. It is expected that series A funding will gather up to $100 million for further development.

Technology stack

Platform

Google Cloud Platform

Programming languages

Java, JavaScript, TypeScript

Frameworks and tools

React.js, Node.js, Express.js, Apache Kafka, Apache Druid, Sequelize

Database

PostgreSQL

You May Also Like

Automation of In-field Job Planning and Performance Optimization
Java
JavaScript
PostgreSQL
Information technology
Marketing
Call Recording, Analytics, and Workforce Optimization Solution
.NET
jQuery
C#
JavaScript
MS SQL
Information technology
Highly Scalable System for DNA Analysis
Hadoop
Java
Information technology
Healthcare
Sport
A Highly Secure Smart Home System Wins a Kickstarter Funding
Ruby
Ruby on Rails
JavaScript
Angular
PostgreSQL
MySQL
Information technology
The Image Recognition System
Java
MongoDB
NoSQL
e-Commerce
Integrated logistics solutions to the offshore industry
Android
LikeFolio: Best Practices of Cloud and Ruby Development for Application Optimization
NoSQL
MySQL
Ruby
Ruby on Rails
Marketing
Social media
Telecommunications
Finance
Data-Driven Analytics
Software for Selecting and Mixing Paint
.NET
MS SQL
C#
WP
Information technology
Retail
Software Suite for Mobile Technicians and Field Service Management
.NET
MS SQL
iOS
Android
Logistics and transportation
The System for Emergency Control Centers
.NET
C#
MS SQL
Healthcare
Sport
Logistics and transportation
The Cloud-based Document Exchange System
Java
jQuery
NoSQL
Information technology
e-Commerce
The Marketing Information Messaging System
.NET
C#
MS SQL
iOS
Marketing, Social media
Telecommunications
The NuoDB Migrator for Moving SQL Data to a NoSQL Database
Java
NuoDB
MySQL
PostgreSQL
Information technology
Manufacturing
Toyota Automates Its System for Holding Tenders
.NET
C#
Manufacturing
Warehouse Workload Monitoring Application
.NET
C#
MS SQL
WP
Logistics and transportation
Web-Based Personal Styling
Ruby
Ruby on Rails
JavaScript
jQuery
MySQL
Social media
e-Commerce
Web-Based System for Retailers
Ruby
Ruby on Rails
MySQL
MongoDB
Retail
e-Commerce
A Blockchain-Based Platform for Automating Bond Issuing Worth $10M
Bash
JavaScript
Blockchain
Finance

Contact us

Contact us and get a quote within 24 hours

Headquarters

Toll-free