Enhancing a Data Modeling Tool for 300,000+ Database Users

Description

Brief results of the collaboration:

• The customer accelerated the delivery of lineage analysis, enabling its 300,000 users to track data pipelines/views.
• The company replaced legacy SQL parsers for 5 databases, doubling the speed of automatic model retrieval and ensuring easier and faster maintenance.
• The data modeling tool now can accommodate database projects with 10,000+ tables—due to a 20% reduction in memory consumption and other optimization efforts assisted by Altoros.
• The customer added connectors for 4 new databases, which resulted in acquiring new enterprise customers and bringing in additional revenues.
• Migrating some of the functionality to microservices contributed to improving maintenance.

The customer

Headquartered in the US, the company offers low-code data modeling software for creating and updating database structures via drag-and-drop diagrams. Supporting 10+ popular data platforms, the tool enables versioning, governance, lineage analysis, DataOps, project collaboration, etc.

The need

The customer had operated its product for several years and sought to speed up development, raising $2.5 million investment. Realizing the technical complexity of the system, the company faced the need for sourcing top-level data engineers capable of navigating advanced database structures.

Seeking assistance, the customer turned to Altoros for the .NET expertise. Satisfied with the delivered lineage analysis functionality, the company then asked for expanding database connectivity and other optimizations.

The challenges

The team had to address the following issues:

• For the third-party SQL parsers used within the platform initially, it took up to 6 months before they supported the new releases of database vendors, which limited connectivity.
• The implementation of low-code data modeling implied a deep knowledge of forward/reverse engineering.
• The system was expected to accommodate database projects comprising 10,000+ tables.
• Focus on security was also a must for a sophisticated data product of this scale.

The solution

Stage 1. Analyzing the customer's requirements and recognizing development needs in both America and Europe, Altoros provided engineering experts located in both regions.

Stage 2. After the developers at Altoros implemented the lineage analysis functionality, a dedicated software architect shaped the vision for new database connectors, while a business analyst helped to translate the stakeholders’ ideas into a prioritized backlog/specifications.

Stage 3. To address the issues with third-party SQL tools used for retrieving a database model, the architect built custom code parsers that worked 2x faster and became easier to maintain.

Stage 4. Achieving a 20% reduction in memory consumption and delivering other performance optimizations, the engineers ensured the platform can handle projects with 10,000+ tables.

Stage 5. Fostering security, the experts at Altoros implemented Cross-Origin Resource Sharing, encrypted sensitive data, updated libraries, and prevented code injections by verifying that user input did not translate into executable commands.

Stage 6. The engineers helped to address all the intricacies of gradual migration from .NET Framework 4.7 to .NET 7, as well as participated in transitioning to a microservices to boost efficiency, scalability, and maintainability.

Stage 7. Finally, the experts at Altoros offered further optimization recommendations—e.g., on the code-first approach to facilitate easier deployment and rollback, as well as DevOps, etc.

The outcome

Relying on Altoros for more than 2 years, the customer accelerated development, delivering features to 300,000+ end users faster. The company replaced 5 SQL parsers, doubling the automatic model retrieval speed, as well as introduced the support for complex database structures, accommodating projects with 10,000+ tables. With assistance from Altoros, the company enabled data modeling for 4 new databases, which boosted the acquisition of enterprise customers and resulted in additional revenues. Ongoing plans include the development of visual transformation modeling for data warehouses and expanding connectivity to more cloud databases.