Data Engineering Services

The customer

The project is a pyrosequencing system that enables high-resolution detection and analysis of biomaterial for genetic mutations.

The customer is a global provider of sample and assay technologies for molecular diagnostics, applied testing, and academic/pharmaceutical research. Its solutions help to transform biological material into valuable molecular insights. Headquartered in the Netherlands, the company operates 35 offices worldwide with 4,000+ employees, serving 500,000+ customers.

The need

The customer turned to Altoros to improve its biotechnology system that analyzes DNA samples for mutations in the early stages. The legacy tool was able to de-duplicate only 1,000 samples maximum — due to memory and CPU limitations — and it still took hours (or even days) to process the pipeline. The goal was to fix performance bottlenecks as well as enable linear scalability for processing 10,000+ biosamples at a time.

The challenges

Since the existing system was already operating on the superior hardware, vertical scaling was no longer an option.

The team was also challenged to identify the parts of the legacy code that would allow for parallel processing of DNA samples with Hadoop.

Finally, the system should have been seamlessly migrated to production.

The outcome

Altoros has delivered a highly scalable analytical system for de-duplication of genome samples - as a part of the customer’s analytical platform. Thousands of hospitals and laboratories worldwide use the system to detect DNA mutations, saving thousands of lives. The analysis takes minutes now, not hours; it allows for processing 10x more genome samples compared to the performance of the legacy system.

Altoros’s engineers have also proposed a reference architecture for updating a reporting solution. Inspired by our recommendations, the customer went on improving the system with open-source data analytics technologies, which will eventually allow for saving thousands of dollars on expensive Oracle BI licenses.

The customer

The customer is a global company providing affordable antenna solutions, eco-friendly energy saving software, and medical devices of the highest quality design. The company regularly performs researches to ensure conformity to the best standards before any device is manufactured.

Galtronics wanted to support the Green IT technologies by enabling users to access power consumption data without paper bills. The company turned to Altoros to build an energy management system that would be able to record, store, and manage petabytes of power consumption data and scale up from a single server to thousands of machines.

The need

The energy management system had to record and store petabytes of data. It had to start with collecting the electricity consumption data from devices and appliances located in 1,000 houses, which would result in 50,000 records written to the database every 15 seconds.

Every record had to be not only stored, but also processed to enable power consumption management by room, house, or device.

The challenge

The company planned to add 4,000 houses to the system shortly, which would result in 5,000,000 records sent to the database every 15 seconds. The customer’s plans to provide symHome to electricity companies across the US, so the system has to feature the ability to scale next to endlessly. Therefore it was needed to enable adding big amount of data that should be processed without delays, as well as, ensuring data storage and computing during a long period of the system operation.

The outcome

Monitoring energy usage enables saving from 5 to 15 percent on electricity bills. The system can be scaled next to endlessly and the number of houses that are connected to it grows.

Partnering with Altoros the customer managed to:

• save the efforts on keeping the system sustainability, as the architecture provides the possibility to scale in future being very cost-effective
• concentrate on creating new features or optimizing the existing ones reduce the equipment cost by 32% (due to utilizing free software, which doesn’t require special server hardware)
• reduce the equipment cost by 32% (due to utilizing free software, which doesn’t require special server hardware)

The application helps the company to achieve its Green IT objectives by continuously reducing the environmental impact and encouraging eco-friendly life-style.

The customer

The customer helps scientists and laboratories to conduct research and experiments in the field of life sciences. Their key services include next-generation sequencing, bioanalytical and mass spectrometry, as well as DNA sequencing. The customer turned to Altoros to develop a solution that would detect SNP in digitized DNA sequences saved in the FASTA/FASTQ format easier and less time-consuming.

The need

A common problem for researchers who work on genome analysis is the need to store and process terabytes of data fast. To address this issue, Altoros delivered an automated system for single-nucleotide polymorphism detection that provides better performance at a smaller cost. Deployed on Amazon public cloud, it was powered by Amazon Web Services and Amazon EMR. With this optimal solution our customer was able to process 150 GB of genome sequencing data within 24 hours and in the most cost-efficient manner.

The challenges

Apart from building an algorithm for detecting SNP, we were to determine what hardware configuration could provide the required data processing speed.

The outcome

Altoros delivered an automated system for singlenucleotide polymorphism detection that provides better performance at a smaller cost. Deployed on Amazon public cloud, it was powered by Amazon Web Services and Amazon EMR. With this optimal solution, our customer was able to process 150 GB of genome sequencing data within 24 hours and in the most cost-efficient manner.

We started with the development of a prototype to test the possible deployment options and make sure the functionality works correctly. The system for SNP detection was later installed on the customer’s private distributed infrastructure and data processing was performed with Apache Hadoop.

The customer

The company is a global provider of technologies, infrastructure, as well as vehicles for rail transportation. Its portfolio includes railway signaling, control, electrification, and automation systems. The customer also produces commuter, regional, high-speed, and intercity trains / locomotives.

The need

The company had a legacy system for monitoring railway crossings—tracking accidents and equipment malfunctions. However, the solution was designed as a monolithic app, which failed to scale and made it difficult to introduce new modules and functionality. So, the customer partnered with Altoros to achieve flexibility in maintenance and to sustain petabytes of data from multiple devices installed at the railway stations. Having achieved this, critical notifications about potential or happening accidents needed to be delivered in real-time.

The challenges

Under the project, the team at Altoros had to address the following issues:

• The app deployment required a lot of time-consuming manual steps, which slowed down the delivery of critical functionality into production.
• The IoT system needed to support a range of old devices installed at the stations—until they got replaced with the new ones.

The outcome

Collaborating with Altoros, the customer delivered a scalable solution for monitoring railway crossings and notifying about accidents or malfunctions in real time. With a microservices architecture, the company can now easily extend functionality without affecting the whole system. The solution can also sustain petabytes of data daily, processing megabytes per second. The system already gathers IoT data from nearly 5,000 edge devices installed at 2,500 railway crossings in the USA and Canada.

The customer

The company is a U.S. provider of currency hedging and exchange execution services to investors and financial institutions. Since its foundation in 2013, the customer raised $4.1 million in funding.

The need

The company had a platform that facilitated currency hedging and asset management to mitigate risks imposed by currency fluctuations. Nine global corporations were using the platform to offset potential losses on investments due to exchange rates. At some point, the customer recognized value in white-labeling its product. As part of this plan, the organization partnered with one of the leading financial holding companies in the world, growing its clientele to 95 corporations.

Each of these corporations had different processes, systems, and technology stacks that the platform needed to easily integrate and comply with. This required introducing a lot of customizations in an agile manner.

Relying on Altoros, the customer wanted to automate and speed up feature delivery. With the white-labeling strategy in force, the company also sought scalability, high availability, and ease of maintenance.

The challenges

Under the project, the team at Altoros had to address the following issues:

• As the platform analyzed sensitive financial information, ensuring proper data security was crucial.
• It was important to maintain high integrability of the product due to different internal processes, technology stacks, etc., of corporations using the system.

• 1 week

  from idea to production

• 2x–4x

  faster delivery

• 2x

  less infrastructure costs

The outcome

Partnering with Altoros, the customer was able to successfully execute its white-labeling strategy by offering a FinTech product that eases an adoption curve, features high integrability, and minimizes customization. The migration to a cloud-native PaaS that has a mature ecosystem of services enabled the company to reduce release time by 2x–4x, cut down expenses on infrastructure maintenance by 2x, as well as ensure scalability and high availability. Thanks to a variety of the implemented security measures, the organization can also address the primary concerns of the financial industry.

The customer

Based in the USA, the customer is a leading provider of data integration and business intelligence solutions. The company serves 38,000 enterprises in 100+ countries. Gartner lists the firm among the top suppliers of analytics and BI platforms for 10+ years in a row.

The need

To expand its global reach and a portfolio of services, the customer acquired a company that developed an ETL (extract, transform, load) suite. The product enabled enterprises to gather data from various sources (databases, third-party services, social networks, etc.). To fit in the existing ecosystem of tools, the suite had to comply with the workflows in place. For instance, it was vital to automate manual processes, such as the creation of custom connectors.

Before the acquisition, Altoros participated in the development of the suite. Satisfied with the results and our .NET development expertise, the customer decided to proceed with the partnership.

The challenges

Under the project, the team at Altoros had to address the following issues:

• As the connectors dealt with sensitive data, it was important to strengthen security.
• Under the hood, connectors utilized drivers from third-party vendors. Licensing for such drivers was both costly and time-intensive, increasing end-user budgets.
• A monitoring mechanism of the existing Salesforce connector logged the incidents without prioritizing them. This made it difficult to find the root cause of critical issues.

• 10+

  years of collaboration

• $100,000

  saved on drivers yearly

• 120+

  source connectors

The outcome

Partnering with Altoros for over 10 years, the company released new ETL connectors and automated the creation of custom ones down to just a few hours. The customer also ensured that the acquired suite of connectors integrated well into the existing ecosystem of proprietary tools. The delivered JDBC connector will promote the use of open-source drivers, saving hundreds of thousands dollars on vendor ones and simplifying licensing. The security measures in place helped to protect sensitive data of 38,000 enterprises.

The customer

Likefolio.com is an application based on the concept “invest in what you like,” which helps users to find potential investment opportunities through analyzing brand awareness in social networks. It aggregates conversations, status updates, likes, and check-ins from social networks into a proprietary database. Then it links these keywords and phrases to publicly traded companies and translates the data into investment ideas.

The need

SwanPowers, LLC turned to Altoros to apply best practices in cloud transformation Ruby development to their existing system in order to satisfy the high requirements to the level of concurrency, load, response times, etc.

The challenge

The Web site was based on a distributed architecture that was not able to scale as expected. In addition, there were several issues with DB contention and background jobs.

It was estimated that the solution would have to serve 10,000 users simultaneously with at least 100 new user registrations per minute.

Every new user would need to view a partial portfolio right after signing up. The system had to interact with APIs of multiple social networks and remain within the limits of request quotas. It also had to apply complex business logic to extract investment information from social network data.

Additionally, LikeFolio profiles of every existing user had to be synchronized with all the latest updates on their social networks.

The outcome

Thanks to the improvements introduced by our team, the customer was able to launch several marketing campaigns without being afraid of performance issues due to increased traffic. The application’s overall uptime and performance have been improved significantly.

Taking into account the LikeFolio’s focus on social networks, Altoros made sure that it complies with the required API quota/limits.

The customer

Normally, e-mail service providers limit the size of data sets that can be sent at a time. In addition, users have to employ third-party software to create documents with complex structures, multiple fields, diagrams, and tables. The customer is a provider of advanced document exchange services who designed software that eliminates these drawbacks. The new system simplifies the process of creating documents for various types of businesses and industries. Users can exchange documents, preview correspondence, create custom templates for documents and letters, assign various access restrictions, review, and mark up documents.

The system was deployed in a cloud environment to provide a scalable data storage and ensure high availability of all services.

The need

The customer is a start-up company that came to Altoros with an idea to develop a service similar to “virtual FedEx” that delivers electronic documents. The main advantage of the emerging system was to be a simple and intuitive user interface. Ease of navigation and availability of the planned features were tested on a prototype developed by Altoros.

The challenge

The system had to store a large number of documents and provide fast access to any of them. It was decided to use cloud technologies to ensure high availability, quick response times, and easy horizontal scalability. The system was integrated with EMC, FileTransfer services, and a third-party Reporting Tool selected by Altoros’s experts. The system had to be integrated with local document management solutions of the customers.

The outcome

The customer released a new solution that makes document exchange easier and more efficient. The system has already become rather popular, thanks to a user-friendly interface, smart navigation, enhanced security, and unlimited storage space in the cloud. By testing prototypes, we were able to carefully study all requirements, confirm functionality, and access usability before we started development.