Spicule - Data Processing Experts

The Customer


NASA JPL is a federally funded research and development centre in California, and Celgene Incorporated (CELG) is an American biotechnology company that discovers, develops and commercialises medicines for cancer and inflammatory disorders, based in Summit, New Jersey. These two organisations came together in 2015 to work on a large scale Genomics project.

“We know that Spicule did all the heavy lifting on this project, and that the platform wouldn’t have made it this far without them. Spicule are a talented group of developers and we wouldn’t hesitate to recommend them.” Dave Ballard
Associate Director

The Challenge

Spicule was contacted in 2016, midway through the initial deployment, and immediately tasked with two specific roles. The first of these was based on a complex data processing toolkit, OODT, which Spicule employees have extensive knowledge about. The second task was to create a flexible solution to deploy the platform quickly and easily, at scale and in the AWS cloud. For us to achieve our goals, we would have to break up the deploy chain into more manageable deployments.

The Solution

Spicule spent the first few months helping the team with their integration of the data processing platform. We also built  custom interfaces to make life easier for the data scientists when ingesting data into the platform. Once the initial job was complete, we focused on splitting the monolithic build up into more manageable chunks, and then designed a strategy to deploy the platform into the Cloud in a straightforward and repeatable manner. To allow for this we employed a number of cutting-edge containerisation solutions and build time enhancements.

Some of the Technologies We Used

The Result

The finished platform consisted of a multi-module Docker deployment that scaled across a large number of AWS based servers, built and controlled by a central build server. This allowed the Celgene systems administrators to deploy the application with the click of a button and provide nearly 100% up-time of the Genomics search solution for their data scientists.

Could We Help You?

Request a Callback