Cloud Listening: Stories of How to Derive Best Value from Cloud

cloud listeningIf you ever harken back to the good ole times of the pre-smartphone era, you would probably remember reading the newspaper on the way to work. Times have changed and how! We don’t have to wait until the paper hits the curb to find out what is in news. Instant news is here! You will never be more than a notification away from what’s trending thanks to the internet and the smartphone. The internet, the smartphone and let me add the cloud to the list, are all great technologies. One might say, they are fundamental to life as we know it (now).

While the cloud may not be as visible as the first two, it is expansive in its utility. What could a simple app that enables collaboration built by a startup and a complex genomic data processing application of a premier research institution have in common? They both use the cloud, and are well-served by it.

Advocating The Cloud

No! You read the title right. We are not doing a “10 Reasons you must get on the cloud now!” post. We know you probably have read enough of those already. But, as a potential cloud customer this is something you must do: keep your eyes open to the interesting ways that other businesses and institutions are extracting value from the cloud.

Case Study:1

Riding The Cloud Wave Like A Seasoned Surfercapture to cloud

How to get the best of everything on the table.

Who?

CaptureToCloud: Startup wanting to build a cloud-based collaboration app.

What were they building?

The app will allow people to capture and share digital content (webpages, email, images and documents) with others through social networking tools.

How they did it?

Option 1: Set out to rent infrastructure (IaaS) from a public cloud vendor.

Option 1 is thrown out the door. Why?

IaaS services are effort and time intensive. You need to have a large IT team in place to maintain the VMs.

They are a startup; they need Lean IT infrastructure.

Option 2: They look around a bit and find that signing up for a Platform-as-a-service (PaaS) solution would free them from IT administration tasks.

Search result: Google App Engine

BONUS! Google App Engine comes with Google’s highly scalable infrastructure.

Option 2 is IN!

Building on App Engine: Why it Worked

  • Zero ADMIN
  • Faster time to Market
  • Cost-effective scaling
  • More Value Add for customers
  • Intuitive dashboard for effortless monitoring of app performance
  • Quick Feedback and Service improvements
  • Quick release cycles                                                                                                                                                                                                                                                                                     CaptureToCloud was able to build and deploy new software releases in Flat 10 mins!
  • Easy refinement and product enhancement
  • Testing multiple versions of code in parallel

Deploying new features and testing the performance usability and user response is easy on App Engine

Best part: All of this can be done without taking down the existing version. App engine automatically and seamlessly moves users logged in on the existing version to the new release.  

  • Redundancy is built-in

CaptureToCloud used Google App Engine’s High Replication Cloud Datastore to build redundancy

  • Auto-scaling takes care of any usage spikes
  • Limitless scaling allows businesses to be aggressive in adding new customers (Bye-Bye, IT limitations).
  • Integration with Google Apps for a superior customer experience

What they Got?

  • 18000 downloads
  • Perfect user rating on chrome webstore, google apps marketplace, google and everywhere else!
  • Saved more than $70,000 on infrastructure costs
  • Saved staff costs of over $300,000 per year (on an in-house IT team)

Lesson: If you have a clear vision, the cloud is your ticket to success.

Case Study:2

Broad Institute of MIT & Harvard

Cloud Savings = $$$ for Research

Who?

Broad Institute of MIT and Harvard is a research institute running the genomics project. The institute is one of the largest genome sequencing centers in the world. They provide sequencing services and undertake large-scale genomic data analysis.

What were they building?

The Challenge: Large Datasets need to be processed. The volume will continue to rise.

Genome processing is time consuming and costly. DNA sequencing in Broad institute runs all through the year and produces terabytes of genomic data every day. The amount of data being generated is accelerating.

Rate of data production = 2X every 8 months

The Solution: Scalable cloud infrastructure which can store, process and analyze high volumes of data.

How they did it?

Step 1: Shift the largest production pipeline to the cloud

Mini-step 1 – Collaboration: Engineers at Broad and Google genomics collaborated on this one. The former learning about cloud computing architecture and the latter about genomic data science and genomic data engineering.

Mini-step 2 – Uncoupling data generation from data processing pipeline.

Mini-step 3 – Porting production pipeline to cloud by re-architecting the sequencing environment for the cloud.

Result: The Genome Analysis Toolkit (GATK) is completely ported to Google cloud platform (GCP).

Step 2: Shifting other production pipelines to GCP

Others in line for migration to cloud

  • Tools for RNA-seq
  • Tools for cancer genome analysis

Step 3: Debuting new services using the cloud

The scientific community at Broad Institute has created some great software tools to make the processing of large genome-datasets easier. These tools which served Broad Institute well, can now be made available to others through the Software-as-a-Service (SaaS) model.

What they Got?

$aving$

“Conventional Wisdom”

Cloud delivers less savings as you scale-up

Our Experience

Public cloud is cheaper than on premise systems.

Yes! Even if you are a high volume user

Mere running the pipeline on cloud will not deliver cost savings. If you have to reduce costs, you must put the cloud architecture to optimal use. Here is how the team at Broad did it:

Goal 1: Make GATK runs quicker and cheaper

What they did:

  1. Parallelizing computationally intensive steps. Eg: comparing DNA sequence with reference sequence
  2. Used Preemptible VMs

These steps reduced turn-around time 8-fold

Reduced cost of running the GATK pipeline by 20%

GOAL 2: Reduce Storage Costs

What they did:

  1. Used Google Cloud Storage Nearline for storing less frequently accessed data.

Most genomics users, access raw DNA sequences very infrequently. This is because they need it only when running a new algorithm or a new assembly of the genome.

By using Nearline and Data compression methods, storage costs were reduced by 50%.

Using GCP for both computational and storage needs, reduced the total cost of ownership for Broad institute relative to the on premise storage and processing costs.

GOAL 3: Easier Collaboration

What they did:

  1. In the scientific research community, collaboration is critical to success. An environment where data can be securely stored and analyzed will solve problems of multiple groups copying and paying for transmission and storage of the same data.

Cloud enables anytime, anywhere secure access to data. Secure Data sharing for large multi-group remote teams

The broad institute will soon transform from a

                                                    “Physical Hub”

                                                            to a

                                                     “Digital Hub”

cloud

Cloud: Where New Services Are Born

Take the case of Broad Institute, their on-premise model limited them to provision of Sequencing services. Migrating to cloud unlocked a new source of revenue for them: Data Services

Did we get you thinking? It’s definitely time to mull on…

What the cloud could do for you?

Murali Dodda is a Cloud Technology Specialist with over 15 years of experience. He graduated from the prestigious IIT Madras. Murali provides 'technology and business leadership' to startups and has overseen successful exits for several of them. He is currently leading a team of technologists at Bitmin, a hot new startup delivering cloud services. Murali uses his weekends to catch up on the latest developments in technology innovation, product development, and entrepreneurship domains. Being an enthusiastic blogger, he shares exciting developments & his experiences with designing & deploying cloud strategies through his blog. If you want an inside view of cloud deployment for real-world clients, follow this blog.

Leave a reply:

Your email address will not be published.

Site Footer