Private cloud for elasticity scale and zero downtime

Hi,

To be very honest, this was a POC and I personally enjoyed it very much.

Although, I am not allowed to disclose the concept as is, so sharing an abstract idea of building a cloud environment that could provide automated production ready environment.  All the deployments, configuration changes, nodes creation for traffic sharing will be dynamic in nature.

We had a team of 6 brilliant engineers for building up this POC. I was fortunate enough being a part of it.

Here we go...

PROBLEM TAKEN

Some background: This solution was mingled/cooked with more than 10 technologies. 

Problems targeted were hot deployment, zero downtime configuration changes, application switching contexts at run time, environment keeping track of machine health, creation of new virtual machines dynamically, new machines becoming part of load balancer dynamically and so on; wow, isn’t exciting?

PICTORIAL REPRESENTATION

This is what expected. Click here for enlarged image

TECHNOLOGY STACK

This is something fascinating. Enjoy the list J.

• vmWare ESXi Host Client / Server
• Hyperic
• Splunk
• Zookeeper
• Puppet
• C-Tier
• NoSQL,
• Queues
• Spring
• SVN
• Jenkins-Hudson
• Nexus

SOLUTION APPROACH

So what we did, huhh!!

How it works – Whenever a developer make any changes in the code base and commit to the SVN, Hudson first runs few plugins including test cases on the build.  Once the green is received from all plugins and test cases run successfully, a fresh build is prepared and copied to the Nexus repository.

C-Tier copies that build and copies it over all the instances of Application Server running over different hosting machines. Being dynamic in nature Application Server will deploy the new builds on the fly.  If there are any configuration changes then C-Ties pushes those changes to ZooKeeper server.  The moment ZooKeeper receives configuration changes it notifies to all of its clients programs running.

Now this is application responsibility to track down ZooKeeper client’s configuration changes and apply it to the code. 

After successful deployment of a build on application server, two hawks hyperic (system monitoring tool) and splunk (log monitoring tool) will keep eye on that instance of application server and will report to the base in case of any alarming situation.  Right Sir!! (In a military style J)

In case traffic is increasing on the application, hyperic will raise an alarm based on certain configured rules then C-Tier will initiate a puppet agent.

A puppet agent will copy a deployment ready vmWare image and creates new virtual machine.  New agents for splunk, hyperic and c-tier will be installed additionally on it.  And finally application server and the application will become part of that new virtual machine.  At last new virtual machine will be registered in to load balancer for traffic sharing. 

Other tasks c-tier performs: If application server goes down at any moment, then hyperic agent; based on configured rules; take action (in case run a script) to start the application server again.

That’s it.

Please let me know if you need any other help or more information.

Thanks

Shailendra