Wednesday, December 3, 2014

Private cloud for elasticity scale and zero downtime

Hi,

To be very honest, this was a POC and I personally enjoyed it very much.

Although, I am not allowed to disclose the concept as is, so sharing an abstract idea of building a cloud environment that could provide automated production ready environment.  All the deployments, configuration changes, nodes creation for traffic sharing will be dynamic in nature.

We had a team of 6 brilliant engineers for building up this POC. I was fortunate enough being a part of it.

Here we go...


PROBLEM TAKEN
Some background: This solution was mingled/cooked with more than 10 technologies. 

Problems targeted were hot deployment, zero downtime configuration changes, application switching contexts at run time, environment keeping track of machine health, creation of new virtual machines dynamically, new machines becoming part of load balancer dynamically and so on; wow, isn’t exciting?


PICTORIAL REPRESENTATION
This is what expected. Click here for enlarged image





TECHNOLOGY STACK
This is something fascinating. Enjoy the list J.

  • vmWare ESXi Host Client / Server
  • Hyperic
  • Splunk
  • Zookeeper
  • Puppet
  • C-Tier
  • NoSQL,
  • Queues
  • Spring
  • SVN
  • Jenkins-Hudson
  • Nexus

SOLUTION APPROACH

So what we did, huhh!!

How it works – Whenever a developer make any changes in the code base and commit to the SVN, Hudson first runs few plugins including test cases on the build.  Once the green is received from all plugins and test cases run successfully, a fresh build is prepared and copied to the Nexus repository.
C-Tier copies that build and copies it over all the instances of Application Server running over different hosting machines. Being dynamic in nature Application Server will deploy the new builds on the fly.  If there are any configuration changes then C-Ties pushes those changes to ZooKeeper server.  The moment ZooKeeper receives configuration changes it notifies to all of its clients programs running.

Now this is application responsibility to track down ZooKeeper client’s configuration changes and apply it to the code. 

After successful deployment of a build on application server, two hawks hyperic (system monitoring tool) and splunk (log monitoring tool) will keep eye on that instance of application server and will report to the base in case of any alarming situation.  Right Sir!! (In a military style J)

In case traffic is increasing on the application, hyperic will raise an alarm based on certain configured rules then C-Tier will initiate a puppet agent.

A puppet agent will copy a deployment ready vmWare image and creates new virtual machine.  New agents for splunk, hyperic and c-tier will be installed additionally on it.  And finally application server and the application will become part of that new virtual machine.  At last new virtual machine will be registered in to load balancer for traffic sharing. 

Other tasks c-tier performs: If application server goes down at any moment, then hyperic agent; based on configured rules; take action (in case run a script) to start the application server again.

That’s it.
Please let me know if you need any other help or more information.

Thanks

Shailendra 

Tuesday, July 23, 2013

Using ElasticSearch to cache more than 4.6 million records and integrating with jQuery auto-complete highlighting


Hi,

The topic is really interesting, right :)
Buzz words in the topic “Using ElasticSearch to cache more than 4.6 million records and integrating with jQuery auto-complete highlighting” is ElasticSearch.  Let's go through it.

PROBLEM TAKEN
Some background: In one of my project there is requirement to show auto completion box to display company names.  These company names were stored in one of the oracle table and in count total companies records are more than 4.6 million.

It was expected to
-          Fetch the records from database; populate the elastic-search index using these records. 
-          Integrate the calls for elastic-search from jQuery using ajax.
-          Server hits as the user types, fetching data from elastic-search and populate in jQuery auto-complete box.
-          Highlight the data matching in auto-complete box.

PICTORIAL REPRESENTATION

This is what expected.



TECHNOLOGY STACK
  •           ElasticSearch
  •          jQuery


SOLUTION APPROACH
In the approach, there were multiple tasks that were completed.
1.       ElasticSearch
Elastic search is a “flexible and powerful open source, distributed real-time search and analytics engine for the cloud”.

I installed the elastic search 0.90.2 version.   

By default elastic search does not provide any way to populate its indices via any data source.  There is a plugin called jdbc-river (https://github.com/jprante/elasticsearch-river-jdbc).  River is basically a terminology used to populate indices with specific key.  There are other types of rivers available using which you can populate indices via CSV, Oracle, MySQL etc.

After downloading elastic search I set it up and then install the river-jdbc plugin (instructions are available on site).  Now using the river I executed below command to fetch the records from database and build indices.

PUT Body:
{
    "type" : "jdbc",
    "jdbc" : {
        "driver" : "oracle.jdbc.driver.OracleDriver",
        "url" : "jdbc:oracle:thin:@//[IP]:1521/[DB]",
        "user" : "user",
        "password" : "pwd",
        "sql" : "select col1, col2, col3 from table"
    },
    "index" : {
        "index" : "index_name",
        "type" : "type_name"
      
    }
}

You can put any string as index and type.  You can use any REST client from FF or Chrome to fire above query.  Just after successful execution, elastic search will start fetching the data and building indices.  In my case elastic search fetch 4.6 million records.  I specifically fetched 2 columns and third column was concatenation of first two.

Now elastic search is ready and populated.  I just need to hit the REST services of elastic search to get desired data. 


You can test your elastic search using below command
     Url: http://localhost:9200/index_name/type_name/_search
     POST Body
     {"from":0,"size":10, "query":{"field":{"COL1":"*760*" } } }


2.       jQuery
As a second part, jQuery need to be integrated.  I created one html page that has code to call elastic search REST services after user has typed 3 characters in one of the text box.  I call the records in the chunk of 10. 
After getting 10 matching records, process it to make a java-script array and pass it to auto-complete control. 
Then there is one hack or you can say MonkeyPatch (a terminology to override thirdparty’s function for custom behavior.) as a java script function to match the words with spaces in search result and highlight their all occurrences.

<html lang="en">
<head>
  <meta charset="utf-8" />
  <title>jQuery UI Autocomplete - Default functionality</title>
  <link rel="stylesheet" href="jquery-ui.css" />
  <script src="jquery-1.9.1.js"></script>
  <script src="jquery-ui.js"></script>
  <script>

var AUTOCOMPLETE_VIEW = 1;

function monkeyPatchAutocomplete() {

          // Don't really need to save the old fn,
          // but I could chain if I wanted to
          var oldFn = jQuery.ui.autocomplete.prototype._renderItem;

          jQuery.ui.autocomplete.prototype._renderItem = function( ul, item) {
              //var re = new RegExp("^" + this.term, "i") ;
              var searchStr = this.term;
              if (searchStr.indexOf (" ")!=-1)
              {
                  var re = new RegExp((replaceAll (" ", searchStr, "|")), "ig") ;
                  t = item.label.replace(re,"<span style='font-weight:bold;color:Blue;'>" + "$&" +  "</span>");
              } else {
                  var re = new RegExp(this.term, "i") ;
                  var t = item.label.replace(re,"<span style='font-weight:bold;color:Blue;'>" + "$&" +  "</span>");
              }

              return jQuery( "<li></li>" )
                  .data( "item.autocomplete", item )
                  .append( "<a>" + t + "</a>" )
                  .appendTo( ul );
          };
      }


function replaceAll(oldStrPattern, str, newStrPattern)
{
  var temp="";
  if(str!=null && oldStrPattern!=null)
   {
     var idx = str.indexOf(oldStrPattern);
     while (idx > -1)
     {
         temp += str.substr (0, idx);
         temp += newStrPattern;
         str=str.substr (idx+oldStrPattern.length,str.length);
         idx = str.indexOf(oldStrPattern);
     }
     temp += str;
   }
   return temp;
}

  jQuery(function() {

    jQuery.support.cors = true;
    monkeyPatchAutocomplete();

    jQuery( "#tags" ).autocomplete({

      source: function( request, response ) {
     
        var isDuns = false;
        var val = jQuery("#tags").val();
        var arr = new Array ();
        try {
            var i = parseInt (val);
            isDuns = !isNaN(i);
        } catch (e) {
            isDuns = false;
        }

        if (isDuns){
            val = "*"+val+"*";
        } else {
            val = "+"+val+"*";
        }

        if (AUTOCOMPLETE_VIEW==2)
        {
            var dt = {"from":0,"size":10, "query":{ "field" : { "COL2" : val } } };
            if (val.indexOf(" ")!=-1)
            {
                val = replaceAll (" ", val, "* +");
                dt = {"from":0,"size":10, "query":{ "field" : { "COL2" : val } } };
            }
        } else {
            //var dt = {"from":0,"size":10,"query":{"wildcard" : { "COL1" : val+"*" }},"sort":[{"COL1":{"order":"asc"}}]};
            var dt = {"from":0,"size":10, "query":{ "field" : { "COL1" : val } } };
            if (isDuns){
                dt = {"from":0,"size":10, "query":{ "field" : { "COL3" : val } } }
            }
            if (val.indexOf(" ")!=-1)
            {
                val = replaceAll (" ", val, "* +");
                dt = {"from":0,"size":10, "query":{ "field" : { "COL1" : val } } };
                isDuns = false;
            }
        }

        if (val != null && val.length > 2) {

            jQuery.ajax({
                type: "POST",
                url: "http://192.168.35.106:9200/index_name/type_name/_search",
                dataType: "json",
                async: false,
                data: JSON.stringify(dt),
                contentType: "application/json; charset=utf-8",
                success: function(data) {
                    //console.log(data);
                    var result=data;
                    var idx = 0;
                    var hitsJson = (data.hits.hits);                  
                    for (var key in hitsJson) {
                      if (hitsJson.hasOwnProperty(key)) {
                          if (AUTOCOMPLETE_VIEW==2) {
                            arr.push ((hitsJson[key]._source.COL2));
                          } else {
                            if (!isDuns){
                                arr.push ((hitsJson[key]._source.COL1));
                            } else {
                                arr.push ((hitsJson[key]._source.COL3));
                            }
                          }
                      }
                    }                                  
                    response(arr);

          }
                , error: function (xhr) {
                    alert ("err");
                    alert (JSON.stringify(xhr));
                }
            });
        }
    }, minLength: 3, delay: 300
    });
});

  </script>
</head>
<body>

<div class="ui-widget">
  <label for="tags">Tags: </label>
  <input id="tags" />
</div>


</body>
</html>

I used jQuery word instead of $ as in project multiple JS frameworks were used and conflicts were detected.

That’s it.
Please let me know if you need any other help or more information.

Thanks

Shailendra 

Monday, July 1, 2013

Spring security @PreAuthorize example and generic exception

Hi,

In my recent project I have working on securing spring methods exposed by controllers via @RequestMappings. 

PROBLEM TAKEN
I have to cross check the permissions/authorities assigned to the user who has logged-in to the system.  If the authority mentioned above the method matches then execution goes successfully else exception is throws.

TECHNOLOGY STACK
·         Spring
·         Hibernate
·         Tomcat
·         Maven
·         jQuery

SOLUTION APPROACH
I made the class that checks the user who is logged-in at present (I made the UserInfo object to be flowed in each request object after successful authentication.  For details please refer my blog http://sv-technical.blogspot.in/2013/07/spring-how-to-make-bean-object.html).

Ingredients to cook above code are:
Firstly I made my custom class extending org.springframework.security.access.PermissionEvaluator as BasePermissionEvaluator.  This class stores the object of UserInfo (refer blog mentioned above for details) and has below structure.


            BasePermissionEvaluator.java
public class BasePermissionEvaluator implements PermissionEvaluator
{
    @Autowired
    private UserInfo userInfo;
   
    /**
     * @return the userInfo
     */
    public UserInfo getUserInfo()
    {
        return userInfo;
    }

    /**
     * @param userInfo the userInfo to set
     */
    public void setUserInfo(UserInfo userInfo)
    {
        this.userInfo = userInfo;
    }

    @Override
    public boolean hasPermission(Authentication authentication, Object targetDomainObject, Object permission)
    {
        boolean hasPermission = false;

        if (authentication != null && permission instanceof String)
        {
            hasPermission = userInfo.hasPermission((String)permission);
        } else
        {
            hasPermission = false;
        }
        return hasPermission;
    }

    @Override
    public boolean hasPermission(Authentication authentication, Serializable targetId, String targetType, Object permission)
    {
        throw new RuntimeException("Id and Class permissions are not supperted by this application");
    }
}

Corresponding applicationContext.xml entries
<security:global-method-security pre-post-annotations="enabled">
    <security:expression-handler ref="expressionHandler" />
</security:global-method-security>
<bean id="webExpressionHandler" class="org.springframework.security.web.access.expression.DefaultWebSecurityExpressionHandler" />

    <bean id="expressionHandler"
        class="org.springframework.security.access.expression.method.DefaultMethodSecurityExpressionHandler">
        <property name="permissionEvaluator" ref="permissionEvaluator" />
    </bean>

    <bean id="permissionEvaluator"
        class=" BasePermissionEvaluator" />


If the exception is thrown by any of the method then handles it using below entries
<bean
        class="org.springframework.web.servlet.handler.SimpleMappingExceptionResolver">
        <property name="exceptionMappings">
            <props>
                <prop key="java.lang.Exception">/pages/common/error</prop>
            </props>
        </property>
    </bean>
Here you are supposed to create on error.jsp inside pages/common folder of you context root folder.


Below is one of the Controller’s methods secured via @PreAuthorize annotation.
    @RequestMapping(value = "service/user/list.do", method = RequestMethod.GET)
    @PreAuthorize("hasPermission('null','can-view-users')")
    public @ResponseBody
    List<User> list()
    {
        return userService.list();
    }


Error.jsp
<%@ page isErrorPage="true" %>
<!doctype html>
<html>
<head>
     <meta charset="utf-8">
     <title>vCare - for you</title>
     <link rel="stylesheet" type="text/css" href="<%=ctx%>/content/css/bootstrap.css" />
     <link rel="stylesheet" type="text/css" href="<%=ctx%>/content/css/bootstrap-responsive.css" />
     <link rel="stylesheet" type="text/css" href="<%=ctx%>/content/css/default.css" />
     <script type="text/javascript" src="<%=ctx%>/content/scripts/jquery-1.8.2.js"></script>
     <script type="text/javascript" src="<%=ctx%>/content/scripts/bootstrap.min.js"></script>
</head>
<body>
     <div class="header">
           <div class="wrapper row-fluid">
                <div class="span12">
                     <div class="logo">
                           <a href="#">
                                <img src="<%=ctx%>/content/images/logo-150-60.png" border="0" />
                           </a>
                     </div>
                </div>
           </div>
     </div>
     <div class="main-content">
           <div class="wrapper">
                <div class="error-wrap">
                     <h1>Error</h1>
                     <h3>Sorry, an error occurred.</h3>
                     <hr />
                     <p>Below are the details: </p>
                     <div class="alert alert-error">
                           <%=exception.getMessage()%>
                     </div>
                     <hr />
                     <a class="btn btn-success" href="<%=request.getContextPath ()%>/">Go Back</a>
                </div>
           </div>
     </div>
</body>
</html>

Please let me know if you need any other help or more information.

Thanks
Shailendra 

Spring - How to make a bean object available via @Autowired at any layer

Hi,

Recently in a project I faced a situation where there was a need that one java object needs to be accessed by different layers like Controller, Biz, DAO layers etc. 

PROBLEM TAKEN
I had a UserInfo object that contains the User information who has logged into the system and his authorities.  You can think it as ThreadLocal object containing user information and his authorities.  And I want to access it in most of the other spring component classes.

TECHNOLOGY STACK
  •       Spring
  •       Hibernate
  •       Tomcat
  •       Maven
  •       jQuery


SOLUTION APPROACH
To solve this problem I made one CustomSuccessHandler by extending org.springframework.security.web.authentication.SavedRequestAwareAuthenticationSuccessHandler.

This handler populates the User object in UserInfo class after successful authentication.  Below is the structure of UserInfo class.

            UserInfo.java
@Component
public class UserInfo
{
    private User ctxUser;
    ....

    /**
     * @return the ctxUser
     */
    public User getCtxUser()
    {
        return ctxUser;
    }

    /**
     * @param ctxUser
     *            the ctxUser to set
     */
    public void setCtxUser(User ctxUser)
    {
        this.ctxUser = ctxUser;
        populateGroupsAuth();
    }

    ....
    ....
}

You can place additional variables if required.  Here User is my own pojo class.  It has some fields and getters/setters.

Now to support
    @Autowired
    User ctxUser;

In any java file within Spring Context I include below entries in applicationContext.xml or you can do that in any of your context representing xml file.         

First way:
    <bean id="userInfo" class="UserInfo"
        scope="session">
        <aop:scoped-proxy />
    </bean>

    <bean
        class="org.springframework.web.context.support.ServletContextAttributeExporter"
        p:attributes-ref="servletContextAttributesMap">
    </bean>

    <util:map id="servletContextAttributesMap">
        <entry key="appCtx">
            <ref bean="userInfo" />
        </entry>
    </util:map>

Second way:
Assuming all your views are going via some controller that is intercepted by spring’s InternalResourceViewResolver
    <bean id="viewResolver"
        class="org.springframework.web.servlet.view.InternalResourceViewResolver">
        <property name="prefix">
            <value>/pages/</value>
        </property>
        <property name="suffix">
            <value>.jsp</value>
        </property>
        <property name="exposeContextBeansAsAttributes" value="true"/>
        <property name="exposedContextBeanNames">
            <list>
                <value>userInfo</value>
            </list>
        </property>
    </bean>

                In this case you can now access UserInfo bean at class level and use it.

Third way:
To allow a bean object to be available in your request object below is the configuration for the same class
    <bean id='userInfo' class='UserInfo' />

    <bean scope="request" factory-bean="userInfo"
        factory-method="getCtxUser">
        <aop:scoped-proxy />
    </bean>

Please do let me know if you need more information.

Thanks
Shailendra