Websphere Application Server heap size monitoring with wsadmin

In Middleware infrastructure, These two issues are very common.

1. Connection Pool Overflow
 2. OutOfMemory(OOM)

Both of them will lead the application to the rock bottom and application unavailability, So its always recommended to have on eye on both of these key factors, by having some kind of Run time Monitoring and alerting system tools. (Tivoli, Wily, Nagios etc..)

Sometime, We will not be having the sufficient monitoring tool to capture this OutOfMemory and Connection Pool Overflow.

That's when, we fall back into the efficient and easiest way,  which is scripting tools,  like WLST and WSADMIN etc.

Hope you have read our another post about "Connection Pool Overflow monitoring and Alerting System" if not click here

Now, here is the Jython script designed to help monitor the "HeapSize"  which can let us able to proactively measure and take action for OOM issue.

As mentioned earlier in our post, You can have some kind of scheduling tools like Autosys (or) Cron and run it in a frequent interval along with Shell/Perl script and generate an email (or) ticket.

Let me know if there is any help required in designing of Shell/perl (or) in the scheduling part.

Latest version of this code can be found here

TheCode

Running_JVMS=AdminControl.queryNames("*:type=Server,*").split(java.lang.System.getProperty("line.separator"))
ignorelist=['nodeagent','dmgr']
for JVM in Running_JVMS:
        ServerName=AdminControl.invoke(JVM ,"getName()")
        if ServerName not in ignorelist:
                JVMName=AdminControl.completeObjectName('type=JVM,process='+ServerName+',*')
                JVMObject=AdminControl.makeObjectName(JVMName)
                perf=AdminControl.completeObjectName('type=Perf,process='+ServerName+',*')
                perfObject=AdminControl.makeObjectName(perf)
                Obj=AdminControl.invoke_jmx(perfObject,"getStatsObject",[JVMObject,java.lang.Boolean('false')],['javax.management.ObjectName','java.lang.Boolean'
])
                current=Obj.getStatistic('HeapSize').getCurrent()
                used=Obj.getStatistic('UsedMemory').getCount()
                usage=float(used)/float(current)*100
                uptime=float(Obj.getStatistic('UpTime').getCount())/60/60/24
                print "--------------------------------------------"
                print "ServerName      :", ServerName
                print "uptime(in days) :", int(uptime)
                print "--------------------------------------------"
                print "CurrentUsage    :", current
                print "Usedmemory      :", used
                print "Usage in Percent:", int(usage)
                print "--------------------------------------------"

Save this with .py extention and run it with wsadmin

The Result

bash-3.00# ./wsadmin.sh -username wasadmin -password wasadmin -lang jython -f ./python_scripts/getHeapStatus.py
WASX7209I: Connected to process "dmgr" on node mwiCellManager01 using SOAP connector;  The type of process is: DeploymentManager
--------------------------------------------
ServerName      : mwi_server01
uptime(in days) : 13
--------------------------------------------
CurrentUsage    : 119936
Usedmemory      : 85675
Usage in Percent: 71
--------------------------------------------
--------------------------------------------
ServerName      : mwi_server02
uptime(in days) : 50
--------------------------------------------
CurrentUsage    : 483392
Usedmemory      : 319485
Usage in Percent: 66
--------------------------------------------

Hope it helps.

For any question related to this. Please feel free to reach us via comment.
Always you can write an email to us at admin@mwinventory.in

Cheers and Have a great and prosperous new year. Best wishes from teammwi.

Apache Tomcat Web Application Development with Form Based Authentication

Designing WEB Applications with Web Browser based security is very common.

There are three types of Authentication methods generally being used such as

1. Basic Authentication
2. Form Based Authentication
3. Client Cert Authentication

Most of the Web Applications are using Form Based Authentication method where  the user will be
submitting their credentials through the html form. Which will be further transmitted to the server over HTTP(SSL) as an additional security measure.

Now we are going to see how to develop a web application with Form Based Authentication.

Prerequisites

• Apache Tomcat Server 6 (or) above
• Editors like Notepad ++

Step1:  Create one directory named "TestSecApp"

Step2:  Under "TestSecApp" Create another directory named "WEB-INF"

Step3:  Copy All these files with their respective name under "TestSecApp" Directory

error.html

<html>
 <head><title>ERROR PAGE 404</title></head>
 <body>
<H1> CUSTOMER ERROR MESSAGE: THE PAGE YOU HAVE REQUESTED IS NOT FOUND </H1>
 </body>
</html>

fail_login.jsp

<html>

  <head>
  <link rel="stylesheet" href="style1.css" type="text/css" media="all">
    <title>Login failed</title>
  </head>
  <body bgcolor=#ffffff>
  <div id="login">
  
  <center><h2>Authentication Failure.</h2>
  
  <a href="login.jsp">Return to login page</a> </center>
  </div>
  
  </body>
</html>

login.jsp

<html>
  <head>
  <link rel="stylesheet" href="style1.css" type="text/css" media="all">
    <title>Security WebApp login page</title>
  </head>
  <body bgcolor="white"> 
  <div id="login">
  <blockquote>
  
  <h2>Please enter your user name and password:</h2>
  <p>
  <form method="POST" action="j_security_check">
    <div id="box1">  
      
      <p>Username:<input id="textbox1"  type="text" name="j_username"> </p>
      <p>Password:<td><input id="textbox1" type="password" name="j_password"> </p>
   <br>
   <input id="button1" type=submit value="Submit">
    </div>
  </form>
  </blockquote>
  </div>
  </body>
</html>

logout.jsp

<html>

  <head>
  <link rel="stylesheet" href="style1.css" type="text/css" media="all">
    <title>Welcome Back</title>
 </head>
 <body bgcolor=#ffffff>
  <div id="login">
  <center>
  <br>
  <br>
<%@ page session="true"%>

User '<%=request.getRemoteUser()%>' has been logged out.

<% session.invalidate(); %>
<br>
<a href="login.jsp">Click here to login again</a>
</center>
  </div>
  
  </body>
</html>

Test.jsp

<!DOCTYPE html>
<html>
 <head>
   <title>Browser Based Authentication Example Welcome Page</title> 
   <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>

   <script>
   $(document).ready(function(){
        
        $("#div1").animate({left:"200px", top: "200px" , right:"200px", bottom: "200px"});
});
</script>
 
 <style>
#div1
{
   position: absolute;
   left : 0px;
   background-color: #68A3B3;
   border: 2px silver solid;
   color: white;
   text-align: center;
   
  
}
</style>

 </head>  
<div id="div1">
<h1> Browser Based Authentication Example Welcome Page </h1>  

<h1> <p> Welcome <%= request.getRemoteUser() %>!  </h1>

<a href="logout.jsp"> Logout </a>

</div>

</body>

</html>

style1.css

#login
 {
   position: absolute;
   left : 400px;
   right: 400px;
   top: 200px;
   bottom: 200px;
   border: 2px silver solid;
   color: #389093;
 }
 h2
 {
 color: 389093;
 
 }
 
 #message
 {
  color: violet;
  
 }
 #box1
 {
   padding: 5px;
 }
 #textbox1
 {
   position: absolute;
   left: 140px;
  padding: 5px;
  width: 200px;
 }
 
 #textbox1:hover
 {
   position: absolute;
   left: 140px;
   padding: 5px;
   width: 200px;
   border-color: violet;
   color: black;
 }
 
#button1
 {
  background-color: green;
  color: silver;
  font-weight: bold; 
 }
 
#button1:hover
 {
  background-color: green;
  color: white;
  font-weight: bold; 
 }

Step4: Under "WEB-INF/classes" directory copy the below file

This  servlet is to print the session creation in the tomcat STDOUT. check the catalina.log once deployed to verify.

MySessionListener.java

import javax.servlet.http.HttpSessionListener;
import javax.servlet.http.HttpSessionEvent;
public class MySessionListener implements HttpSessionListener {
       private static int sessionCount = 0;

       public void sessionCreated(HttpSessionEvent se) {
               sessionCount++;
                // Write to a log or do some other processing.
        System.out.println("Session has been created");
        System.out.println("Current Session count is"+sessionCount);
        
       }
       public void sessionDestroyed(HttpSessionEvent se) {
                if(sessionCount > 0)
                         sessionCount--;
                     //Write to a log or do some other processing.
        System.out.println("Session has been destroyed");
        System.out.println("Current Session count is"+sessionCount);

           }
}

compile the code  using javac and make sure you are getting a class file getting generated  MySessionListener.class

"javac MySessionListener.java"

Step 5: create "web.xml"  (deployment descriptor) file under  "TestSecApp/WEB-INF"

<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" 
id="WebApp_ID" version="2.5">
<welcome-file-list>
 <welcome-file>Test.jsp</welcome-file>
</welcome-file-list>
    <error-page>
        <error-code>404</error-code>
        <location>/error.html</location>
    </error-page>

<security-constraint>

                <web-resource-collection>
                      <web-resource-name>Success</web-resource-name>
                      <url-pattern>/Test.jsp</url-pattern>
                     <http-method>GET</http-method>
                     <http-method>POST</http-method>
                </web-resource-collection>

                <auth-constraint>
 
             <description>
                 These are the roles who have access.
             </description>

                    <role-name>webuser</role-name> 

                </auth-constraint>

          
  <user-data-constraint>
            <description>
                This is how the user data must be transmitted.
            </description>
            <transport-guarantee>NONE</transport-guarantee>
        </user-data-constraint>


</security-constraint>

<login-config>
              <auth-method>FORM</auth-method>
              <form-login-config>
  <form-login-page>/login.jsp</form-login-page>
  <form-error-page>/fail_login.jsp</form-error-page>
       </form-login-config>
</login-config>

<security-role>
               <role-name>webuser</role-name>
</security-role>

<listener>
  <listener-class>MySessionListener</listener-class>
</listener>


</web-app>

Step7: Build war file using "jar -cvf" command like given below.

Step 8:  Update the "TOMCAT_HOME/tomcat-users.xml" - <tomcat-users> tag.


Here "mwiuser" is the user id that we are going to use and it has been mapped to "webuser". As per the configuration in web.xml. Any member of "webuser" group can login and access the secured resource.

Step9:  Deploy the Application to tomcat using the console  (i.e: localhost:8080/manager) (or) just copy the war file you have created to the "Tomcat_home/webapps/"

Step10: Restart the Tomcat  Server.

Step11: Access the deployed application using the url

http://localhost:8080/TestSecApp

Note*:  TestSecApp is the war file name which will be taken as a context root by default.

listing directories only from tar file

Recently I faced one requirement where i had to list only certain level of directories with "tar -tvf" (or) "jar -tvf" output.

Here is my sample tar file. where I need to list only two levels of files/directories.

bash-3.2# tar -tvf content.tar
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2a/
drwxr-xr-x   0/0        0 Dec 29 17:00 2015 direcotry2/directory2a/directory2a_media/
-rw-r--r--   0/0       33 Dec 29 17:01 2015 direcotry2/directory2a/directory2a_media/home.html
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2a/directory2a_business/
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2b/
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2c/
drwxr-xr-x   0/0        0 Dec 29 16:56 2015 directory1/
drwxr-xr-x   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/
drwxr-xr-x   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_media/
-rw-r--r--   0/0       33 Dec 29 17:00 2015 directory1/directory1_a/directory1a_media/media.html
-rw-r--r--   0/0       33 Dec 29 17:00 2015 directory1/directory1_a/directory1a_media/home.html
drwxr-xr-x   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_press/
-rw-r--r--   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_press/press.html
-rw-r--r--   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_press/welcome.html
drwxr-xr-x   0/0        0 Dec 29 16:55 2015 directory3/

The "awk" way:

Listing the parent directories alone:

bash-3.2# tar -tvf content.tar |awk -F/ '{if(NF==3) print}'
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/
drwxr-xr-x   0/0        0 Dec 29 16:56 2015 directory1/
drwxr-xr-x   0/0        0 Dec 29 16:55 2015 directory3/

Listing the Parent directories with its sub - directories(or) files

bash-3.2# tar -tvf content.tar |awk -F/ '{if(NF==4) print}'
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2a/
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2b/
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2c/
drwxr-xr-x   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/

Another level down

bash-3.2# tar -tvf content.tar |awk -F/ '{if(NF==5) print}'
drwxr-xr-x   0/0        0 Dec 29 17:00 2015 direcotry2/directory2a/directory2a_media/
-rw-r--r--   0/0       33 Dec 29 17:01 2015 direcotry2/directory2a/directory2a_media/home.html
drwxr-xr-x   0/0        0 Dec 29 16:58 2015 direcotry2/directory2a/directory2a_business/
drwxr-xr-x   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_media/
-rw-r--r--   0/0       33 Dec 29 17:00 2015 directory1/directory1_a/directory1a_media/media.html
-rw-r--r--   0/0       33 Dec 29 17:00 2015 directory1/directory1_a/directory1a_media/home.html
drwxr-xr-x   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_press/
-rw-r--r--   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_press/press.html
-rw-r--r--   0/0        0 Dec 29 16:57 2015 directory1/directory1_a/directory1a_press/welcome.html

Just change the "NF" value according to your requirement.

you can use this in case of  "jar -tvf" as well

Hope it helps. Comments welcome.

Websphere Connection Pool monitoring and Alerting System ( Connection Pool runtime information)

Many of the infrastructure monitoring tools  brings their own JMX monitoring now a days to monitor all the JVM components including connection pool (DataSource) JMS etc.

But, If in case you wanted to set  up your own alerting system (Email Notification etc) for Connection pool.  You can use this  wsadmin script.

Design

1. It will find all the running servers registered with DMGR (cell) and take a list of running connection pools targeted on those servers.
2. It will automatically set 80 percent of the connection max_limit as a threshold.
3. It will print a alert message, in case of possible connection pool overflow. When the current connection limit reaches the threshold.

If you do not want any alerting system and want to see the connection pool usage manually in case of requirement (or) on demand basis. you can just use this jython script with wsadmin. In case if you wanted to setup alerting system refer the additional notes for some ideas of designing the alerting system.

Additional notes

1. you can run this script in a frequent interval like 2 minutes (or) 10 minutes using the external shell script scheduled in crontab and get the alert generated. ( let me know in comment section if you need help) 
2. If you have Autosys, you can rather schedule it with JIL and run this in a periodic interval.
3. After scheduling it to run on certain interval you can use any log parse monitoring tools like Tivoli and get alerts/tickets upon your infrastructure set up.

Version compatibility

was6.1 and above ( for was 6 there is little bit modification required, write a comment if you need)

wsadmin script

import re
Running_JVMS=AdminControl.queryNames("*:type=Server,*").split(java.lang.System.getProperty("line.separator"))
ignorelist=['nodeagent','dmgr']
TMPFILE='/tmp/PoolContents.tmp'
global current_conn
for JVM in Running_JVMS:
        ServerName=AdminControl.invoke(JVM ,"getName()")
        if ServerName not in ignorelist:
                DS=AdminControl.queryNames('*:process='+ServerName+',type=DataSource,*').split(java.lang.System.getProperty("line.separator"))
                for entry in DS:
                        if AdminControl.invoke(entry, "getStatus()") != '99':
                                print "============================================"
                                print "ServerName    :", ServerName
                                DSN=AdminControl.invoke(entry, "getName()")
                                print "DataSourceName:", DSN
                                JNDN=AdminControl.invoke(entry, "getJndiName()")
                                print "JNDIName      :", JNDN
                                MaxConn=AdminControl.invoke(entry, "getMaxConnections()")
                                print "MaxConnections:", MaxConn
                                print "StatusCode    :", AdminControl.invoke(entry, "getStatus()")
                                ##########################
                                fout=open(TMPFILE, 'w')
                                data=AdminControl.invoke(entry, "showPoolContents")
                                fout.write(data)
                                fout.close()
                                percent=0.8
                                threshold=int(MaxConn)*float(percent)
                                print "Threshold     :", threshold
                                try:
                                        fin=open(TMPFILE)
                                        filedata=fin.readlines()
                                        for line in filedata:
                                                #match 1= re.search(r'\:\s\d', line)
                                                #matchstr=re.search(r'Total number of connection in shared pool', line)
                                                matchstr=re.search('(.....\s......\s..\s..........\s..\s......\s....):(\s\d)', line)
                                                if matchstr:
                                                        #current_conn=match.group().split(":")[1].strip(" ")
                                                        current_conn=matchstr.group(2)
                                                        if int(current_conn) >= int(threshold):
                                                                ALERT='YES'
                                                                ALERTSTRING='Connection Pool %r reached 80 percent of its max_limit on Server %r'%(DSN,ServerName)
                                                                print ALERTSTRING
                                                        else:
                                                                ALERT='NO'
                                                        break

                                                else:
                                                        current_conn=0


                                        fin.close()
                                        print "Currently used:", current_conn
                                        print "============================================"
                                except IOError:
                                        print  'Something went Wrong.'

how to run

Save the above code with ".py" extension and run it with wsadmin.

In example

wsadmin.sh -lang jython -username <username> -password <password> -lang jython -f <saved py file name>

Cheers,
Sara Velu

Follow us at  : https://www.facebook.com/middlewareinventory/

Install JDK7.0 websphere 8.5.5.0 -Unix CommandLine

Download and install Java 7.0, which is shipped with WAS v8.5.5 from Passport Advantage:


Note: Java 7.0 fixpacks are shipped with every WAS v8.5.x fixpack, butbase Java 7.0 must be installed first.

• Java 6 is the product's default Java SDK that is installed as part of the installation from the offering.
• Java 7.0 is a feature extension and can be downloaded from Passport Advantage or by using Live repository.
• Java 7.1 is a feature extension which can be installed on WAS v8.5.5.2. It is not available for Solaris and HP UNIX.

Refer to this document for the Java versions that is shipped with WebSphere Application Server


Once you have downloaded the archive files unzip them and install them using Installation manager

WS_SDK_JAVA_TEV7.0_1OF3_WAS_8.5.5.zip
WS_SDK_JAVA_TEV7.0_2OF3_WAS_8.5.5.zip
WS_SDK_JAVA_TEV7.0_3OF3_WAS_8.5.5.zip

List available packages first using Installation manager (imcl)

/opt/IBM/InstallationManager/eclipse/tools/imcl listAvailablePackages -repositories .

(here "." represents my present working directory, where I have un compressed the JDK files)

Install the JDK

/opt/IBM/InstallationManager/eclipse/tools/imcl install com.ibm.websphere.IBMJAVA.v70_7.0.4001.201405109_2203 -repositories . -installationDirectory /opt/websphere85/appserver -acceptLicense -showProgress -log installv8_jdk_install.xml

 Note*: we should use the <WAS_INSTALLATION_HOME> as the Installation directory. Otherwise we will encounter the following error.



Now the JVM is successfully installed into the machine.

                                         Switching SDK Between Profiles/JVMs

Installing the optional Java 7.0 or Java 7.1 does not imply that profiles can take advantage of this new version of Java.The managesdk command has to be used to switch Java (or) the WAS Admin Console. wsadmin can also be used.

List the available JDKs first

Use managesdk script to find the available JDKs.

./managesdk.sh -listAvailable -verbose

Currently I have 1.6_64 version installed and available.

In my case. I have a "Cell Profile" ,

In other words, I have a Dmgr Profile named "Dmgr01" with Federated Application Server profile named "AppSrv01"

Set Command Default

After setting the command default SDK to SDK 7, commands will use SDK 7 when issued from the bin directory under the WebSphere Application Server
installation root directory (<was_install_root>/bin


Now we have to set the JDK 7 for the profiles and its components like node and Application Server

I am using Managesdk to set the JDK7 as default for DMGR profile
Now we have to set for the Application Server Profile.
 
Node level
I am using wsadmin to see what is the Available SDK on Node.


If you are still not able to see the JDK7 under Node. You should first sync the node using syncnode.sh after stopping the nodeagent.

In my case, SOAP port of dmgr is 78003 .

Refer your serverindex.xml file of dmgr to know the SOAP port. You can find the file at



Use the syncNode.sh script available under AppSrv01/bin



Now come back again and check, whether the JDK7 is reflecting in Node level



Now we can set the JDK7 for the node level.



Check whether the DefaultSDK of Node points to Jdk7 using wsadmin



Now its available at Node level and you can set the JDK7 for all its Application Servers now. I use managesdk here.



Hope it helps.