Capacity Planning for J2EE applications

As per Wiki, Capacity planning is the process of determining the production/serving capacity needed by an organization to meet changing requirements for its products/solutions.  This process strategically assess requirements for new solution, additional network capacity and underlying IT architectures. The information provided by capacity planning helps to : 

• Characterize the solution workloads more accurately
• Analyze the performance of various modules
• Model contention for application servers, and ensure scalability
• Model and plan for communications infrastructure
• Forecast and cope with peak demand
• Project the impact of agent technologies and non-PC devices.

Capacity planning is more cost-effective and efficient if done prior to deployment. Performance problems resulting from a lack of capacity are more complex and costly to resolve after deployment. However, in post deployment scenarios, it is possible to identify the impact on:

• Code changes required
• Existing Java Heap footprint
• Native Heap footprint
• CPU utilization or other negative side effect

Benefits of this exercise:

• Avoid losing customers due to site crashes
• Performance modeling and capacity planning for infrastructure
• Build and analyze customer behavior models
• Plan to avoid frequent upgrades and migrations.
• Identify potential bottlenecks in the architecture

There are various methodologies and proven theories available to conduct this exercise. Some of them are:

• Discrete-event simulation, 
• Mean value analysis of product-form net-works, 
• Analytical identification of bottleneck resources in multiclass environments, and
• Workload characterization with fuzzy clustering.

Above methodologies in detail are complex and out of scope to discuss here. Instead let us look at the easier way to understand the whole process.  

The first and foremost step here is  to understand the IT requirements as explained below.

 

Data Gathering :

To properly determine resource requirements for an application, it requires architectural information, along with a functional description of anticipated usage. The completeness and accuracy of the sizing depends on the quality of the information received. When portions of  information are unknown or missing, the risk factor for incorrect sizing increases. Following type of information are required to drive the capacity planning analysis:

• Percentage of new function supplied by solution
• Percentage of new data elements created
• Business-use scenarios
• Data transferred to and fro
• Peak load users size
• Solution architecture definition:
• Business function, scenarios and supporting models
• Data architecture, solution architecture — business and deployment architecture diagrams
• Technical architecture and schematic (client, server, network, Web)

Once the information is collected in standard templates,  it is time to apply right calculations considering each and every criteria. When standard benchmark data is available, the analysis need to be performed and results gets documented.

The components that need to be considered or validated are given below. This might not be the complete list and depending upon the IT system in design it could differ.  

1. Operating systems
2. Application servers
3. Network protocols
4. Data access services:DB systems
5. Programming languages
6. UI/Client Frameworks: AJAX, Java scripts etc.
7. Distribution services: NFS, DFS, Kerberos
8. Systems management: SNMP, AntiVirus, ADSM, TME
9. Application interface with legacy data/systems
10. Peak load: Data throughput.

 Afterthe initial assessment of architecture sizing has been completed, it is time for coding and application development. Once the development is over and solution is in deployable stage, test-based sizing process can be started. This process provides the validation sizing analysis that need to be performed and results are documented in this stage.

Hope this helps to start with and I'm thinking to post some case studies and sample reports in my next article.

Websphere and OSGi woes with Java mail/Commons email attachments

Ever faced below issues sending emails in web/enterprise applications with OSGI framework deployed in Websphere?  Such an easy and straight forward logic which works everywhere else in all other application servers like Tomcat, Jboss and Weblogic but only fails in Websphere. Spent some time figuring out this and thought of sharing the solution I found to help others.

• Not able to send emails with attachments and facing error like no object DCH for MIME type */*
• Not able to send attachments but only email
• Error like : javax.mail.messagingexception java.net.sockettimeoutexception read timed out

Luckily we have solutions for these issues. In my case, I had a custom OSGi bundle which makes use of mail-activation-wrapper-1.0 and commons-email-1.2 bundles in same OSGi container to send emails with attachments. When the ear containing this deployed in Websphere it started failing with mime type not finding error. Also found other two errors given above in the fixing process. Finally could able to fix this following simple steps given below.

To start with, it is required to temporarily change class loader to point to the Session class loader of javax.mail as shown below. Also, make sure to reset it back using finally block.

public void sendMail(){

ClassLoader tcl = Thread.currentThread().getContextClassLoader();

try {

Thread.currentThread().setContextClassLoader(javax.mail.Session.class.getClassLoader());

// Call java mail or commons email API...

} finally {

Thread.currentThread().setContextClassLoader(tcl);

}

}

Next step is to manually add mime types to MailCommandCap object. Here, the OSGi bundle which calls Java mail API,  which in turn makes use of javax.activation, fails to locate META-INF/mailcap resources bundled as part of mail bundle resulting in UnsupportedDataTypeException.

MailcapCommandMap mc = (MailcapCommandMap) CommandMap.getDefaultCommandMap();

mc.addMailcap("text/html;; x-java-content-handler=com.sun.mail.handlers.text_html");

mc.addMailcap("text/xml;; x-java-content-handler=com.sun.mail.handlers.text_xml");

mc.addMailcap("text/plain;; x-java-content-handler=com.sun.mail.handlers.text_plain");

mc.addMailcap("multipart/*;; x-java-content-handler=com.sun.mail.handlers.multipart_mixed");

mc.addMailcap("multipart/mixed;; x-java-content-handler=com.sun.mail.handlers.multipart_mixed");

CommandMap.setDefaultCommandMap(mc);

Endorsed Approach

The other easiest but not so clean approach is to add "endorsed" folder in websphere/appserver/java/jre/lib and place the activation and mail jars there. This would be better in case if java class modification is not possible or required.

Hope this helps!

SoapUI issue

Facing this below error with SoapUI tool and struggling to resolve it. Here is the solution...

SoapUI ERROR:java.net.ConnectException: Connection refused: connect

Throws above error when some webservice request is being made after successfully generating mock requests from WSDL. I was unable to understand when the WSDL is hit and mock requests are generated properly, why this error appears from mock request. After struggling for a while find out the reason as given below.

Solution steps:

1.       Double click on “xxxPortBinding” from left menu which holds all the mock requests.

2.       Click on “Service EndPoints” to see if EndPoint URL formed is correct. In my case it was omitting context path in the URL field. 

Strangely, SoapUI tool forms new URL to hit omitting the context path. Could be a defect in this tool.

Also, for bad credentials error, add credentials from clicking “Auth” menu and add username and password details there.

Tips on Websphere AS 7

Thought of consolidating couple of issues I faced in making my EAR &WAR files work in WAS 7. Sure, it could save some time in analysing and finding right solution to most of the novice users.  

Application with WebServices - IBM default WS Disabling

First and foremost, application embedded with web services APIs and especially has client logic to invoke  web services, it may fail to work when deployed in WAS 7. This is because WAS 7 comes bundled with Apache Axis 2 and takes control of any web services call from the application deployed and ignores the API provided inside application. To overcome this, follow below steps:

1. Apply latest fix pack (I have tried FP 21) 
2. While deploying the EAR file, option "Classes loaded with local class loader first (parent last)” should be selected in both “Class loading and update detection“ and “Manage Module” sections.
3. Disabling default IBM JAXWS engine - To disable at global (server) level, follow below steps:
1. From admin console add a custom property to disable default engine. Go to : Servers > Server Types > WebSphere application servers > server1 > Java and Process Management > Process definition > Java Virtual Machine
2. Add below property in “Generic JVM arguments” field:-Dcom.ibm.websphere.webservices.DisableIBMJAXWSEngine =true
4. To disable only for particular application 
5. 
   
1.   Add below entry in manifest.mf file of war files (This happens at application level)   DisableIBMJAXWSEngine: true

In case of either of the above steps not working for some reason, follow below step of modifiying the default provided axis2 jar file.

Open '<WAS_HOME>/plugins/org.apache.axis2.jar' and remove 'META-INF/services/javax.xml.ws.spi.Provider'

   5. After above changes restart the server. 

Resolving CXF problem 

CXF in WAS 7 with ClientProxy component fails with below error making one scratch head to figure out solution!

java.lang.ClassCastException: org.apache.axis2.jaxws.client.proxy.JAXWSProxyHandler incompatible with org.apache.cxf.frontend.ClientProxy

Code which fails: 

MyWebService ss = new   MyWebService (wsdlURL, SERVICE_NAME);
instance = ss.getTestHttpPort();Client cxfClient = ClientProxy.getClient(instance); 


This issue occurs only in WAS 7 for CXF when generated class used (from WSDL2Java tool) for creating proxy. JaxWsProxyFactoryBean comes to rescue in such scenarios and to resolve this issue.

With JaxWsProxyFactoryBean for creating JAX-WS proxies, slight modification as given below need to be done the codebase:


JaxWsProxyFactoryBean factory = new JaxWsProxyFactoryBean();
factory.setAddress(wsdlURL);
factory.setServiceClass(MyWebService.class);
MyWebService port = (MyWebService) factory.create();
//and then call ClientProxy as usual but with the object created using JaxWsProxyFactoryBean
Client client = ClientProxy.getClient(port); 


Heap setting change never brings up the server

Heap size change from admin console calls for restarting the WAS but sometimes when the input crossed limit or some mess up happened, sever never comes up. Novice users like me spend hours together to fix this horrid problem. Hope this helps bothered developers like me.

When server is not starting because of your wrong input, just look at this file.

profiles

 AppSrvX

   config

      cells
        cellX
          nodes
             nodeX
               server
                server.xml 

Find jvmEntries element in this xml which will have initial and maximum heap size attributes and change them back to original/proper value.

JDBC Driver Error with MSSQL DB

Encountered this JDBC driver error when my multiple threaded client was inserting records to the MSSQL DB. 


java.sql.SQLException: Transaction (Process ID 59) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:368)


The quick way of solving it is to modify SQL server settings. From SMSS,  advanced settings can be configured to help prevent unintentional issues such as a query that causes locking or blocking. 


To access these settings in SSMS, choose Tools -> Options…  ->  Query Execution -> SQL Server -> Advanced dialog box.


1. Change SET TRANSACTION ISOLATION LEVEL to READ UNCOMMITTED :
This will minimize the impact of your ad-hoc queries by allowing dirty reads. While this can be beneficial for many production environments, make sure to understand the implications of this setting before implementing.


2. Change SET DEADLOCK_PRIORITY to Low :
This will tell SQL Server to select your session as the victim in the event of a deadlock.


3. Change SET LOCK TIMEOUT to a smaller, defined value, such as 30000 milliseconds (30 seconds):
By default, SQL Server will wait forever for a lock to be released. By specifying a value, SQL Server will abort after the specified timeout period when a lock is encountered. 


Though, this solution from server side solves the above issue, it is always not recommended as it could applies dirty read setting to all the DB instances and other transactionally important application may behave odd due to this.

CXF Embedded Application in JBoss-6.x AS

The CXF libraries stuffed in web applications lib folder to enable webservices capability usually works fine with servers like Tomcat (at least till version 6) which has by default no CXF related libraries at server path. But same web application fails even to get deployed in JBoss6.x conflicting with server provided CXF libraries.

When I faced the same issue, it took solid 8-10 hours to find out working solution. Hope my findings helps someone in same despair.

This issue made me take serious look at the class loading behavior of JBoss and I was certain that JBoss would have feature to disable or yield parent (server) classes loading. I have seen the same feature in Websphere application server. With full confidence, I have tried JBoss recommended ways of adding jboss-app.xml, jboss-classloading.xml and jboss-scanning.xml to web application path to make it look for classes from its own lib folder first.

I have tried giving the jboss-app.xml file different names like jboss-app.xml, jboss-web.xml and jboss-.xml etc. to see it work. I was thinking that I was missing correct syntax here and tried many options in vain.

Similarly tried many options with jboss-classloading.xml and I was sure about the following element and unfortunately, this also did not solve the problem.

 I knew surely that the above approach is cleaner and recommended way to go for which was not yielding any result.

As a last resort, I have physically removed all the CXF libraries from server end itself. I have found some cxf related files at common/lib and removed all cxf*.jar from there. Also, from server/default/Deployers/jbossws.deployer directory removed all jars. After these changes, when server was restarted some errors in boot.log appears related to the missing cxf jars but the web application got deployed successfully and functionality was intact.

Probably, it is good to use Jboss 7 as it claims to provide flexibility in choosing webservices kit (either cxf or native) and choosing native kit could resolve cxf classloading issues. Never tried though!


Please take note that this could stop other applications (if any) which depends on JBoss provided CXF libraries! 

Managing JVMs in RHEL

Java fraternity new to Linux environments usually face the trouble of managing JVM instance in terms of upgrading or migrating them. Some time back, when I wanted to remove the old JVM instance, I too faced this issue and hope this solution I found, would help others save some time.

Below step by step approach would help the beginners:

1. Download rpm.bin file
2. run : ./*.bin
3. Follow below steps to remove the old java and add new java into environment

Re configuring the default Java configuration for Red Hat / Fedora

This is done as root or equivalent.

First remove /var/lib/alternatives/java file by typing;

rm /var/lib/alternatives/java

When asked press the 'y' key,

Now to create the new (corrected) alternatives file for java type
the following commands as root (modify for jdk as needed);

/usr/sbin/alternatives --install /usr/bin/java java /usr/lib/jvm/jre-1.4.2-gcj/bin/java 1

/usr/sbin/alternatives --install /usr/bin/java java /usr/java/jre1.6.0/bin/java 2

/usr/sbin/alternatives --config java

You should now see for example:

There are 2 programs which provide 'java'.

Selection Command
-----------------------------------------------
1 /usr/lib/jvm/jre-1.4.2-gcj/bin/java
*+ 2 /usr/java/jre1.6.0/bin/java

Enter to keep the current selection[+], or type selection number:

Type:

In the example above java is already configured correctly [*+ 2] to use Sun's Java, no changes are needed, just press the Enter key here. If you have been following the instructions then you should have the same results (version numbers may be sightly different).

Now type; /usr/sbin/alternatives --display java

You should see for example;

java - status is manual.
link currently points to /usr/java/j2re1.5.0_09/bin/java
/usr/lib/jvm/jre-1.4.2-gcj/bin/java - priority 1
/usr/java/jre1.6.0/bin/java - priority 2
Current `best' version is /usr/java/jre1.6.0/bin/java.

Next you might want to create (or edit) /etc/profile.d/java.sh
file, example below;

export JAVA_HOME="/usr/java/jre1.6.0/bin/bin"
export JAVA_PATH="$JAVA_HOME"
export PATH="$PATH:$JAVA_HOME"

When done creating or editing the file type;

source /etc/profile.d/java.sh

Now any user root or other wise should be able to use the command;

which java

and the results should read something like;

/usr/bin/java

Also any user root or other wize should be able to use the command;

java -version

and the results should read something like;

java version "1.6.0"
Java(TM) SE Runtime Environment (build 1.6.0-b105)
Java HotSpot(TM) Client VM (build 1.6.0-b105, mixed mode, sharing)