Solution to JBoss Wildfly 8.0.0 Hanging

Ever faced Wildfly 8.0.0 server hanging while load testing it? Faced this frustration and thought of sharing the solution I found out after a long struggle.

When ten users with ramp up time of ten seconds are tried using JMeter tool, Wildfly 8.0.0 soon become unresponsive. However, the JBOSS JMX console application was still available. The log files like error.log, server.log, jboss.log and application logs has no error reported.  The JBoss server had to be restarted every time whe this issue occurred. However the issue was only reproducible when loaded using load testing tool. But not with multiple users accessing the system.

As usual the first suspect was on web server side of JBoss. That is when I have realized how Wildfly architecture entirely differs from older version of JBoss AS. Soon was digging into the “Undertow” component of it. Undertow claims to be a flexible, fast, web server written in Java that is based on the J2SE New Input Output (NIO) API. It is crucial to understand the blocking and non blocking APIs which formed the base of all this. Other than that there is fair amount of understanding on how java NIO works is also required as Undertow uses XNIO, a simplified low-level I/O layer.

Now let us look at how to resolve the hanging issue from the JBoss console. The JBoss console would be available even if the Jboss server hangs and it can be accessible usually by http://<host>:<port>/console.

The configuration required in this scenario would be from Profile tab of the console.

Core Configuration:

Undertow handlers create Undertow XNIO thread which drives the web request. Insufficient amount of IO threads would result in bottleneck at the webserver end and the requests fails to reach the deployed applications. Following that, the "Task-core -threads" and the "Task-max-threads" are used to determine if the request is served or if it is going to be discarded. 

Undertow threads are configured through the IO subsystem. With WildFly 8, a Worker element needs to be created in the IO subsystem, which is a kind of Thread Executor and helps to tune the Web server Thread pool.

Expand the Core option from left menu tree and select IO and select the Default handler from Worker tab. I found in my configuration, the IO thread count was mere 3 and I had to increase it to 50. This is an important tuning parameter which needs to be increased for Web applications which experience a high traffic. Also, increased "Task keepalive" time to 6 60 seconds to wait for the next request from the same client on the same connection.  This would enable browser to eliminate a full round trip for every request after the first, usually reducing the full page load time. "Task-max-threads" is increased to 60 and "Task-core -threads" are increased to 10. This helps in handling concurrent requests.

Undertow IO Buffer configuration need to be verified to optimize the usage of buffer pool.

Web Configuration:

Once core IO is configured, the same worker need to used with HTTP request. From the left tree menu expand the Web subsystem and select the HTTP option. Click on the “Default” element contained in the table and check that the Worker element is associated with your IO Worker. The worker needs to be enabled as well in order to be usable. 

After the above changes, JBoss server needs to be restarted. Note that these changes would also reflect in standalone.xml.

Now, with all these settings, the server would be ready to take the high HTTP load and with this setting I could load the system for longer time without experiencing hanging.