Le Café Performant: A Blog On Java Performance

A tale about a JVM crash

2011-12-17T08:56:00.003+02:00

It happened again :)

After a long time it happened again....

I was called to investigate a JVM crash. Usually it is the sign of a long adventure along side with the Java platform. A thrilling and frustrating event in the same time. This time I was lucky it took me 4 hours to reach a valid solution ....

Chapter I : First Encounter

First of all I looked at the directory from which the process started running. I started looking at the hs_err_pidXXXXX.log files that existed there about... 50 such files as there is a watchdog process which relaunch the application immediately after the crash. In this case just for the sake of crashing again.

I was happy to discover they accumulate very fast the process needs merely 45 seconds to crash. Someone who is not familiar with those kind of problems may be startled by fact that the process crashes so fast but for me that was heaven! 45 seconds to reproduction.

I started looking at the crash files. They were very disappointing, each one them crashed on different address the thread was applicative. In addition stack trace was missing . I snooped around to make sure no native code was running on this process(no native code good news).

All I had was a pale description of a JVM frustration:

# A fatal error has been detected by the Java Runtime Environment:

# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000000000000, pid=4384, tid=8364

# JRE version: 6.0_26-b03

# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode windows-amd64 compressed oops)

# Problematic frame:

# C 0x0000000000000000

# If you would like to submit a bug report, please visit:

# http://java.sun.com/webapps/bugreport/crash.jsp

--------------- T H R E A D ---------------

Current thread (0x000000001029e000): JavaThread "XXXXXXXXXXXXX" daemon [_thread_in_Java, id=8364, stack(0x000000001a840000,0x000000001a940000)]

siginfo: ExceptionCode=0xc0000005, ExceptionInformation=0x0000000000000008 0x0000000000000000

The fact that crashing thread was an applicative thread was indicating that the crash was not related to GC.I tried switching the GC mechanism anyhow with no change in behavior. I also removed compressedOops and crash remain there standing still.

My next guess was the JIT compiler so as a first step I added the following command line option -Djava.compiler=NONE. That guess was working like a charm JVM stopped crashing. That was still not a sign for euphoria, running interpreted mode is not a very good choice so the next step included divide and conquer.

Chapter II: Lion hunting

I added the following flags to the JVM -XX:+LogCompilation -XX:+UnlockDiagnosticsVMOptions

When a HotSpot JVM is running with log compilation flag it is creating hotspot.log file which includes detailed info regarding all of its JIT activity. For a detailed document regarding this log file take a look here.

Truth be told I still don't fully understand the generated XML but I was looking for the following lines :

Compilation of a method is performed asynchronously by specific JVM threads and every request for JIT is queued before compilation. Grabbing all these lines will give me a full list of all compilation attempts done by the JVM. This list now was taken to a text editor and manipulated so it can be fed to the .hotspot_compiler file.

.hotspot_compiler is a JIT exclusion file which should be placed in the working directory from which the JVM is launching. It contains lines of the following format:

exclude java/util/Properties$LineReader readLine ()I

Telling the JVM|:

"Dear JVM

Do not compile the method realLine which resides in the inner class LineReader of the class java.util.Properties, Thanks in advance,

A desperate programmer"

I started with approx. 2000 such lines and fed them to the .hotspot_compiler file (after removing the compile=NONE flag of course ) and the JVM did not crash. After the divide and conquer process I was left with a single method which its compilation made the JVM freak out.

Everybody is happy crash eliminated with minimal effect on performance.

Chapter III: After-max

Everybody is happy, except of me.... a lot of questions remain unanswered.

The method which aggravated the crash looks like this:

protected boolean isXXXXXXXXX(double stdDev,double mean){

//|stddev/mean| > 10^-5

if(stdDev==0.0){

return mean==0.0;

}

if(Double.isNaN(stdDev) || Double.isNaN(mean)){

return true;

}

return Math.abs(mean/stdDev) < 100000.0;

}

What? That's a method which can cause the JVM to crash ? I don't think so...

My guess was that it is related to some kind of interaction.....

I tried to reproduce the crash on an isolated test and as expected, nothing.

Could not find anything in the internet since I don't even know how to ask the correct question.

What am I going to do next ?:

Try to upgrade the JDK sub version though I could not find a fixed bug related to this in the release notes.
Try JDK 1.7
Try 32 bit JVM
Try to disable certain optimizations such as Escape analysis
Use -XX:+PrintOptoAssembly

But the most important of all hoping for some input from the community......

Java 7 and String.intern()

2011-11-14T18:55:00.000+02:00

For a long time interning strings in HotSpot-VM would store the result in permanent generation. Since Java 7 this is no longer true. Interned strings are stored in the heap. As a result if you have an application where relied on intern() you may need to re-size your perm and your heap. Heap need to be larger Perm need to be smaller.
I understood that permanent generation is going to be removed from the HotSpot-VM in futures releases of Java.
Maybe this is the first step in this direction :)
Reference
http://www.oracle.com/technetwork/java/javase/jdk7-relnotes-418459.html

Java 6u25 released

2011-04-27T10:53:00.003+03:00

Two new operations in ThreadingMBean
- getThreadCPUTime - over threadID array
- getThreadAllocateBytes - allocated bytes on thread basis

http://www.oracle.com/technetwork/java/javase/6u25releasenotes-356444.html

- Tiered compilation is in...

HotSpot diagnostic information has been expanded in several ways:

Tracking of cumulative Java heap bytes allocated on a per-thread basis (Threading mbean was added an operation getThreadAllocateBytes
On OutOfMemoryError, indication of the faulting thread in the heap dump
Improved handling of unexpected exceptions in application native code
Better indication of native heap exhaustion
More details in hs_err files

On JDK 7 G1 enables compressed Oops by default

2011-04-06T20:45:00.001+03:00

Didn't understand why CompressedOops are not enabled by default....
Take a look at this
http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2011-February/002312.html

Thread Top

2011-03-22T13:56:00.000+02:00

Hi,
I have finally found time to write a small tool which can help profiling Java applications.
Its name is threadTop and it connects to your application via JMX.
It can measure

thread contention
thread CPU usage

I think it can be very useful when combined with jstack.

Waiting for your feedback :

http://www.performize-it.com/tools/thread-top

Controlling GC logs at runtime

2011-03-15T19:31:00.000+02:00

GC logging can be controlled using the following command line options:

-XX:+PrintGC (same as -verbose:gc) - Displays basic GC details
-XX:+PrintGCDetails - Displays advanced GC details
-XX:+PrintGCTimeStamps - Displays the time stamp for every GC event reported
-XX:+PrintGCDateStamps - Displays the date for every GC event reported
-Xloggc:c:\gc.log - Writes GC logging to the specified file.

It is usually recommended to run with -XX:+PrintGC -XX:+PrintGCTimeStamps -Xloggc:c:\gc.log even in production. The amount of data written to those files is not huge and it can help to analyze GC throughput ,pause time and even to expose bugs in the GC mechanism.
On the other hand these files are not managed by a "log4j style" rolling appender and will not be trimmed by default. So what would we choose ?
The good news is that logging GC can be configured at runtime. It can be enabled and disabled at will via JMX.
The JMX object com.sun.management:type=HotSpotDiagnostic has an attribute named DiagnosticOptions. This attribute contains 12 boolean sub-attributes (as for Java 1.6.0_24). All of them are very interesting but for the GC logging we need to take a look on 4 of them:

PrintGC
PrintGCDetails
PrintGCTimeStamps
PrintGCDateStamps

The bad news is that -Xloggc cannot be configured via JMX so you will need to configure it on startup.

All these can be modified using jconsole or VisualVM using the operation setVMOption.

My recommendation is that even if you do not want to log GC by default add the following params to your command line -Xloggc:c:\gc.log -XX:-PrintGC -XX:+PrintGCTimeStamps
(Notice that the PrintGC has a - sign before it)
When you will need to analyze GC just turn on PrintGC and possible the PrintGCDetails.

-XX:PrintFLSStatistics=1

2011-03-03T11:50:00.001+02:00

When trying to understand what is going on with GC logging on G1GC. I found an undocumented JVM param that can really help when using CMS. -XX:PrintFLSStatistics=1 will report extra data for every GC:

8.790: [ParNew: 72703K->6341K(76672K), 0.0067245 secs] 72703K->6341K(816832K)After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 94724096
Max Chunk Size: 94724096
Number of Blocks: 1
Av. Block Size: 94724096
Tree Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max Chunk Size: 0
Number of Blocks: 0
Tree Height: 0
, 0.0068238 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]

This can really help when trying to understand CMS concurrent mode failure due to fragmentation

Pity it is not reporting data in a single line though :-)

JDK1.6.0_24 released

2011-02-25T10:17:00.000+02:00

Nothing interesting though.... JDK1.6.0_23 included the latest version of VisualVM but this is old news by now.

First impressions from G1GC

2011-02-25T10:16:00.000+02:00

I was trying to see how G1GC will cope with an application that I am investigating.
Here are my disappointing results:
1. With JDK 1.6.0_20 G1GC is not reporting memory correctly via JMX. problem is solved in later jdks but I am not able at this stage to upgrade the JVM
2. VisualVM is not reporting memory correctly with G1GC need to open a bug for this issue.
3. I cannot find thorough documentation regarding what the GC logs mean (when using G1GC with -Xloggc)
4. The application crashed every few minutes. Again this was with 160_20 and I need to recheck with 160_24

Java6u21 is released 4 days ago

2010-07-12T18:05:00.000+03:00

That's good news from performance perspective since it includes the latest VisualVM version (1.2.2)
According to Oracle release notes this include the following functionality

HeapWalker performance improvements
VisualVM-Sampler performance improvements
BTrace4VisualVM plugin introduces BTrace 1.1
Profiling engine bugfixes
Built on NetBeans Platform 6.8

It still not a full replacement to a commercial profiler but I hope it will be soon :-)

Page file size on machines with large amounts of memory

2010-05-18T10:25:00.000+03:00

There is a rule of thumb that page file size (and swap space partition) should be 1.5 times the physical memory of the machine. Now that we have 64 bit machines with 16-64GB of memory this looks like an absurd.

We do not want to reserve so much disk space
It does not make sense to rely so heavily on disk

I found two nice references in the internet regarding Linux and Windows

http://www.cyberciti.biz/tips/linux-swap-space.html
http://support.microsoft.com/kb/889654

So my new rule of thumb page file should not be more than 70% full

Paging File\\% Usage Peak _Total < 70%

iPad Performance

2010-05-12T20:43:00.002+03:00

Just received my new iPad and I must say that I really amazed by its performance. Everything is so snappy kudos to apple. I do want to understand how they were able to create this experience on such a device.

Can someone arrange me a meeting with Steve?

Dangling Locks

2010-04-25T08:28:00.000+03:00

Happy about Java not suffering from dangling pointers ? Think again the Java5 concurrency utils while very powerful looks like a language flaw:
The following code is an example where a lock cannot be released anymore the thread which holds it is dead. Moreover there is no indication who is holding this lock ...... jstack won't help you......
Oracle please fix it !!!

public class DanglingLock {
   static ReadWriteLock l = new ReentrantReadWriteLock();
   public static void main(String [] args) throws InterruptedException {
   Runnable r = new Runnable() {

   public void run() {
   l.writeLock().lock();
   try {
   Thread.sleep(20000);
   } catch (InterruptedException ex) {
   Logger.getLogger(DanglingLock.class.getName()).log(Level.SEVERE, null, ex);
   }
   }

   };
   Thread t = new Thread(r);
   t.start();
   Thread.sleep(5000);
   l.writeLock().lock();
   }

}

1 gadget per year policy

2010-04-08T10:36:00.000+03:00

Well as I am trying to convince myself that I really need an iPad I use the following arguments
- I must try new technology
- I must understand the hype around tablets

and the most absurd argument is that I need a policy for buying gadgets...
The policy is one gadget per year .....

Sounds reasonable no ?

I really want an iPad but do I really need it ?

2010-04-04T11:48:00.000+03:00

It is cool I fantasize reading books with it viewing movies on the go and browsing the internet in the park....

I am not sure i will continue using it after more than few weeks?

Will I start carrying another device ? not sure.....

I guess that I will continue to hold my breath and wait for Hebrew keyboard, jailbreak and the 3G version....

JDK1.6.0_19 is released

2010-04-04T11:44:00.000+03:00

No major news just a few bug fixes...
BTW JDK1.6.0_18 included a new version of the VisualVM which is much more powerful and also includes a sampler..... so I was very anxious to see if 1.6.0_19 includes visualvm1.2.2 but it does not ....
Never mind we can always download it directly from the website....

AWS is offering consolidated billing system

2010-02-11T17:04:00.002+02:00

It took them one year but finally......
If you are a company and wanted to use AWS you every user needed to create an acoount and enter billing details.
This is very cumbersome and does not enable centralized management and tracking of cloud costs.
Since not it is going to be changed. You will be able to consolidate all your acoount and get aggregated report for your organization.
I guess that now it will be also possible to limit an acoount budget according to the organization policy.....
Kudos Amazon

Cloud era: Pay for your poor performance

2009-12-30T13:19:00.003+02:00

I have read an interesting post which claims that in the cloud computing era poor performance directly converts to higher operational costs. http://www.infoq.com/news/2009/11/Price-for-Performance
I agree with this to some extent. Paying for machines is not exactly as paying for CPU but the general idea is interesting.
Thinking about it a bit more it makes me wonder that it will be nice to be able to find the equilibrium point of performance investigation investment. The amount of money that you invest of performization should be less than the amount of time you loose for inefficiency.
A new research area is born :-)

diagnosis tools for JDK 1.6

2009-12-08T17:40:00.000+02:00

A nice Sun document on diagnosis tools for JDK 1.6
http://java.sun.com/developer/technicalArticles/J2SE/monitoring/

KSE2 is around the corner

2009-11-01T17:48:00.004+02:00

Hi,
On the 17th of November the second Knowledge share event is going live.
It will include 3 sessions:
1. Fork/Join Framework
2. What makes good performance
3. Short stories on simple tools

For more info and registration
http://www.performize-it.com/kse/kse2

Mac OSX Snow leopard better performance

2009-09-12T21:06:00.002+03:00

I have upgraded my Macbook to Snow Leopard this weekend and indeed experienced better performance. I have bumped into an interesting article which explains why : (follow link )
To make short story shorter Mac maps 32bit processes to two 4GB address spaces one for user mode and one for kernel mode. As a result any kernel mode system call will require a TLB flush.
Windows by limiting user address space to 2GB and kernel address space to 2GB do not suffer from this problem (but suffer from much lower memory limit). Snow leopard is full 64 bit which solves this issue. User mode and kernel mode address space share the same TLB

A few comments on my previous post

2009-09-03T10:50:00.002+03:00

When you are able to RDP to console session. Most of the times you will be able to connect to the process if you will give it the pid from the windows task manager.
jps which lists only java processes will not work. However jconsole -p will. So there are cases where this technique is not required.

However there are cases when you are not able to connect to the console session. In that case enabling interaction with console option will not help but you can still communicate with files.
Just write a batch file and execute it with cmd.exe -C
you can redirect your input and output to files so you will be still able to communicate with the shell

A neat way to monitor windows services

2009-07-06T14:53:00.008+03:00

The JDK supplies a very impressive toolchest which enables you to monitor the JVM(
jconsole, jps, jmap,jvisualvm etc...). There are also other tools such as BTrace which can attach to the JVM and solve performance problems.
On UNIX you only need to use su or sudo and act as the user running the JVM. Doing that you can attach to it and apply all commands.
In Microsoft Windows the situation is more complicated. The service processes run under a different user and in another session and you are not able to connect to them.

But do not panic:-) I am suggesting here a simple workaround.

Step 1:
Download and install the Resource Kit Tools from Microsoft. (The 2003 version will also run on Windows XP). Didn't try Vista/2008 but then again who uses Vista.

Step 2:
Run the following command:
%RESKIT_HOME%\Tools\Instsrv.exe" SrvCMD "%RESKIT_HOME%\Tools\Srvany.exe"
It will create a service named SrvCMD (in case you have such a service use a different name)

Step 3:
Open services.msc and update the service properties to run manually. Check "allow service interact with desktop" box.

Step4:
Execute regedit.
Open the key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SrvCMD
Add a sub-key named Parameters
To this sub-key add a string value named Application. Its value will be the name of the executable you want to run. In our case c:\WINDOWS\system32\cmd.exe.

Step5:
Note: If you are connecting with remote desktop make sure you are connecting with /console parameter (otherwise you will not see the command window)
Start the service.
A window with command prompt running in the session of the service will be opened.
jps will now discover your JVM.

Enjoy!

BTrace session slides

2009-07-03T11:26:00.003+03:00

BTrace session is over.
I worked very hard preparing it though I think I needed more time for working on the running examples and integrating them seamlessly with the presentation.
It was a fascinating experience for me.It is the first time for me that I am arranging a public session and I am sure that more will come.
37 Attendees(out of max 50 available seats) two and a half hours of covering BTrace.
The responses were very positive. People were very enthusiastic about the tool.
I have published the slides of two presentations from the event in my web site:

The participants of the event are entitled to participate in a lottery upon submitting a final exercise!

10 Days left

2009-06-20T00:29:00.002+03:00

It is 10 days away. My BTrace session. I am so excited about it.
This the first event I am organizing and I hope that ppl will be tolerant toward any organization problems that may occur....
I have just discovered to that my BTrace post was cited in the BTrace project homepage :-)
And today while working on my presentation I discovered that BTrace do not on my Mac (till now used it only on Windows machines)
Browsing the web a bit I found a post by Ken Sipe who already solved the problem.
Downloaded his patched version and bang it worked !!!!!
Lucky me :-)

ScalableJ First Knowledge Share Event

2009-06-03T11:45:00.005+03:00

I have been thinking about building up an event by myself for a long time and this idea has finally came to life. I am going to give a lecture on BTrace on the 1st of July in Petah Tikva,Israel
Registration is free of charge.
More information can be found at http://www.performize-it.com/training/kse/kse1
You are welcome to participate......

JDK 1.6.0_14

2009-05-31T13:19:00.002+03:00

Well it appears that this revision is not a one to ignore if you care about performance (and you have the guts of an early adapter:-) The "small" revision update is not small at all and contains 3 important performance improvements:
1. The long anticipated G1 garbage collector
2. Escape analysis
3. Compressed OOPs

I will elaborate on these features in my next posts.... for now you may look at http://java.sun.com/javase/6/webnotes/6u14.html

What does this say about JDK1.7 ? Down porting features before 1.7 is released...... makes you wonder is 1.7 going to be Sun's Vista?

BTrace the ultimate JVM probe

2009-05-27T09:18:00.003+03:00

Hi,
I have been working with BTrace in the last few months and the more I use it the more I appreciate it. BTrace enables you to write a small java class that will instrument you java application on the fly and will give you ultimate flexibiliy to pin-point the requested information.
It can help in debugging/ profiling/ deadlock analysis and solving other stability problems
I am going to give a session 2.5 hours session on the 1/7 in Israel(Petah Tikva). Details will follow..... If you want to attend send me an email to haim.yadid@scalablej.com
Here is a tricky example which will help you identify where in the code a you write a certaion message to stdout..... enjoy!

package com.scalablej.btrace;

import com.sun.btrace.annotations.*;
import static com.sun.btrace.BTraceUtils.*;

@BTrace public class PrintlnLocator {
@OnMethod(
clazz = "java.io.PrintStream",
method = "println",
location = @Location(value = Kind.ENTRY))
public static void onPrint(Object a, String l) {
boolean b = compare(
get(field(classForName("java.lang.System"), "out")), a);
if (b) {
println(l);
jstack();
}
}

}

Utilizing the Cloud for Performance Validation

2009-04-21T13:50:00.004+03:00

Yesterday I gave a lecture to the Israeli Association of Grid Technologies (IGT) as part of the cloud computing work group. I was discussing the benefits of executing performance lab for the purpose of load testing inside the cloud (opposed to purchasing required hardware).
You can find the presentation in my website http://www.scalablej.com/cloud-computing

IGT cloud computing Workgroup

2009-04-13T18:55:00.002+03:00

Hi
Next Monday (20-Apr-09 14:00-17:00 IDT) I am going to present a case study which will discuss the benefits of performing application load test in the cloud. My presentation will take place in IGT offices in Herzelia and will be part of a meeting of the Israeli cloud computing work group along with two other sessions of Nati Shalom CTO of GigaSpaces and Moshe Kaplan CEO of Rocketier. Registration is free just send an email to info@grid.org.il.
For more information on my session and other session in the WG meeting follow this link
http://www.grid.org.il/?CategoryID=384&ArticleID=137&Page=1

Hope to see you there....

Nice enhancement of JDK1.6.0_14

2009-03-17T17:04:00.002+02:00

From JDK1.6.0_14 a stack trace dump is added to a hprof memory snapshot this can help profilers display thread and stack is shown for GC roots of types "Stack Local" and "JNI Local".
Yourkit 8.0.3 already utilizes this feature.
Nice!

Yet another reason to move to JDK 1.6

2008-12-16T23:33:00.002+02:00

My first production memory leak analysis with Java 1.6 was almost a very pleasant experience.

While in previous versions of java I needed to restart the JVM and set appropriate profiler command line option, Java 6 comes with a built in alternative. jmap which was not available for Windows distributions is now available for use. Taking a memory snapshot is a point and click task:

jmap -dump:format=b,file=dump.hprof

Take the hprof dump generated and use you favorite profiler to analyze it.

If you are only interested in a short summary of memory usage by class:

jmap -histo

So, why was it only an almost pleasant experience ? Because you need to make sure that you are connected to the same windows session as the process you are trying to dump. If you are using terminal services make sure you use the /console command line option, otherwise jmap will not work. Beats me why..

10 People - a Billion USD revenue

2008-12-03T05:18:00.009+02:00

I have been attending this week to the"World summit of cloud computing" which was quite an impressive conference compared to other Israeli conferences I have been to with representatives from Google,HP,Intel,Amazon,EBay, Yahoo,SUN, Salesforce, GridGain, Forrester,GigaSpaces, MerrilLynch and more. Each one of those vendor has a different definition and point of view so after those two days I am ready to give my own definition and what it means to me.
So, Cloud computing is a deployment option for your application which has the following four characteristics:

Outsourced
Your run your stuff on other people machines and infrastructures(IaaS). These resources can be conזsumed either from the the intranet for small large enterprises and government use or reside in the in the Internet for small/midsized companies. But the main issue here is that all you pay for the infrastructure in that case is money.

Economical
Using this deployment option should save you money and possibly a lot of money.

Reduced CAPEX:, meaning money you will need to invest upfront inoder to support the expected and unexpected changes in capacity.

Reduced OPEX: Let cloud providers do what they do the best, maintain large machine infrastructure. Hopefully (with the right amount of competition) that will reduce your IT OPEX.
New financial model: All the ?aaS concept is presuming a different(and more fair) revenue model compared to what we are used to in the traditional software industry you only pay for the resources you use no payment upfront and not commitment for long term.

Elastic
You are able to adjust seamlessly the amount of infrastructure you consume. This helps you handle peaks more efficiently. You no longer need to choose between compromising on your SLA in peak times or let your servers rot in the cellars when you really don't need them.

SelfService
You as a developer(and I am intentionally focusing on developers) are able to allocate and deallocate resources by yourself through an API no need to sign forms no paperwork. You can write on use software which will do the job for you.

And not to my thoughts:
So I have a crazy vision regarding this new concepts what I see in my mind is a billion USD internet company with a size of 10 people which altogether bring unique (probably patented) value and all their job is to orchestrate outsourced resources to the benefit of the company.
A CEO - orchestrating vision and execution with a set of VAs
VP sales - orchestrate sales operation through an effective network affiliation
VP infrastructure - manages the application in the cloud
VP support - orchestrate outsourced call center and
VP finance
VP R&D -
4 developers - developing only core business logic consuming open source projects for the other stuff.

I know it is a bit extreem but I don't care I am sure it will happen one day.

IGT 2008 next week

2008-11-24T08:35:00.002+02:00

The world summit of cloud computing is taking place in Israel next week.
I have big expectations toward this summit as I believe that the future is in the cloud.
This is not an affiliation link :-) you are welcome to register I'd love to see you there!
http://www.cloudcomputing.org.il/index.html

Last post Recall or why use HTTP1.1

2008-11-23T20:45:00.006+02:00

Today I saw that Ford are recalling 1000 cars due to problems in steering wheel liquid. This makes me feel a bit more comfortable about the need to take back what I said just few days ago :-).

So why do I need to recall the post. Not because SSL connection establishment is not expensive. It is indeed. The issue is that HTTP1.1 (opposed to HTTP1.0) can pool the connections and reuse them for later requests thus diminishing the effect of HTTPS latency).

So why the Apache web server was not behaving well (and put me into confusion)? Configuration .....

The AWS was configured to use HTTP1.0 for HTTPS connection under Internet explorer.

I did not understand yet why(an issue for a later post) is it so, probably bugs in older versions of IE. Anyway commenting out the following lines in httpd-ssl.conf brings back the missing .1:

BrowserMatch ".*MSIE.*" \

nokeepalive ssl-clean-shutdown \

downgrade-1.0 force-response-1.0

And voila after that only one roundtrip for SSL requests.....

Absolutely great :-)

Speed of light vs SSL

2008-11-18T20:07:00.003+02:00

(Please see next post for update)

I have been trying to measure WAN efffect over an application I am working on. Due to a bug the application was running in SSL and I was quite surprised by the results.
After investigating it a little I want to share with you the following information. https (SSL) requests to the server involve two extra round trips for certificate authentication and key exchange. As a result a short HTTP request will require 3 times the network round trip time (6 times the latency). For example if latency is 90 ms and server time is 20ms https request will be 90*6+20= 560ms. While an http request will require 200ms. Quite impressive difference.
Saying that it is extremely important to reduce number of https requests to a server combining https requests is 3 times more important than regular http requests.

TSSJS Day 3

2008-06-24T15:51:00.004+03:00

I want to cover two sessions from the third day:

The busy Java developer’s Guide to Scala

Ted Newman is definitely a charismatic presenter. With great passion he reviewed basic functionality of Scala DSL. Well Scala is a functional language and after the session I really want to learn it.

Performance Tuning a web shop with open source tools

Dr. Jeroen Borgers showed how performance improvement process can be done using Open Source tools. He was using JMeter (while I usually use LoadRunner) for load generation. Jprobe for profiling (I prefer YourKit BTW). JAMon for continuous monitoring (I used SiteScope commercial tool). For reporting purposes they wrote their own tool and placed on SourceForge (JaRep).

TSSJS Day 2

2008-06-24T15:49:00.001+03:00

I want to cover 3 sessions on the secion day

Languages-Oriented Programming: Shifting Paradigms
Computer languages evolution enables us to pave over disturbing problems (e.g. GC as a mechanism to pave over error-prone memory allocation, Neal Ford introduced a very important observation.. One of the most powerful aspects in Java platform is the community and the amount of open source frameworks. This creates a new problem to pave over. Each framework has its own jargon and adapting this to the java language leads to a complicated syntax which usually is very wet (not DRY – Don’t Repeat Yourself). Niel offer the use of DSLs to approach this problem. In other words frameworks will be transformed to a carefully designed DSL. AntLR as a lexical analyzer and environments like Jetbrains MPS can help reaching this with less efforts.

Extreme Transaction Processing, Low latency and performance
John Davis a banking expert gave a startling session on design criteria in the online trading arena. In this world a 1ms delay in processing a message can lead to losses of $100M. As a result for example banks try to locate the trading infrastructure as close as possible the trading backbone (usually in London) in order to reduce latency. In addition GC can be a real problem we are used to think of GC taking 200ms as something reasonable but again it is x200 from the 1ms threshold. You do not want to loose money on GC. This leads to weird solutions such as restarting the VM before the first GC and redirecting to a different cluster member in the meanwhile. Another point mentioned is that traditional RDBMS are not capable of handling tens of thousands of transaction on a reasonable price. The solutions will be to use in memory databases or caching mechanisms (e.g. GigaSpaces, Oracle Coherence, Terracotta etc… )

Concurrency and High Performance
Kirk Pepperdine session had an important punch-line. Processors clock speed is stuck at the 3Ghz boundary and this situation is not likely to change in the near future. CPU vendors are going to achieve Moore’s law by doubling the number of cores every 18 months. This is a fact developers can’t afford to ignore. Leading to the inevitable conclusion (punch-line ahead) Is your application ready to double it’s concurrency in the next 18 months?

The inevitable delay

2008-06-24T15:30:00.003+03:00

A professional blogger will never let this happen. I admit that as a newbie when my wife joined me I preferred traveling and not blogging… Here is a link to the pictures. When I came back to Israel I was dead tired. So I continue to write only now (I do not regret it though)

Yourkit 7.5 is released

2008-06-22T08:14:00.002+03:00

Yourkit is my favorite profiler. It is a bit of old news but I am happy to hear that Yourkit 7.5 is released (on the 22-May-08). It includes of usability improvements and bug fixes. The most important thing I think is the CPU measurements for stack trace snapshots. It can actually show you which treads consume CPU before starting to profile. Previously I used intrnal tools for such purposes... I did not try this feature yet and I wonder if it can be used with out configuration of the profiler agent (great value for production)

TSSJS day 1

2008-06-19T10:47:00.006+03:00

A: Beautiful

B: Who?

A: The City.

B: Which city?

A: Prague

B: So why don’t you say Prague? Why are you going around and around?

A: Well, since I have got here I have an unstoppable urge to stroll in the beautiful streets.

B: So why don’t you?

A: Well....

I have landed in Prague Yesterday in 08:45 directly into the conference. (Comment: Try to avoid these scenarios as much as you can, I have been dead tired all day long). To the conference I have arrived two hours late and missed the first two sessions :-(.

While I really want to share as much knowledge as I can I am reluctant of writing long descriptions and summaries on sessions. A lecture heard can enlighten one’s mind but it is hard to transfer the essence of this enlightenment to a blog . Therefore, I am going to share with you a single most important piece of information I took with me from sessions.

The two most effective sessions I did attend on day 1 are (Drums !!!!):

Monitoring Management and Troubleshooting in the Java SE6 Platform

Jean-Francois Denis from SUN gave a very interesting session on JDK 6 new JMX abilities and tools. The lecture which started with the very basics did move to more advanced issues. The most useful piece of information from my point of view is the VisualVM. A real open source lightweight Java profiler!!

Java Performance Tooling

Dr Holly Cummins from IBM is a very colorful person and a funny lecturer. She gave a nice introductory session on performance troubleshooting. Holly exposed me to the term ‘lock-bound’ which is a brilliant terminology for saying ‘Well we use locking mechanism extensively and there is a lot of contention on these locks… this is why our application sucks’. From this lecture I am have learned that IBM have set of nice free tools which one can use even if he is not using IBM JVM.Follow this link for more information.

TSSJS

2008-06-18T05:18:00.003+03:00

Hi,
I am on my way to TSSJS. This is the first time I am attending this conference and I am really thrilled as I heard great deal about it.
I will update my blog during the conference whenever I will have something interesting to say :-)
If you attend the conference come and say hello :-)

I have a flight to catch....

Client side Memory Leaks IE and GWT - A practical guide

2008-06-09T03:46:00.007+03:00

By now I guess that you read the previous post and you are eager to get some practical knowledge.
So I will summarize my knowledge regarding this issue. While I agree it is a bit shallow it is much better than nothing at all.
Tools like IESieve and IEdrip proven to be inefficient when coming to GWT leaks the code is too big for them and the transformation from Java to JS complicates things.
I will be more that delighted to be proven an idiot, if someone have a better (proven) approach please let me know.

Code Review
When coming to solve /prevent memory leaks the best way will always be code review. Hence I will give a list of guidelines for reviewing GWT code:
First list of guidelines Stating the obvious:

Avoid writing JSNI code. Google made quite a good job in writing "almost" leak free it is so easy to ruin it if you do not know what you are doing. Remember every JSNI code you write will lower your productivity.
Do not use the DOM.* methods (except the setStyle... which are safe) . Manipulating the DOM yourself will lead you directly toward a memory leak.

Second list of guidelines - less obvious

Static variables containing(even indirectly) references to widgets and dom objects may cause a leak.
According to Google it should not happen :-) but in some cases event listeners may leak. unregistered them when window unloads.

Lion in the desert approach
When code review by it self is insufficient, the desperate developer should try the following approach. As stated in the previous post when a memory leak in GWT occur the entire JavaScript is locked in the memory. This will lead to a huge memory leak (megabytes in large modules). This major drawback which forces us to deal with memory leaks is also our savior. Since the memory leak is so evident it is possible to defferentiate between cases where it is existant and cases where it is eliminated. Track the VM size of the IE process while refreshing the page for several times. If the VM size inceases dramatically you have got a memory leak. Vice versa, memory do not increase constantly memory leak is solved.
Saying that all you need to do is comment out pieces of code until memory leak is eliminated.

This process is not easy nor fun some tips to improve its effectiveness:

Memory leaks are ilusive sometimes removing an irellevant piece of code will stop the memroy leak. As a result you should try to reach to the smallest peice of code which is still leaking before eliminating it. This will ensure you do not shoot at the wrong target .
Work directly on javascript files. It takes some experitse to be able to understand what you are doing but it worth it. GWT comiplation phase is long and working directly on the JS file shortens the search cycle dramatically.

Thats all folks :-)
Enjoy

Client side Memory Leaks IE and GWT - Introduction

2008-05-30T01:35:00.011+03:00

I believe that software development should be fun. This is why for a long time I avoided the wild world of client side and JavaScript. Even though there is a big traction toward AJAX and WEB2.0 I am still not convinced that a group of average developers can develop a large scale, maintainable, enterprise ready and long lasting project based on this technology. JavaScript in just not that kind of language and the freakishly incompatible Browsers world which do not supply comprehensive set of monitoring tools is absolutely a nightmare. Moreover this world is ruled by IE, the two headed beast, which encompasses two GC mechanisms each one of them is not aware to the other. It is so easy to introduce new memory leaks and it is so hard to eliminate them.

I am currently involved in a project written in GWT. I must admit that on the first look this ambitious technology looks quite promising. Developing in Java is much better than JavaScript, GWT offers mitigation to browser incompatibility and IE memory leaks.

Indeed, development phase was quite productive developers did enjoy to develop the code and delivered fast results. (Complications came back when JavaScript native code needed to be written in order to supply needed functionality not supported by GWT--> we were back in the JS alley)

When we approached stabilization phase things became nasty. On one hand performance optimizations where required and we could not ignore the fact that the target engine was JavaScript engine. Moreover the biggest nightmare was memory leaks in IE 6/7. Each indirect circular reference between the DOM and the JS engine caused the entire JS to be locked in memory leading to megabytes of leak. The effort needed to solve memory leaks was huge and involved most of the code written. I will not delve into project constraints and design criteria that might have made this problem so acute. I want to focus on some guidelines to help in avoiding and solving memory leaks. Since this post is getting a bit long I will split those guidelines to the next one.

And one good news to close this post: I opened a bug to Google regarding a memory leak when closing a window 5 months ago. I forgot all about it, buts suddenly out of the blue I got an email. Bug was verified and fixed. Fix will be available for GWT1.5

Moore's Law for the average Mosquito

2008-05-27T23:55:00.000+03:00

I really planned to go to sleep, but I was really amazed by the amount of mosquitoes that one have to hunt before he can go to sleep quietly. It suddenly occurred to me that the average mosquito is becoming twice as fast every approx two years. It is maneuvering much better. It's flight is much more silent and its bite is becoming stronger.

When are we going to see dual core mosquitoes?

System.out.println("Hello World!");

2008-05-27T23:20:00.000+03:00

Hello world

I wanted to open a professional blog for a long time. It took me a few months to convince myself that I can transform this desire into commitment. I have a lot of thoughts related to java and performance that I want to share with you all. I really hope it will be a trigger an effective and lively discussion.

There goes away one hour of sleep a day... I hope that I will not fall asleep in the middle of the posts :-)

So, now I will check again that post title is not empty and....
System.out.flush();