Friday, 23 March 2012

Resolving OutOfMemoryError: unable to create new native thread

(This was tested on Red Hat EL6 JDK 1.6.31 64-bit)

If you've ever come across this error, it can be very misleading.
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:640)

The common suggestion (as the error suggests) is a memory related problem. Some tips but ultimately unhelpful:
http://candrews.integralblue.com/2009/01/preventing-outofmemoryerror-native-thread/
http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
The common theme amongst all of them is the stack size.

Came across this post with same test code:

http://www.odi.ch/weblog/posting.php?posting=411
However the tests were inconsistent, modified it to only go 50 calls deep. Modified code here.
We discovered that changing the options made very little difference to the maximum number of threads.

Originally we thought it was related to changing the linux option for hard/soft maximum number of files that we did the day before:
# vi /etc/security/limits.conf 
testuser soft nofile 4096
testuser hard nofile 10240
# ulimit -Hn
10240

However that didn't seem to be it.

Then we finally came across max process per user:
ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515005
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Which is set to 1024 by default. So adding this to ~/.profile fixed it for us:
ulimit -u 4096

Update 2013/08/15:
To see the number of process for a user run this (replacing [user] with the username):
ps -eLF -u[user] | wc -l