Friday, 23 March 2012

Resolving OutOfMemoryError: unable to create new native thread

(This was tested on Red Hat EL6 JDK 1.6.31 64-bit)

If you've ever come across this error, it can be very misleading.
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
 at java.lang.Thread.start0(Native Method)
 at java.lang.Thread.start(Thread.java:640)

The common suggestion (as the error suggests) is a memory related problem. Some tips but ultimately unhelpful:
http://candrews.integralblue.com/2009/01/preventing-outofmemoryerror-native-thread/
http://www.caucho.com/resin-3.0/performance/jvm-tuning.xtp
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html
The common theme amongst all of them is the stack size.

Came across this post with same test code:

http://www.odi.ch/weblog/posting.php?posting=411
However the tests were inconsistent, modified it to only go 50 calls deep. Modified code here.
We discovered that changing the options made very little difference to the maximum number of threads.

Originally we thought it was related to changing the linux option for hard/soft maximum number of files that we did the day before:
# vi /etc/security/limits.conf 
testuser soft nofile 4096
testuser hard nofile 10240
# ulimit -Hn
10240

However that didn't seem to be it.

Then we finally came across max process per user:
ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515005
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Which is set to 1024 by default. So adding this to ~/.profile fixed it for us:
ulimit -u 4096

Update 2013/08/15:
To see the number of process for a user run this (replacing [user] with the username):
ps -eLF -u[user] | wc -l

Friday, 6 January 2012

Bulk upload of maven artifacts to remote repo

If you have an artifact you want to upload to a remote repository (i.e. internal company repo) then you use the command:
mvn deploy:deploy-file -Durl=file://C:\m2-repo \
                       -DrepositoryId=some.id \
                       -Dfile=your-artifact-1.0.jar \
                       [-DpomFile=your-pom.xml] \


However if you have more than a few files and they're in a nested directory structure (like your local repo) then this is not the way to do it. There doesn't seem to be any tool for either maven or nexus to do this, so here's a quick and dirty way I hacked together:
/**
 * run as java Importer [dirtosearch]
 **/
public class Importer {
   private String directory;
   private final static String repositoryId = "repo1";
   private final static String url = "http://remoteserver/nexus/content/repositories/repo1/";
   private final static String mvn = "C:/java/apache-maven-3.0.3/bin/mvn.bat";

   /**
    * @param directory
    */
   public Importer(String directory) {
      super();
      this.directory = directory;
   }

   public static void main(String[] args) {
      Importer importer = new Importer(args[0]);
      importer.go();
   }

   private void go() {
      try {
         File dir = new File(directory);

         doDir(dir);
      }
      catch (Throwable e) {
         e.printStackTrace();
      }
   }

   private void doDir(File dir) throws IOException, InterruptedException {
      File[] listFiles = dir.listFiles(new PomFilter());
      if (listFiles != null) {
         for (File pom : listFiles) {
            doPom(pom);
         }
      }

      File[] listDirs = dir.listFiles(new DirFilter());
      if (listDirs != null) {
         for (File subdir : listDirs) {
            doDir(subdir);
         }
      }
   }

   private void doPom(File pom) throws IOException, InterruptedException {
      File base = pom.getParentFile();
      String fileName = pom.getName();
      String jarName = fileName.substring(0, fileName.length() - 3) + "jar";
      File jar = new File(base.getAbsolutePath() + "/" + jarName);

      String exec = mvn + " deploy:deploy-file -DrepositoryId=" + repositoryId + " -Durl=" + url;
      if (jar.exists()) {
         exec += " -Dfile=\"" + jar.getAbsolutePath() + "\"";
      }
      else {
         exec += " -Dfile=\"" + pom.getAbsolutePath() + "\"";
      }
      exec += " -DpomFile=\"" + pom.getAbsolutePath() + "\"";
      exec(exec);

   }

   private void exec(String exec) throws InterruptedException, IOException {
      System.out.println(exec);
      Process p = Runtime.getRuntime().exec(exec);
      String line;
      BufferedReader bri = new BufferedReader(new InputStreamReader(p.getInputStream()));
      BufferedReader bre = new BufferedReader(new InputStreamReader(p.getErrorStream()));
      while ((line = bri.readLine()) != null) {
         System.out.println(line);
      }
      bri.close();
      while ((line = bre.readLine()) != null) {
         System.out.println(line);
      }
      bre.close();
      p.waitFor();
      System.out.println("Done.");
   }

   private class PomFilter implements java.io.FileFilter {

      @Override
      public boolean accept(File pathname) {
         return pathname.getName().endsWith(".pom");
      }
   }

   private class DirFilter implements java.io.FileFilter {

      @Override
      public boolean accept(File pathname) {
         return pathname.isDirectory();
      }
   }
}