Tuesday, 24 February 2015

Diagnosing MQ client and resource adapter problems via trace logging

Standalone tracing WebSphere MQ classes for JMS applications

It’s a relatively simple process to enable logging for MQ JMS client libraries.
1. Add a reference WebSphere MQ classes for JMS configuration file.
Here is the command line arguments to reference the common services properties files called “mqjms.properties” for MQ.
Note: the location for the MQ classes for JMS is a uri (i.e. prefix it with file:) whilst the location for MQ classes for Java is a regular path and annoyingly it won’t give you an error if it can’t correctly read the file.
Also included is a reference to a java keystore and turning on java.net logging for SSL and connection handshaking as getting MQ and SSL working seems to be a constant battle, with cipher suites being the most common cause. I even had trouble trying to use the same ssl/cipher from WLP in both standalone JMS and in the non-JMS setup. It seems to require some trial and error for each of the 3 ways of using MQ listed here.
-Dcom.ibm.msg.client.config.location=file:mqjms.properties
-Djavax.net.ssl.keyStore=certStore.jks
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.debug=ssl,handshake
2. Add the settings for Tracing WebSphere MQ classes for JMS applications into the configuration file.
mqjms.properties:
com.ibm.msg.client.commonservices.trace.status=ON
com.ibm.msg.client.commonservices.trace.outputName=logs/mq-jms-trace.log
com.ibm.msg.client.commonservices.trace.level=8
# This setting below seemed to have no effect
# com.ibm.msg.client.commonservices.trace.include=com.ibm.mq.jmqi.remote;com.ibm.mq.jms;com.ibm.msg.client.wmq

Tracing WebSphere MQ resource adapter inside an application server – websphere liberty profile

This is probably the easiest to set up, you can either set it up using properties set on the ResourceAdapter similar to above as documented here: Tracing the WebSphere MQ resource adapter. Or you can use a finer grained control over packages' logging and level using websphere's tracing framework: WLP Logging and Trace
For WLP it's a simple as adding these lines to the server instance's bootstrap.properties:
# enable jav.net level logging
javax.net.debug=ssl,handshake
# default value
com.ibm.ws.logging.trace.file.name=trace.log
com.ibm.ws.logging.trace.specification=*= INFO:com.ibm.ws.mq.*=DEBUG=enabled:com.ibm.mq.*=DEBUG=enabled:WMQ.*=DEBUG=enabled:com.ibm.msg.client.wmq.*=DEBUG

Tracing WebSphere MQ classes for Java applications

If you still using the 'old' MQ for Java (which is useful if you want to barebones access to MQ), there are 2 ways to enable trace logging. As documented in Tracing MQ for Java, you can use the MQEnvironment class which is handy if you want to only long a few specific operations:
MQEnvironment.enableTracing(1);   // start trace
 ...                              // these commands will be traced
MQEnvironment.disableTracing();   // turn tracing off again
Or using the commonservices properties file by using adding the command line:
-Dcom.ibm.mq.commonservices=mqjava.properties
Having a file mqjava.properties in the current directory:
# Base WebSphere MQ diagnostics are disabled 
Diagnostics.MQ=disabled  
# Java diagnostics for the WebSphere MQ Java Classes are both enabled
Diagnostics.Java=wmqjavaclasses
# High detail Java trace
Diagnostics.Java.Trace.Detail=high
# Java trace is written to a file and not to the console.
Diagnostics.Java.Trace.Destination.File=enabled
Diagnostics.Java.Trace.Destination.Console=disabled  
# Directory for Java trace file
Diagnostics.Java.Trace.Destination.Pathname=logs
# Directory for First Failure Data Capture
Diagnostics.Java.FFDC.Destination.Pathname=logs\\ffdcdir
# Directory for error logging
Diagnostics.Java.Errors.Destination.Filename=logs\\wmq-errors.log

A common gotcha – cipher specs

Cipher specs always seem to cause the most issues, this is a list of MQ CipherSpec to JSSE CipherSuite for 7.5.0. With this patch list added support for non-IBM JDK's. There a discussion here about the support.
The only one I managed to get working was this sslCipherSuite using Oracle JDK 1.7.0_51: SSL_RSA_WITH_3DES_EDE_CBC_SHA

Tuesday, 3 February 2015

Keeping a windows process after a jenkins job

One of our jobs on Jenkins is to deploy and startup an application server on a remote slave. However we were having 2 problems with it:
1. the job wasn’t finishing
2. when we terminated it from jenkins, it killed the process it had spawned (the application server).

I spent ages butting my head against Jython on windows, which in hindsight, I could have saved a lot of time if I'd seen how little of the functionality I need from python/windows was implemented in jython/windows.

(1) was solved by getting the start command to write stdout/stderr to a file (irrespective of whether anything was written).

(2) I tried many elaborate solutions, still thinking it was a jython or windows related problem. After seeing it mentioned in blog posts a few times I finally got what they were saying.
Jenkins has a section on their site: Spawning processes from build. Now the key thing here I missed was the BUILD_ID environment variable. So unset it and as long as you've spawned a new process it should fine.
The simple solution should be start_background.bat
set BUILD_ID=
start /B "" cmd /C %*
And just pass your full command to start_background.bat

Unfortunately that didn't work in my odd case, so here's my overkill solution for those that need it:
# Use csscript.exe to trigger the call async and not attached to this process as os.spawnl(command, os.P_DETACH)
    def windowsAsync(self, systemCommand, jobName):
        cmd = open(jobName + '.cmd', 'w')
        cmd.write("set BUILD_ID=\n")
        cmd.write(systemCommand + "\n")
        cmd.write("exit\n")
        cmd.close()
        startCommand = 'start "" /B cmd /C ' + jobName + '.cmd'
        # saving the output to a temporary file to make sure we don't have handles. don't know if this acutally does it
        tOut = tempfile.NamedTemporaryFile()
        tIn = tempfile.NamedTemporaryFile()
        print "Using output temp file: " + tOut.name
        print "Invoking OS command: " + systemCommand
        print "Embeded command: " + startCommand
        process = subprocess.Popen(startCommand, shell=True, stdin=tIn, stdout=tOut, stderr=tOut)
        #process.wait()
        print "Sleeping"
        time.sleep(30)
        
        print "--- consoleOutput start:"
        # Rewind and read the text written
        # to the temporary file
        tOut.seek(0)
        lines = tOut.read()
        print lines
        tOut.close()
        sys.stdout.flush()
        tIn.close()
        
        lines = ""