Tuesday, 14 March 2017

Populating stored procs into a HSQL DB

I recently encountered a problem trying to load stored procedures into a HSQL DB used for testing. The problem was caused by the script runner provided by spring which separates each statement to be executed in a script file by a semicolon. If a stored proc has statements inside it (which most do), then the proc isn't executed as a single statement. This is further compounded by each statement executed must be understandable by JDBC. For example the following stored proc causes issues:
CREATE PROCEDURE MY_PROC(IN  param1 VARCHAR(30), OUT out_param VARCHAR(100))
  READS SQL DATA
  BEGIN ATOMIC
    SELECT the_value INTO out_param FROM my_table WHERE field = param1;
  END
.;
This problem is solved by using the script runners provided by HSQL in the "org.hsqldb:sqltool" dependency as they parse can correctly parse the scripts containing stored procedures. Here is a Spring Boot test, using an in memory database but using HSQL's script runners:
@RunWith(SpringRunner.class)
@AutoConfigureJdbc
@Slf4j
public class MyDaoTest {

  @Autowired
  private MyDao dao;

  @Test
  public void myTest() throws Exception {
    String response = dao.invokeProc("1234");
    assertThat(response, notNullValue());
  }

  @Configuration
  @ComponentScan("net.devgrok.sample.dao")
  public static class Config {

    @Bean
    public EmbeddedDatabaseFactoryBean dataSource() {
      EmbeddedDatabaseFactoryBean factory = new EmbeddedDatabaseFactoryBean();
      factory.setDatabaseType(EmbeddedDatabaseType.HSQL);
      factory.setDatabasePopulator(databasePopulator("./src/db/sql/stored_proc.sql"));
      return factory;
    }

    @Bean
    public HsqlDbPopulator databasePopulator(String... scripts) {
      return new HsqlDbPopulator(scripts);
    }
  }

  public static class HsqlDbPopulator implements DatabasePopulator {
    private final String[] scriptFiles;

    public HsqlDbPopulator(String[] scripts) {
      this.scriptFiles = scripts;
    }

    @Override
    public void populate(Connection connection) throws SQLException, ScriptException {
      FileSystemResourceLoader resourceLoader = new FileSystemResourceLoader();
      for (String scriptFile : scriptFiles) {
        try {
          SqlFile file = new SqlFile(resourceLoader.getResource(scriptFile).getFile(), null, false);

          log.info("Running script {}", scriptFile);
          file.setConnection(connection);
          file.execute();
        } catch (IOException | SqlToolError e) {
          log.error("Error executing script {}", scriptFile, e);
          throw new UncategorizedScriptException("Error executing script " + scriptFile, e);
        }
      }
    }
  }
}
Note: this test uses an in memory database, if you want to use another variation, you'll need to create a custom EmbeddedDatabaseConfigurer instance to do this which is slightly painful due to Spring's insistence on making everything either private or final or both.

Friday, 10 February 2017

Dev Setup of a Mac

After working in my 2nd consecutive company that uses Mac's for developers and having forgot everything I used the first time, I thought I better write down all the tweaks, work arounds and config changes that got me using my mac efficiently.

Remap Fn-C to copy

Mac's use Mac-C instead of Ctrl-C to copy (and X, V to cut and paste). This is really annoying if you switch between a mac and Windows machine a lot. Fortunately you can use Karabiner to map Fn-C to copy as the Fn key is located where Ctrl is on a windows keyboard. If your on OSX Sierra you'll need to use Karabiner Elements for now.

homebrew

This needs to be installed first as it installs most dev tools.
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

A better terminal

brew install iterm2

Colour ls 

From this Stackexchange post, edit ~/.bash_profile:
export CLICOLOR=1
export LSCOLORS= gxBxhxDxfxhxhxhxhxcxcx

AWS etc

awslogs is a cloudwatch log watcher (you may have to run the install as sudo)

brew install python
brew install awscli
pip install awslogs

Tunneling

sshuttle - a handy python ssh tunnelling that allows you to easily route ranges of IPs over ssh.

pip install sshuttle

Ruby

brew install rebenv
Add the following to ~/bash_profile
eval "$(rbenv init -)"
Then run
rbenv install 2.2.3
rbenv global 2.2.3

Java

brew cask install java
brew install gradle
Then add this to ~/.bash_profile:
export JAVA_HOME=$(/System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java_home)

Sunday, 29 November 2015

Running a meteor shell on a standalone server

For those that want to connect to meteor's shell running on a standalone/self-maintained server, the standard command you use in the development environment doesn't work. Fortunately you can 'trick' it into allowing the `meteor shell` command.
APPDIR=/opt/bitnami/apps/myapp
export METEOR_SHELL_DIR="$APPDIR/.meteor/local/shell"
# other settings
# ...

exec node $APPDIR/bundle/main.js
cd /opt/bitnami/apps/myapp
mkdir -p .meteor/local/shell
echo > .meteor/packages
echo 'METEOR@1.2.1\n' > .meteor/release

meteor shell

Monday, 5 October 2015

Javascript (ECMAScript 5.1) Refresher

I’ve been recently working heavily on JavaScript based applications in both Node.js and a Chrome extension. After working mostly on Java, I thought I’d share the syntax, conventions and the new features of ECMAScript 5.1 and parts of the still pending version, ES6/ECMAScript 2015, that I’ve come across.

Comparisons Operators

==
equal to
===
equal value and equal type
!=
not equal
!==
not equal value or not equal type

I saw something else I thought was a fancy notation:

if (!!something)
//which is not a special operator, it just converts it to boolean, i.e a double not.
(!!something) === (!(!something))
//this would only be useful if you want to set a boolean to something truthy, i.e.
var a = "somevalue";
var asBoolean = !!"somevalue";

Object Initializer

Something I had thought was not a fully adopted syntax, was being able to specify the getter / setter functions in the initializer. (MDN Object initializer)
var o = {
func1: function (p1, p2) {},
get prop1() { return this._prop1; },
set prop1(value) { this._prop1 = value; },
};
This creates:

  1. a function called func1: o.func1(param1, param2)
  2. setter function  for "prop1": o.prop1 = 'newvalue';
  3. getter functions for "prop2": var oldValue = o.prop1;
Which is equivalent to more the verbose way:
var o = Object.create({}, {
func1: {
value: function(p1, p2) { },
},
prop1: {
get function() { return this._prop1; },
set function(value) { this._prop1 = value; },
},
});

Other odd syntax


The OR operator:
// this allows fall-back assignment in case a variable is not defined or set
var myNamespace = window.myNamespace || {};
// which is equivalent to: (there are many more ways to write this but this is the simplest direct translation)
var myNamespace;
// if you just use myNamespace not window.myNamespace you'll get a variable not defined error
if (window.myNamespace) // if defined and set to a value
myNamespace = window.myNamespace;
else
myNamespace = {};

This is sometimes written like this:

var myNamespace = window.myNamespace || window.myNamespace={};

Which although it seems wrong, is actually valid syntax. An assignment of a variable statement returns a value, so (window.myNamespace={}) is equal to {}. It’s nice and compact but does border on readability vs magic syntax (those trying debug Groovy scripts will understand what I mean).


Next the AND operator:
DEBUG && console.log("My debug statement");

// is a compact way of writing
if (DEBUG)
console.log("My debug statement");
Don't abuse this one, as its easy to make JavaScript look like a completely different language. My recommendation - used for a few scenarios and use it consistently. Pretty code, is readable code.
The COMMA operator:

This one I really did think was some magic syntax until I looked it up. By the example given on MDN comma operator reference my best bet is it was intended only for the for loop, which in hindsight I vaguely knew it was legal syntax there.
This just executes each statement in turn, returning the value of the last statement.

if (doSomething)
return retValue = invokeMethod(), console.log(retValue), retValue;
else
return false;

Scope & Closures

Reading through a lot of the articles there is a lot of talk about closures and warnings about scope when using nested function. Reading some of the examples out there could lead you to code over defensibly (which I realised I had been doing) or just left wondering what the big deal is. Function scoped variables in JavaScript behave almost the way you would think at first glance.

  • Nested functions are able to access the variables of parent functions scope (all the way up to the global scope --> window)
  • Parent functions can't access the variables of nested functions
  • Nested functions can still access the parent functions variables after the parent function(s) have finished executing --> they get a copy of the stack
  • Each invocation of the parent function creates a new stack so someFunc(1) defined variables won't overlap with someFunc(2)
  • If myFunc1 and myFunc2 were defined in parentFunc:
    var obj = parentFunc(123);
    obj.myFunc1();
    obj.myFunc2();
    Then myFunc1 and myFunc2 could both alter variables defined in parentFunc

This last point is the only one that you have to be wary of. This blog post has more detail, however here is the snippet which highlights the problem:

function f() {
var arr = [ "red", "green", "blue" ];
var result = [];
for(var i=0; i<arr.length-1; i++) {
var func = function() {
return arr[i];
};
result.push(func);
}
return result;
}

This has the effect of every function in the result returning "blue" as each function sees the latest value of i, which is the length of the array.


Function expressions, declarations etc

For the most part it doesn't matter whether you use a function declaration or a function expression, there's listing of all the function types and their minor differences here: (MDN Functions reference). The only difference that should affect most people is:


  • Function declarations can be used before they are declared as they are 'hoisted' automatically by the parser.
  • Function expressions on the other hand, need to be defined before they are invoked or assigned.

Classes


There are many different patterns people use to create classes that together with the ECMAScript 6 shorthand. I won’t show them all but here is the one I use for ECMAScript 5.

var Person = function Person(name) {
this.name = name;
this.canTalk = true;
};
_.extend(Person.prototype, {
prop1: null,
prop2: 123,
prop3: {},
prop4: "",
func1: function () {
console.log("%s's prop2 is %s",this.name, this.prop2);
}
});

var tim = new Person("Tim");
tim.func1();
// response is:
// > Tim's prop2 is 123

I’ve seem some people use Person.prototype = {..} which has the effect of overwriting the Person.prototype.constructor property which doesn’t have immediate problems but can cause issues with inheritance and some frameworks. This Stackoverflow discussion shows why.

I use underscore.js in this example, but jQuery does the same, as a shorthand for having to write Person.prototype.prop1 = null; etc to set all the methods and properties on the prototype so they are shared between object instances.

ECMAScript 6 allows you to easily define the functions in a class but doesn’t allow you to set the starting value for instance variables on the prototype. i.e. you’ll still need to do _.extend(Person.prototype, { prop1: … prop2: .. }); but you’ll just define the functions with the class.


Namespaces


In JavaScript, namespaces (aka packages/modules) are written simply as objects:
// global namespace
var MYAPP = MYAPP || {};
// sub namespace
MYAPP.event = {};

They’re nothing magical and not built into the language, more of a technique of scoping your variables and functions so they don’t clash with the numerous JavaScript libraries now in use. MDN has an intro on namespaces in their Object Orientated section. There’s a Stackoverflow discussion: What's the best way to create a JavaScript namespace?

The syntax below I found the best supported by IDE’s as any good programmer knows, sometimes the best way to do something is the quickest/more efficient way by using the supported libraries/tools to make our lives easier. This uses a ‘immediately-invoked function expression’ (IIFE) that passes in the global object (in this case the window) to define the namespace so it can be referenced by other code. This also makes it easier to wire into something that doesn’t run in the browser, i.e Node.js, however the syntax below is better suited to that..
((function(global) {
/**
*
*/
MyNamespace.method1 = function() {
//... do something
};

//export
global.MyNamespace = MyNamespace;
})(window);

I came across this article Essential JavaScript Namespacing Patterns which has the next one. This uses an IIFE to execute the namespace but return an object defining the references to the ‘public’ or exported methods.
var namespace = (function () {
// defined within the local scope
var privateMethod1 = function () { /* ... */ }
var privateMethod2 = function () { /* ... */ }
var privateProperty1 = 'foobar';
return {
// the object literal returned here can have as many
// nested depths as you wish, however as mentioned,
// this way of doing things works best for smaller,
// limited-scope applications in my personal opinion
publicMethod1: privateMethod1,
//nested namespace with public properties
properties:{
publicProperty1: privateProperty1
},
//another tested namespace
utils:{
publicMethod2: privateMethod2
}
...
}
})();


Modules


There are 3 basic flavours to the modules, common.js, Asynchronous Module Definition: AMD (yeh I know, easily confused with the chip manufacturer) and ECMAScript 6 Modules.

The last namespace sample I gave is pretty close to the AMD module definitions, this is taken from require.js:
//my/shirt.js now has some dependencies, a cart and inventory
//module in the same directory as shirt.js
define(["./cart", "./inventory"], function(cart, inventory) {
//return an object to define the "my/shirt" module.
return {
color: "blue",
size: "large",
addToCart: function() {
inventory.decrement(this);
cart.add(this);
}
}
}
);

I personally prefer the common.js style, but since we have to work in the browser as well as the server here is the require.js wrapper for common.js modules:
define(['require', 'exports', 'module', 'dep1'], function (require, exports, module, dep1) {
exports.tripple = function tripple(val1) {
return val1 * 3;
};

//require is useful for cyclic dependencies
exports.dependantFunc = function () {
return tripple(require('dep2').getValue());
};
});

In a later post I will go into more detail on how to use ECMAScript 6 (via Babel and require.js) in the browser.


Documentation (JSDoc)


Writing documentation is the bane of nearly every developer’s existence. However in JavaScript it’s a must. Due to JavaScript’s very loose typing of variables, it makes it near impossible for a modern IDE to give you good autocomplete suggestions which make coding that much more efficient. This lack of typing has resulted in many languages which are compiled into JavaScript such as: TypeScript (which IntelliJ actually uses for a source of it’s API definitions for open source libraries).

The JavaDoc for JavaScript: JSDoc3 / @use JSDoc, supports defining types for variables, self/this, return values etc. Together with the added syntax from Google’s Closure Compiler, you can get near Java-like API autocomplete suggestions in IntelliJ. However, I did have some difficulties with modules, which if I ever remember where I got it right, I’ll post up.


Strict Mode


Strict Mode (MDN), is a feature added in ECMAScript 5 which turns on restrictions in the scope it's defined that help catch coding typos and also helps the compiler boost the performance. Mistakes take were quietly worked around in non-strict mode are raised as errors.
//use strict mode globally within this file
"use strict";

function someStrictFunc() {
//use strict mode within this function, single or double quotes
'use strict';

Additional Topics


Promises

Many frameworks use them and as a result they’ve been included in ECMAScript 6. Chrome already supports them but for other browsers you’ll need a shim like: promise.js. They allow much better chaining of dependant functions.

Mixins

A different take at inheritance. Those who’ve used less or sass will recognise the concept. A fresh look at JavaScript Mixins

General OO (namespaces and classes)

This is another good article covering namespaces and classes in more detail. Preparing Yourself for Modern JavaScript Development

Friday, 3 July 2015

Cygwin, access control, default groups and just getting it playing nice

correcting current and default permissions

If you've been messing with your permissions on copying data across from another NTFS system, some of the owners/groups may be off and the even after correcting the owners and their permissions, any new files don't have the right defaults. This simple script replaces the ACL records for each file and directory, giving the default permissions specified.
find $1 -type f -exec setfacl -f facl {} \;
find $1 -type d -exec setfacl -f dacl {} \;
dacl - directory permissions
user::rwx
group::rwx
other:r-x
default:user::rwx
default:group::rwx
default:other:r-x
facl - File Permissions
user::rw-
group::rw-
other:r--
default:user::rw-
default:group::rw-
default:other:r--

Specifying the default groups for users

The documentation for cygwin is in depth but doesn't simply answer the question: How do I set the default group for a user? (in the out of the box configuration).
The starting point is the mkpasswd utility. These are the following steps, assuming non-special accounts - the SYSTEM ("Local System") account can't be changed.
  1. > mkpasswd -l -u MyUser >> /etc/passwd
    This creates a mapping record in the passwd file with the default group.
  2. > id MyGroup
    uid=197613(mygroup) gid=197613(mygroup) groups=11(Authenticated Users),197613(mygroup)
    The gid is the group id we need, i.e. 197613
  3. > vi /etc/passwd
  4. Change the 4th field to the gid above i.e.
    MyUser:*:197608:197121:U-MY-PC\MyUser,S-1-5-21-818915124-687840057-3584311183-1000:/home/MyUser:/bin/bash
    becomes
    MyUser:*:197608:197613:U-MY-PC\MyUser,S-1-5-21-818915124-687840057-3584311183-1000:/home/MyUser:/bin/bash

sudo - run command as Administrator

This isn't a full implementation but it elevates the current user to admin rights.
Getting the escaping to work properly is tricky, see this article for how the rules are applied with bash.
# TODO the hardcoded path is a bit hacky
if [ "$1" == "-" ]; then
 #echo "interactive shell"
 cygstart --action=runas "C:\\cygwin64\\bin\\mintty.exe" -h always
else
 #echo "evelavated command"
 CMD="cd $(pwd); $@"
 #note the order of the quotes below, we want to send the cmd as a single arg but surrounded by double quotes
 #use cygstart -v to debug
 cygstart --action=runas "C:\\cygwin64\\bin\\mintty.exe" -l sudo.log -h error -e /usr/bin/bash -l -c "\" $CMD \""
fi

#to use:
./runas-admin.sh somescript.sh param1
# or to open a shell
./runas-admin.sh -
The latest version is kept here: runas-admin.sh

su - switch user

This is a near drop-in replacement, not all options are supported and it differs from the standard linux in that in opens up a new window instead of using the same terminal.
if [ $DO_LOGIN == 1 ]; then
 TTYCMD="- $COMMAND"
else
 TTYCMD="$COMMAND"
fi

if [ $USER != 'root' ]; then
 cygstart --hide cmd.exe /c "\"\"%WINDIR%\\system32\\runas.exe\" /savecred /user:$USER \"$(cygpath -w /usr/bin/mintty.exe) $TTYCMD \"\""
else
        cygstart --action=runas "C:\\cygwin64\\bin\\mintty.exe" -h error -e $TTYCMD
fi

# to use, loging into a new shell with MyUser:
./runas-admin.sh - MyUser
Download the full script here: switch-user.sh

Alternate method: cyglsa

Reading through the security documentation, it gives different instructions on how to get login tokens (at a programatic level). From what I can tell its for sshd and/or services only, there is no mention of support for su/sudo commands.
/usr/bin/cyglsa-config