Sunday, 29 November 2015

Running a meteor shell on a standalone server

For those that want to connect to meteor's shell running on a standalone/self-maintained server, the standard command you use in the development environment doesn't work. Fortunately you can 'trick' it into allowing the `meteor shell` command.
APPDIR=/opt/bitnami/apps/myapp
export METEOR_SHELL_DIR="$APPDIR/.meteor/local/shell"
# other settings
# ...

exec node $APPDIR/bundle/main.js
cd /opt/bitnami/apps/myapp
mkdir -p .meteor/local/shell
echo > .meteor/packages
echo 'METEOR@1.2.1\n' > .meteor/release

meteor shell

Monday, 5 October 2015

Javascript (ECMAScript 5.1) Refresher

I’ve been recently working heavily on JavaScript based applications in both Node.js and a Chrome extension. After working mostly on Java, I thought I’d share the syntax, conventions and the new features of ECMAScript 5.1 and parts of the still pending version, ES6/ECMAScript 2015, that I’ve come across.

Comparisons Operators

==
equal to
===
equal value and equal type
!=
not equal
!==
not equal value or not equal type

I saw something else I thought was a fancy notation:

if (!!something)
//which is not a special operator, it just converts it to boolean, i.e a double not.
(!!something) === (!(!something))
//this would only be useful if you want to set a boolean to something truthy, i.e.
var a = "somevalue";
var asBoolean = !!"somevalue";

Object Initializer

Something I had thought was not a fully adopted syntax, was being able to specify the getter / setter functions in the initializer. (MDN Object initializer)
var o = {
func1: function (p1, p2) {},
get prop1() { return this._prop1; },
set prop1(value) { this._prop1 = value; },
};
This creates:

  1. a function called func1: o.func1(param1, param2)
  2. setter function  for "prop1": o.prop1 = 'newvalue';
  3. getter functions for "prop2": var oldValue = o.prop1;
Which is equivalent to more the verbose way:
var o = Object.create({}, {
func1: {
value: function(p1, p2) { },
},
prop1: {
get function() { return this._prop1; },
set function(value) { this._prop1 = value; },
},
});

Other odd syntax


The OR operator:
// this allows fall-back assignment in case a variable is not defined or set
var myNamespace = window.myNamespace || {};
// which is equivalent to: (there are many more ways to write this but this is the simplest direct translation)
var myNamespace;
// if you just use myNamespace not window.myNamespace you'll get a variable not defined error
if (window.myNamespace) // if defined and set to a value
myNamespace = window.myNamespace;
else
myNamespace = {};

This is sometimes written like this:

var myNamespace = window.myNamespace || window.myNamespace={};

Which although it seems wrong, is actually valid syntax. An assignment of a variable statement returns a value, so (window.myNamespace={}) is equal to {}. It’s nice and compact but does border on readability vs magic syntax (those trying debug Groovy scripts will understand what I mean).


Next the AND operator:
DEBUG && console.log("My debug statement");

// is a compact way of writing
if (DEBUG)
console.log("My debug statement");
Don't abuse this one, as its easy to make JavaScript look like a completely different language. My recommendation - used for a few scenarios and use it consistently. Pretty code, is readable code.
The COMMA operator:

This one I really did think was some magic syntax until I looked it up. By the example given on MDN comma operator reference my best bet is it was intended only for the for loop, which in hindsight I vaguely knew it was legal syntax there.
This just executes each statement in turn, returning the value of the last statement.

if (doSomething)
return retValue = invokeMethod(), console.log(retValue), retValue;
else
return false;

Scope & Closures

Reading through a lot of the articles there is a lot of talk about closures and warnings about scope when using nested function. Reading some of the examples out there could lead you to code over defensibly (which I realised I had been doing) or just left wondering what the big deal is. Function scoped variables in JavaScript behave almost the way you would think at first glance.

  • Nested functions are able to access the variables of parent functions scope (all the way up to the global scope --> window)
  • Parent functions can't access the variables of nested functions
  • Nested functions can still access the parent functions variables after the parent function(s) have finished executing --> they get a copy of the stack
  • Each invocation of the parent function creates a new stack so someFunc(1) defined variables won't overlap with someFunc(2)
  • If myFunc1 and myFunc2 were defined in parentFunc:
    var obj = parentFunc(123);
    obj.myFunc1();
    obj.myFunc2();
    Then myFunc1 and myFunc2 could both alter variables defined in parentFunc

This last point is the only one that you have to be wary of. This blog post has more detail, however here is the snippet which highlights the problem:

function f() {
var arr = [ "red", "green", "blue" ];
var result = [];
for(var i=0; i<arr.length-1; i++) {
var func = function() {
return arr[i];
};
result.push(func);
}
return result;
}

This has the effect of every function in the result returning "blue" as each function sees the latest value of i, which is the length of the array.


Function expressions, declarations etc

For the most part it doesn't matter whether you use a function declaration or a function expression, there's listing of all the function types and their minor differences here: (MDN Functions reference). The only difference that should affect most people is:


  • Function declarations can be used before they are declared as they are 'hoisted' automatically by the parser.
  • Function expressions on the other hand, need to be defined before they are invoked or assigned.

Classes


There are many different patterns people use to create classes that together with the ECMAScript 6 shorthand. I won’t show them all but here is the one I use for ECMAScript 5.

var Person = function Person(name) {
this.name = name;
this.canTalk = true;
};
_.extend(Person.prototype, {
prop1: null,
prop2: 123,
prop3: {},
prop4: "",
func1: function () {
console.log("%s's prop2 is %s",this.name, this.prop2);
}
});

var tim = new Person("Tim");
tim.func1();
// response is:
// > Tim's prop2 is 123

I’ve seem some people use Person.prototype = {..} which has the effect of overwriting the Person.prototype.constructor property which doesn’t have immediate problems but can cause issues with inheritance and some frameworks. This Stackoverflow discussion shows why.

I use underscore.js in this example, but jQuery does the same, as a shorthand for having to write Person.prototype.prop1 = null; etc to set all the methods and properties on the prototype so they are shared between object instances.

ECMAScript 6 allows you to easily define the functions in a class but doesn’t allow you to set the starting value for instance variables on the prototype. i.e. you’ll still need to do _.extend(Person.prototype, { prop1: … prop2: .. }); but you’ll just define the functions with the class.


Namespaces


In JavaScript, namespaces (aka packages/modules) are written simply as objects:
// global namespace
var MYAPP = MYAPP || {};
// sub namespace
MYAPP.event = {};

They’re nothing magical and not built into the language, more of a technique of scoping your variables and functions so they don’t clash with the numerous JavaScript libraries now in use. MDN has an intro on namespaces in their Object Orientated section. There’s a Stackoverflow discussion: What's the best way to create a JavaScript namespace?

The syntax below I found the best supported by IDE’s as any good programmer knows, sometimes the best way to do something is the quickest/more efficient way by using the supported libraries/tools to make our lives easier. This uses a ‘immediately-invoked function expression’ (IIFE) that passes in the global object (in this case the window) to define the namespace so it can be referenced by other code. This also makes it easier to wire into something that doesn’t run in the browser, i.e Node.js, however the syntax below is better suited to that..
((function(global) {
/**
*
*/
MyNamespace.method1 = function() {
//... do something
};

//export
global.MyNamespace = MyNamespace;
})(window);

I came across this article Essential JavaScript Namespacing Patterns which has the next one. This uses an IIFE to execute the namespace but return an object defining the references to the ‘public’ or exported methods.
var namespace = (function () {
// defined within the local scope
var privateMethod1 = function () { /* ... */ }
var privateMethod2 = function () { /* ... */ }
var privateProperty1 = 'foobar';
return {
// the object literal returned here can have as many
// nested depths as you wish, however as mentioned,
// this way of doing things works best for smaller,
// limited-scope applications in my personal opinion
publicMethod1: privateMethod1,
//nested namespace with public properties
properties:{
publicProperty1: privateProperty1
},
//another tested namespace
utils:{
publicMethod2: privateMethod2
}
...
}
})();


Modules


There are 3 basic flavours to the modules, common.js, Asynchronous Module Definition: AMD (yeh I know, easily confused with the chip manufacturer) and ECMAScript 6 Modules.

The last namespace sample I gave is pretty close to the AMD module definitions, this is taken from require.js:
//my/shirt.js now has some dependencies, a cart and inventory
//module in the same directory as shirt.js
define(["./cart", "./inventory"], function(cart, inventory) {
//return an object to define the "my/shirt" module.
return {
color: "blue",
size: "large",
addToCart: function() {
inventory.decrement(this);
cart.add(this);
}
}
}
);

I personally prefer the common.js style, but since we have to work in the browser as well as the server here is the require.js wrapper for common.js modules:
define(['require', 'exports', 'module', 'dep1'], function (require, exports, module, dep1) {
exports.tripple = function tripple(val1) {
return val1 * 3;
};

//require is useful for cyclic dependencies
exports.dependantFunc = function () {
return tripple(require('dep2').getValue());
};
});

In a later post I will go into more detail on how to use ECMAScript 6 (via Babel and require.js) in the browser.


Documentation (JSDoc)


Writing documentation is the bane of nearly every developer’s existence. However in JavaScript it’s a must. Due to JavaScript’s very loose typing of variables, it makes it near impossible for a modern IDE to give you good autocomplete suggestions which make coding that much more efficient. This lack of typing has resulted in many languages which are compiled into JavaScript such as: TypeScript (which IntelliJ actually uses for a source of it’s API definitions for open source libraries).

The JavaDoc for JavaScript: JSDoc3 / @use JSDoc, supports defining types for variables, self/this, return values etc. Together with the added syntax from Google’s Closure Compiler, you can get near Java-like API autocomplete suggestions in IntelliJ. However, I did have some difficulties with modules, which if I ever remember where I got it right, I’ll post up.


Strict Mode


Strict Mode (MDN), is a feature added in ECMAScript 5 which turns on restrictions in the scope it's defined that help catch coding typos and also helps the compiler boost the performance. Mistakes take were quietly worked around in non-strict mode are raised as errors.
//use strict mode globally within this file
"use strict";

function someStrictFunc() {
//use strict mode within this function, single or double quotes
'use strict';

Additional Topics


Promises

Many frameworks use them and as a result they’ve been included in ECMAScript 6. Chrome already supports them but for other browsers you’ll need a shim like: promise.js. They allow much better chaining of dependant functions.

Mixins

A different take at inheritance. Those who’ve used less or sass will recognise the concept. A fresh look at JavaScript Mixins

General OO (namespaces and classes)

This is another good article covering namespaces and classes in more detail. Preparing Yourself for Modern JavaScript Development

Friday, 3 July 2015

Cygwin, access control, default groups and just getting it playing nice

correcting current and default permissions

If you've been messing with your permissions on copying data across from another NTFS system, some of the owners/groups may be off and the even after correcting the owners and their permissions, any new files don't have the right defaults. This simple script replaces the ACL records for each file and directory, giving the default permissions specified.
find $1 -type f -exec setfacl -f facl {} \;
find $1 -type d -exec setfacl -f dacl {} \;
dacl - directory permissions
user::rwx
group::rwx
other:r-x
default:user::rwx
default:group::rwx
default:other:r-x
facl - File Permissions
user::rw-
group::rw-
other:r--
default:user::rw-
default:group::rw-
default:other:r--

Specifying the default groups for users

The documentation for cygwin is in depth but doesn't simply answer the question: How do I set the default group for a user? (in the out of the box configuration).
The starting point is the mkpasswd utility. These are the following steps, assuming non-special accounts - the SYSTEM ("Local System") account can't be changed.
  1. > mkpasswd -l -u MyUser >> /etc/passwd
    This creates a mapping record in the passwd file with the default group.
  2. > id MyGroup
    uid=197613(mygroup) gid=197613(mygroup) groups=11(Authenticated Users),197613(mygroup)
    The gid is the group id we need, i.e. 197613
  3. > vi /etc/passwd
  4. Change the 4th field to the gid above i.e.
    MyUser:*:197608:197121:U-MY-PC\MyUser,S-1-5-21-818915124-687840057-3584311183-1000:/home/MyUser:/bin/bash
    becomes
    MyUser:*:197608:197613:U-MY-PC\MyUser,S-1-5-21-818915124-687840057-3584311183-1000:/home/MyUser:/bin/bash

sudo - run command as Administrator

This isn't a full implementation but it elevates the current user to admin rights.
Getting the escaping to work properly is tricky, see this article for how the rules are applied with bash.
# TODO the hardcoded path is a bit hacky
if [ "$1" == "-" ]; then
 #echo "interactive shell"
 cygstart --action=runas "C:\\cygwin64\\bin\\mintty.exe" -h always
else
 #echo "evelavated command"
 CMD="cd $(pwd); $@"
 #note the order of the quotes below, we want to send the cmd as a single arg but surrounded by double quotes
 #use cygstart -v to debug
 cygstart --action=runas "C:\\cygwin64\\bin\\mintty.exe" -l sudo.log -h error -e /usr/bin/bash -l -c "\" $CMD \""
fi

#to use:
./runas-admin.sh somescript.sh param1
# or to open a shell
./runas-admin.sh -
The latest version is kept here: runas-admin.sh

su - switch user

This is a near drop-in replacement, not all options are supported and it differs from the standard linux in that in opens up a new window instead of using the same terminal.
if [ $DO_LOGIN == 1 ]; then
 TTYCMD="- $COMMAND"
else
 TTYCMD="$COMMAND"
fi

if [ $USER != 'root' ]; then
 cygstart --hide cmd.exe /c "\"\"%WINDIR%\\system32\\runas.exe\" /savecred /user:$USER \"$(cygpath -w /usr/bin/mintty.exe) $TTYCMD \"\""
else
        cygstart --action=runas "C:\\cygwin64\\bin\\mintty.exe" -h error -e $TTYCMD
fi

# to use, loging into a new shell with MyUser:
./runas-admin.sh - MyUser
Download the full script here: switch-user.sh

Alternate method: cyglsa

Reading through the security documentation, it gives different instructions on how to get login tokens (at a programatic level). From what I can tell its for sshd and/or services only, there is no mention of support for su/sudo commands.
/usr/bin/cyglsa-config

Friday, 26 June 2015

Getting PHP FastCgi Process Manager (FPM) and nginx working in cygwin

Despite the popularity of nginx and php, I was surprised that it wasn't easiy to find a working configuration for PHP-FPM (fast cgi) with a nginx server in front, running on cygwin.
Once I had the right fragments of settings, it was a case of systematically trying them all out.

/etc/php5/php-fpm.conf:

[global]
pid = /var/run/php-fpm.pid

;note: i create a /var/log/php dir owned by the service user/group
;this allows the permissions to be inherited easily on the filesystem
error_log = /var/log/php/fpm-global.log

; cygwin user default is 256
rlimit_files = 1024

;pool configuration, having a pool config per site means you can easily have a separate log file
[www]
;they have to be set but the cygwin version ignores them
;user=service-user
;group-service-group

; The address on which to accept FastCGI requests.
listen = 127.0.0.1:8001
; or
; listen = tmp/php-cgi.socket
; for socket unset these:
listen.owner=service-user
listen.group=service-group

; this allows the process pool to be queried if it appears to be bogging down
pm.status_path = /status

php_admin_flag[log_errors] = on

/etc/php5/php.ini:

The magic to get the pretty permalinks working is on the wordpress site.
error_reporting = -1
display_errors = On
display_startup_errors = On
log_errors = On
log_errors_max_len = 0

/etc/nginx/nginx.conf:

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}


http {
    access_log off;
    
    include       mime.types;
    default_type  application/octet-stream;

    sendfile        on;
    keepalive_timeout  65;

    # Upstream to abstract backend connection(s) for php
    upstream php {
      #server unix:/tmp/php-cgi.socket;
      server 127.0.0.1:8001;
    }

    include site1.conf;
}

/etc/nginx/site1.conf:

server {
  listen       8000;
  server_name  localhost;
  
  ## This should be in your http block and if it is, it's not needed here.
  index index.php;
  root   /cygdrive/c/Projects/site1/wordpress/;
  
  location / {
      try_files $uri $uri/ /index.php?$args;
  }
  rewrite /wp-admin$ $scheme://$host$uri/ permanent;
  
  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  #
  location ~ [^/]\.php(/|$) {           
    fastcgi_split_path_info ^(.+?\.php)(/.*)$;
    if (!-f $document_root$fastcgi_script_name) {
      return 404;
    }
    # This is a robust solution for path info security issue and works with "cgi.fix_pathinfo = 1" in /etc/php.ini (default)
    
    include fastcgi.conf;
    fastcgi_index index.php;
    fastcgi_pass php;
    
    # fastcgi_intercept_errors on;
    fastcgi_connect_timeout 40s;     

    # set the log file here so each site is different
    fastcgi_param PHP_VALUE "error_log = /var/log/php/php-site1.log";
  }
    
  # pass status page request on
  location ~ ^/(status|ping)$ {
    access_log off;
    allow 127.0.0.1;
    deny all;
    
    include fastcgi.conf;
    fastcgi_pass php;
  }
}

Finally, I had some problems connecting from cygwin php to a windows native mysql. Discovered I just had to use the localhost IP to force it over TCP/IP (instead of trying to use unix sockets).

/var/www/wordpress/wp-config.php:

/** MySQL hostname */
define('DB_HOST', '127.0.0.1');

Update 2015/07/02:

I've hit a few problems using cygwin and it's nearly wholly based around permissioning. I've written up a summary of my ad-hoc solution, mainly because I couldn't easily find sample setups and wasted a lot more time than one should on a popular platform.

The launch scripts themselves are a bit hacky but these are the best I've got so far. The cygrunsrv service launcher directly calls the service so you don't get a shell script (though this could easily be changed). I've found running as the "Local System" account causes too many problems relating to group access so setting up a dedicated user is the easiest.

Here are the 2 service install scripts:


/etc/rc.d/init.d/ngix
/etc/init.d/php-fpm


Thursday, 25 June 2015

Remote debugging PHP from phpStorm

Despite all the documentation out there, it seemed to take me longer than it should. Although in hindsight, it could have been working originally as phpStorm's indicator that it's debugging is subtle.

Debug session within phpStorm

Two articles to get you started:

To setup follow do the following:
  1. For chrome (probably the easiest), install this plugin: Chrome Plugin
  2. Configure the plugin and select phpStorm as your IDE.
  3. Next sign onto your remote server and enable xdebug module
    1. for cPanel, click select php version (only 5.4 and 5.5 supported)
    2. update version
    3. check the xdebug module
    4. click update
  4. Edit the php.ini, e.g. for cPanel, edit or create: /home/[username]/public_html/.user.ini
    ; Settings for xdebug
    module was already loaded on this machine:
    ; zend_extension=/opt/alt/php54/usr/lib64/php/modules/xdebug.so
    xdebug.remote_host=[your modems IP address]
    xdebug.remote_port=[a port forwarded through your modem to firewall, e.g. 9800]
    xdebug.remote_enable=1
    
  5. Setup phpStorm's port:
    Settings->Languages & Frameworks-->PHP-->Debug Xdebug port = [remote_port] set above
  6. Setup phpStorm remote debug settings:
    1. Run --> Edit configurations
    2. Add --> PHP Remote Debug
    3. Give it a name
    4. Ide key: "PhpStorm"
    5. Servers (select or add via the "...")
      1. Host: your remote server host
      2. Port: http port usually 80
      3. Debugger: xdebug
      4. Check "use path mappings"
      5. Setup any mappings
      6. Click validate
      7. Click close
    6. Click ok
  7. Run --> Start listening for PHP Debug Connections
  8. In the browser, click the debug icon in the address bar and select debug
  9. visit your sites URL
    You may need to go to the site first, the select debug, then refresh the page.
Note: .user.ini is cached by apache, so it may not work instantly.
When it works the page won't seem to load and in phpStorm the debugger tab should show a clickable green arrow ("Resume Program Execution") and it should show the current frame & variables.

I went through a couple of settings to get here so I may have missed a little tweak along the way.

Profiler

These settings were taken from: Diagnosing slow PHP execution with Xdebug and KCachegrind.
/home/[username]/public_html/.user.ini
; Settings for xdebug
; module was already loaded:
; zend_extension=/opt/alt/php54/usr/lib64/php/modules/xdebug.so
xdebug.profiler_output_dir=/home/[username]/php
xdebug.profiler_append=On
xdebug.profiler_enable_trigger=On
xdebug.profiler_output_name="%R-%u.trace"
xdebug.trace_options=1
xdebug.collect_params=4
xdebug.collect_return=1
xdebug.collect_vars=0
xdebug.profiler_enable=0
The relevant .trace file can be downloaded and loaded into phpStorm via: Tools->Analyse Xdebug Profiler Snapshot.
See Analyzing Xdebug Profiling Data for more information.