Tuesday, January 6, 2015

Develop Meteor Projects with Codenvy

Meteor provides a good framework for developing web based applications. I also like having projects in a cloud based IDE so I can quickly get back to projects whenever I have time regardless of what computer I happen to be near. Codenvy is a nice cloud IDE with some powerful features; but it doesn't exactly come with support for Meteor out of the box. In order to use Meteor with Codenvy we need a custom Docker builder.

Meteor Docker File
FROM codenvy/shellinabox
# 
RUN sudo apt-get update -qq && \
    sudo apt-get -yqq install curl
#
RUN cd /home/user && \
    sudo curl -o /home/user/install_meteor.sh https://install.meteor.com 2> /dev/null && \
    sudo sh install_meteor.sh 2> /dev/null && \
    sudo meteor create app && \
    sudo rm app/app.*
#
ENV CODENVY_APP_PORT_8081_HTTP 8081
EXPOSE 8081
#
# Add meteor packages
RUN cd /home/user/app && sudo meteor add accounts-ui
# 
ADD $app$ /home/user/app/
# 
CMD cd /home/user/app && sudo meteor --port 8081
#


The solution works well but isn't perfect. Your code gets packaged into the Docker container so you can't edit files in your IDE and have the changes hot-deployed into your running application. You will have to stop and restart your application to see changes. Hopefully that is something Codenvy will fix in the future with Docker volumes.
Hot-deployed changes works when you replace
ADD $app$ /home/user/app/
with
VOLUME ["/home/user/app"]
ENV CODENVY_APP_BIND_DIR /home/user/app
but you also have to make sure you have a .meteor directory in your project with the correct configuration files.

Sunday, October 12, 2014

Provisioning Docker containers with Weave and Upstart

Someday it will be nice to pick a hosting service and have all of your Docker containers magically hook together correctly, securely, and quickly scale. But it just isn't quite there yet. Weave is a nice utility for creating an overlay network which can keep your containers connected on different servers even in different data centers or different hosting providers.

Below is a Upstart configuration that will allow you to keep Weave running and come back after server reboot. Simply do "sudo start weave" to get the service running the first time. You should also edit your /etc/default/docker.io file to add in the options to keep it from restarting containers since our Upstart service is designed to do that for us: DOCKER_OPTS="-r=false"

/etc/init/weave.conf
description "Weave Docker Network Service"
start on filesystem and started docker.io
stop on runlevel [!2345]
respawn
script
 /usr/bin/docker rm -f weave || true
 /usr/local/bin/weave launch 10.100.0.1/16 {other server public ip(s) here}
 /usr/bin/docker attach weave
end script

We make sure docker is running first before starting Weave. This configuration was on Ubuntu 14.04; prior to that the docker service was just docker without the io. You can run this on each of your servers with a different 10.100.0.x ip and specifying all of the other servers ip after it so they stay connected.

Next we can start a typical Docker container like a Mongo database on one of our servers. We will keep it running with Upstart as well and make sure it starts after Weave.

/etc/init/mongo.conf
description "Mongo Database Service"
start on filesystem and started weave
stop on runlevel [!2345]
respawn
pre-start script
 mkdir -p /data/db
end script
script
 /usr/bin/docker rm -f mongo || true
 /usr/local/bin/weave run 10.100.1.1/24 --name mongo mongo:2.6
 /usr/bin/docker attach mongo
end script

Go to one of your other servers and use the script below to connect to your Mongo database running in a container on a different server.
#!/bin/sh
# connect to the mongo db from any weave enabled host

sudo docker rm -f db_connect; sudo weave run 10.100.1.99/24 --name db_connect -it mongo:2.6 /bin/sh -c "mongo 10.100.1.1"; sudo docker attach db_connect


Try it out on Digital Ocean using a couple of their smallest plans. The link will give you $10 credit - enough to run two of the smallest servers for a month.

Wednesday, June 18, 2014

CoreOS Floating IP

Using fleetctl to run Docker containers in CoreOS will start the container on a random member of the cluster and then restart it attomatically on another member if that one becomes unavailable.  This works great for most containers but you need a consistent way to reach an entry point whether it is a load balancer or an apache front end.

We can use the ExeStartPre and ExeStopPost commands from the fleetctl unit file to add and delete a virtual IP when our service is started and stopped.

[Unit]
Description=Example
After=docker.service
Requires=docker.service

[Service]
Restart=always
ExecStartPre=/usr/bin/sudo /usr/bin/ip addr add 172.17.8.100/24 dev enp0s8
ExecStart=/usr/bin/docker run --rm=true --name webapp -p 80:80 coreos/apache /usr/sbin/apache2ctl -D FOREGROUND

ExecStop=/usr/bin/docker stop webapp
ExecStopPost=/usr/bin/sudo /usr/bin/ip addr del 172.17.8.100/24 dev enp0s8

[Install]
WantedBy=multi-user.target

Sunday, August 21, 2011

Quick RESTful web services with the Camel cxfbean component

Trying to create a REST web service using the old Camel cxfrs component was somewhat a pain and not very intuitive in my opinion. It was such a pain I didn't even want to write an example post for it. I really wanted something more automated like Jersey.

Luckly the Apache Camel cxfbean component makes putting together a REST web service much easier. Just annotate your service class and use it as the resource for the cxfbean component.

#!/usr/bin/env groovy

@Grab(group="org.apache.camel", module="camel-core", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-jetty", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-cxf", version="2.8.0")
@Grab(group='ch.qos.logback', module='logback-classic', version='0.9.29')
import org.apache.camel.CamelContext
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.impl.SimpleRegistry
import org.apache.camel.builder.RouteBuilder

import javax.ws.rs.*


@Path("/example")
class MyExampleResource{

  @GET
  @Produces("text/plain")
  @Path("hello")
  public String hello(){
    "Hello World!"
  }
}

CamelContext cxt = new DefaultCamelContext()
cxt.registry = new SimpleRegistry()
cxt.registry.registry.put("myResources", [new MyExampleResource()])

cxt.addRoutes(new RouteBuilder(){
  void configure(){
    from("jetty:http://localhost:8080?matchOnUriPrefix=true")
    .to("cxfbean:myResources")
  }
})

cxt.start()

If you want to produce and consume a data format like JSON, you simply pass the necessary processor to the list of cxfbean processors. The CXF libaries include a JSON processor that you can quickly hook up - but I ran into a couple of issues. First you have to annotate your data classes with JAXB annotations (which you may not be able to do depending on your situation.) Second the processor tried to marshal all bean properties (methods that start with get.)

Since I was using Groovy I ran into an issue where Groovy automatically adds a getMetaClass method that returns an Interface and the CXF JSON processor didn't like that. Without an obvious way to tell the processor not to do that I decided to write my own quick processor using the GSON library. Here is a complete example which marshals data in and out from JSON format.

#!/usr/bin/env groovy

@Grab(group="org.apache.camel", module="camel-core", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-jetty", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-cxf", version="2.8.0")
@Grab(group='ch.qos.logback', module='logback-classic', version='0.9.29')
@Grab(group='com.google.code.gson', module='gson', version='1.7.1')
import org.apache.camel.CamelContext
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.impl.SimpleRegistry
import org.apache.camel.builder.RouteBuilder

import javax.ws.rs.*

import javax.ws.rs.ext.MessageBodyWriter
import javax.ws.rs.ext.MessageBodyReader
import java.lang.reflect.Type
import java.lang.annotation.Annotation
import javax.ws.rs.core.MediaType
import javax.ws.rs.core.MultivaluedMap
import com.google.gson.Gson


class Person {
  Integer id
  String name

  public String toString(){
      return "$id $name"
  }
}

@Path("/example")
class MyExampleResource{

  @GET
  @Produces("text/plain")
  @Path("hello")
  public String hello(){
    "Hello World!"
  }

  @GET
  @Produces("application/json")
  @Path("person/{id}")
  public Person getPerson(@PathParam("id") Integer id){
    Person p = new Person()
    p.id = id
    p.name = "Bob"
    return p
  }

  @PUT
  @Consumes("application/json")
  @Produces("application/json")
  @Path("person")
  public Person addPerson(Person p){
    println "Received ${p.toString()}"
    p.id = 123
    return p
  }
}

class GsonProvider implements
    MessageBodyWriter<Object>,
    MessageBodyReader<Object>{

    Gson gson = new Gson()

    boolean isWriteable(Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType){
        mediaType.subtype == 'json'
    }

    long getSize(Object t, Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType){
        0
    }

    void writeTo(Object t, Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType,
        MultivaluedMap<String, Object> stringObjectMultivaluedMap,
        OutputStream outputStream)
        throws java.io.IOException, javax.ws.rs.WebApplicationException{
        outputStream.write(gson.toJson(t).bytes)
    }

    Object readFrom(Class<Object> tClass, Type type,
        Annotation[] annotations, MediaType mediaType,
        MultivaluedMap<String, String> stringStringMultivaluedMap,
        InputStream inputStream) {
        gson.fromJson(inputStream.text, tClass)
    }

    boolean isReadable(Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType) {
        mediaType.subtype == 'json'
    }
}


CamelContext cxt = new DefaultCamelContext()
cxt.registry = new SimpleRegistry()
cxt.registry.registry.put("myResources", [new MyExampleResource()])
cxt.registry.registry.put("jsonProvider", new GsonProvider())

cxt.addRoutes(new RouteBuilder(){
  void configure(){
    from("jetty:http://localhost:8080?matchOnUriPrefix=true")
    .to("cxfbean:myResources?providers=#jsonProvider")
  }
})

cxt.start()

Since originally writing this I discovered how to get the JSON parson that comes with CXF to work with Groovy classes. Include the XmlAccessorType annotation telling it to use FIELD.

...
import javax.xml.bind.annotation.XmlRootElement
import javax.xml.bind.annotation.XmlAccessType
import javax.xml.bind.annotation.XmlAccessorType
import org.apache.cxf.jaxrs.provider.JSONProvider
...
@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
class Person {
  Integer id
  String name

  public String toString(){
      return "$id $name"
  }
}
...
cxt.registry.registry.put("jsonProvider", new JSONProvider())
...

Sunday, November 21, 2010

Simple ESB with Camel and Groovy

I'm not sure if writing an ESB service can be considered fun - but when using Groovy and Camel together it comes pretty close. Camel provedes a nice Java based DSL which makes the routing very clean and easy to follow. When you mix in the power of Groovy you get a very elegant solution.

With Groovy you can say a closure implements an interface. This allows us to embed Groovy code inline as a Processor in a Camel route.

    from("jetty:http://localhost:8080/hello")
    .process({Exchange exchange ->
      exchange.out.setBody "Hello World"
    } as Processor)

We can go one step futher by taking advantage of Groovy's dynamic metaClass to create our own method which takes a closure directly. The following adds a new method 'x' to the RouteDefinition class.

RouteDefinition.metaClass.x = {Closure c -> ((RouteDefinition)delegate).process(c as Processor)}

Our route definition now becomes much more elegant

    from("jetty:http://localhost:8080/hello")
    .x {Exchange ex -> ex.out.setBody "Hello"}
    .x {Exchange ex -> ex.out.body = ex.in.body + " World"}

Add in some Grab annotations from Groovy's Grape extension to automatically download the Camel libraries and the whole thing becomes a single, small, self-contained script.

#!/usr/bin/env groovy

@Grab(group="org.apache.camel", module="camel-core", version="2.5.0")
@Grab(group="org.apache.camel", module="camel-jetty", version="2.5.0")
import org.apache.camel.CamelContext
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.builder.RouteBuilder
import org.apache.camel.Processor
import org.apache.camel.Exchange
import org.apache.camel.model.RouteDefinition

RouteDefinition.metaClass.x = {Closure c -> ((RouteDefinition)delegate).process(c as Processor)}

CamelContext cxt = new DefaultCamelContext()

cxt.addRoutes(new RouteBuilder(){
  void configure(){
    from("jetty:http://localhost:8080/hello")
    .x {Exchange ex -> ex.out.setBody "Hello"}
    .x {Exchange ex -> ex.out.body = ex.in.body + " World"}
  }
})

cxt.start()

Sunday, November 14, 2010

Git version control with a ClearCase like workspace

I have been using the git version control system for a while now. It took a little time to get comfortable with the commands and to fully understand everything it was doing.

When I first read about the fast branch switching I was envisioning something along the lines of a ClearCase view. It has been a while since I used ClearCase - but from what I remember when you switch from one view to another, the virtual file system saved your whole workspace (including things not checked in yet) and then restored your whole workspace from the new view to the exact state it was in when you left. Git doesn't work exactly this way.

When you change to another branch in git it only switches files it is tracking. Any new files you are working on are not affected and therefore appear to magically come over to the new branch. Also, any files you have changed that are tracked will come over if your changes do not cause a merge conflict.

Git does have a stash command that allows you to save off your changes and then apply them again later - but in my opinion it is a little cumbersome. It does not save new files - only ones that are currently tracked, and to apply a specific stash you have to look at the stash list and then apply one as a numerical offset.

For me I would much rather git worked like a clearcase view so that I can save my workspace as-is and then later load it back up. Luckily git is quite powerful and allows you to create some very complex aliases to use right along side other git commands. I created the following two aliases which can be added to your .gitconfig file.
    save = "!git add -A && BR=$(git symbolic-ref HEAD) && echo saving workspace ${BR##*/} && SAVE=$(git stash create Workspace) && git config workspace.${BR##*/}.save $SAVE && git reset --hard HEAD"

    load = "!BR=$(git symbolic-ref HEAD) && echo loading workspace ${BR##*/} && SAVE=$(git config --get workspace.${BR##*/}.save) && git stash apply -q $SAVE && git reset HEAD && git config workspace.${BR##*/}.hist $SAVE && git config --unset workspace.${BR##*/}.save"
Typing git save will save off your workspace as a revision and use a git variable to associate it with the current branch name. You can then change to another branch and come back later. When you come back, typing git load will look for the saved workspace revision associated to the current branch and reapply those changes. This will save and load all files (tracked and untracked) except for the ones you have specifically told git to ignore.

Sunday, January 31, 2010

Simple Groovy POJO Builder

While using Groovy the other day I wanted to use the builder notation on some simple classes I had created.  The Groovy builder classes are very powerful and allow you to create some very nice custom builders - but that was a little more complex than I was wanting.

Given these simple classes:

class Company{
 public def name = ""
 public def deptList = []

 public add(Department d){
  deptList.add(d)
 }
}

class Department{
 public String name = "default"
 public def Employee = []
}

class Employee{
 public String name = "default"
}

I wanted to be able to write something like the following and have the builder dynamically match up all of the methods and properties without me doing anything.

def c = PojoBuilder.build(Company.class){
  name = "Acme"
  Department{
   name = "HR"
   Employee{
    name = "John"
   }
   Employee{
    name = "Jack"
   }
  }
  Department{
   name = "IT"
   Employee{
    name = "Jimmy"
   }
   Employee{
    name = "James"
   }
  }
 }

As it turns out, the code to achieve this is rather short thanks to the powerful features of Groovy. A method similar to the following is a good start for the build method and allows us to call a closure on an instance of our class. From this closure we have direct access to all of the objects properties and methods - so calling name = "Acme" just works.

class PojoBuilder{

 public static Object build(Class cls, Closure c){
  Object o = cls.newInstance()
  c.delegate = o
  c.resolveStrategy = Closure.DELEGATE_FIRST
  c.call()
  return o
 }


An interesting feature of this is that the Department {...} syntax looks like a method call with a closure as a single parameter. If our Company class had a method
Department(Closure c)
that method would get called... but we don't want to have to add methods like that to our simple classes.

Luckily Groovy has a missingMethod functionality on the MetaClass which gets called when any method is missing. We can use that to add in our own method which can find a class matching the name, check for any method called "add" that accepts our found class, and optionally any properties with the same name that are an ArrayList. If found, dynamically add that method to our class and call it.

The complete example can be found here. It also includes a nice feature where you pass in the classes you want to use and the names you want to use in your builder syntax.