Monday, July 18, 2016

Less pre-processing with Maven and Jetty

Continuing on from the previous post we will set up a pre-processor for .less files that will continue to process file changes for Jetty to pick up. Regardless of the Java web framework you are using the file layout generally looks similar to the following.

src/
   main/
      java/
      less/
      webapp/
         js/
         css/
         index.html
Jetty will pick up and changes to files made under the webapp directory so we want our less pre-processor to monitor files in the less/ directory put the compiled css files into the css/ directory where Jetty will pick them up.

The standard less compiler is actually written in JavaScript and is available as a Node NPM module. There is also a watch-less NPM module that will monitor files and run the less compiler when changes are detected. To use node modules we will use the frontend-maven-plugin.

<plugin>
    <groupId>com.github.eirslett</groupId>
    <artifactId>frontend-maven-plugin</artifactId>
    <version>0.0.29</version>

    <configuration>
        <nodeVersion>v0.11.14</nodeVersion>
        <npmVersion>2.13.4</npmVersion>
        <installDirectory>.</installDirectory>
        <workingDirectory>.</workingDirectory>
    </configuration>

    <executions>
        <execution>
            <id>install node and npm</id>
            <goals>
                <goal>install-node-and-npm</goal>
            </goals>
        </execution>

        <execution>
            <id>npm install</id>
            <goals>
                <goal>npm</goal>
            </goals>
            <configuration>
                <arguments>install</arguments>
            </configuration>
        </execution>
    </executions>
</plugin>
This will install Node and NPM and then install all NPM modules defined in the package.json file in the root of our project.
{
  "dependencies": {
    "watch-less": "emmostrom/watch-less"
  }
}
This is a version of watch-less I have modified to watch std-in and to have an option to just run once and not monitor files. Now we can modify our groovy plugin code from the previous post to build a process for our watch-less program.

if (isJettyRun) {
  new ProcessBuilder(
    ["node/node", "node_modules/watch-less/cli.js", 
     "-d", "src/main/less/app", 
     "-r", "src/main/webapp/css", 
     "-e", ".css", 
     "--watch-stdin"]
  ).inheritIO().directory(project.getBasedir()).start()                             
} else {
  def proc = new ProcessBuilder(
    ["node/node", "node_modules/watch-less/cli.js", 
     "-d", "src/main/less/app", 
     "-r", "src/main/webapp/css", 
     "-e", ".css", 
     "--once"]
  ).redirectErrorStream(true).directory(project.getBasedir()).start()
  proc.waitForOrKill(15000)
  proc.in.eachLine { line -> println line }
}

Wednesday, May 4, 2016

Live pre-processing with Maven and Jetty

Running
mvn jetty:run
works well for development as Jetty will reload your web app files (js, css, html, ...) when they change without needing to restart. However handling files that need pre-processed first like .less files is a little more difficult since Jetty can't detect that a specific process in Maven needs re-run.

One solution is to use an external tool to do the pre-processing. As long as the external tool has a file watcher option we can leave it running in the background and then stop it when Maven stops.

<plugin>
<groupId>org.codehaus.groovy.maven</groupId>
<artifactId>gmaven-plugin</artifactId>
<executions>
    <execution>
        <phase>process-resources</phase>
        <goals>
            <goal>execute</goal>
        </goals>
        <configuration>
            <source><![CDATA[
            /*
             * If running in Jetty - pre-process when they change
             * Otherwise just run once
             */
            boolean isJettyRun = false
            List<String> goals = session.getGoals()
            for (String s : goals)
                if (s.equals('jetty:run'))
                    isJettyRun = true

            if (isJettyRun) {
                new ProcessBuilder(["external_app", "--watch_arg", "--watch_std_in"])
                .inheritIO().directory(project.getBasedir()).start()
            } else {
                def proc = new ProcessBuilder(["external_app", "--run_once_arg"])
                .redirectErrorStream(true).directory(project.getBasedir()).start()

                proc.waitForOrKill(15000)
                proc.in.eachLine { line -> println line }
            }
            ]]></source>
        </configuration>
    </execution>
</executions>
</plugin>
Running Groovy inside Maven is a nice quick way to implement a custom plugin. This one is set up to run in the process-resources phase of the Maven life cycle. The first thing we do is determine if we are running with jetty:run or as part of a normal build because we want to handle the situations differently.

When running as part of the normal build process everything is straight forward - build up a process that uses our external tool and pass it whatever arguments it needs to process files once. We also redirect the error stream so we can get errors and output together to display with the rest of the Maven output. We kill the process if it takes longer than 15 seconds and the loop through the output writing it out to the console.

When running with Jetty, our external tool will have to support some special options. It will need to be able to keep itself running and watch for file changes to re-do the pre-processing. It will also need to watch std-in and exit when std-in closes. That last option may sound a little strange as a requirement but there is a reason for it. We could have added shutdown hooks and killed our process when Maven shuts down; but shutdown hooks won't run if Maven is stopped with a SIG_KILL. If you run Maven inside an IDE like IntelliJ, the stop button does a SIG_KILL and will leave your processes running. So to work around this we require our external tool to watch for std-in to close and then when we build our process we inherit the IO of the parent. Now std-in of our process is the same as the std-in of Maven and when Maven shuts down for any reason our process will exit itself.

The next post will build on this and show a complete example for live pre-processing of .less files.

Tuesday, January 6, 2015

Develop Meteor Projects with Codenvy

Meteor provides a good framework for developing web based applications. I also like having projects in a cloud based IDE so I can quickly get back to projects whenever I have time regardless of what computer I happen to be near. Codenvy is a nice cloud IDE with some powerful features; but it doesn't exactly come with support for Meteor out of the box. In order to use Meteor with Codenvy we need a custom Docker builder.

Meteor Docker File
FROM codenvy/shellinabox
# 
RUN sudo apt-get update -qq && \
    sudo apt-get -yqq install curl
#
RUN cd /home/user && \
    sudo curl -o /home/user/install_meteor.sh https://install.meteor.com 2> /dev/null && \
    sudo sh install_meteor.sh 2> /dev/null && \
    sudo meteor create app && \
    sudo rm app/app.*
#
ENV CODENVY_APP_PORT_8081_HTTP 8081
EXPOSE 8081
#
# Add meteor packages
RUN cd /home/user/app && sudo meteor add accounts-ui
# 
ADD $app$ /home/user/app/
# 
CMD cd /home/user/app && sudo meteor --port 8081
#


The solution works well but isn't perfect. Your code gets packaged into the Docker container so you can't edit files in your IDE and have the changes hot-deployed into your running application. You will have to stop and restart your application to see changes. Hopefully that is something Codenvy will fix in the future with Docker volumes.
Hot-deployed changes works when you replace
ADD $app$ /home/user/app/
with
VOLUME ["/home/user/app"]
ENV CODENVY_APP_BIND_DIR /home/user/app
but you also have to make sure you have a .meteor directory in your project with the correct configuration files.

Sunday, October 12, 2014

Provisioning Docker containers with Weave and Upstart

Someday it will be nice to pick a hosting service and have all of your Docker containers magically hook together correctly, securely, and quickly scale. But it just isn't quite there yet. Weave is a nice utility for creating an overlay network which can keep your containers connected on different servers even in different data centers or different hosting providers.

Below is a Upstart configuration that will allow you to keep Weave running and come back after server reboot. Simply do "sudo start weave" to get the service running the first time. You should also edit your /etc/default/docker.io file to add in the options to keep it from restarting containers since our Upstart service is designed to do that for us: DOCKER_OPTS="-r=false"

/etc/init/weave.conf
description "Weave Docker Network Service"
start on filesystem and started docker.io
stop on runlevel [!2345]
respawn
script
 /usr/bin/docker rm -f weave || true
 /usr/local/bin/weave launch 10.100.0.1/16 {other server public ip(s) here}
 /usr/bin/docker attach weave
end script

We make sure docker is running first before starting Weave. This configuration was on Ubuntu 14.04; prior to that the docker service was just docker without the io. You can run this on each of your servers with a different 10.100.0.x ip and specifying all of the other servers ip after it so they stay connected.

Next we can start a typical Docker container like a Mongo database on one of our servers. We will keep it running with Upstart as well and make sure it starts after Weave.

/etc/init/mongo.conf
description "Mongo Database Service"
start on filesystem and started weave
stop on runlevel [!2345]
respawn
pre-start script
 mkdir -p /data/db
end script
script
 /usr/bin/docker rm -f mongo || true
 /usr/local/bin/weave run 10.100.1.1/24 --name mongo mongo:2.6
 /usr/bin/docker attach mongo
end script

Go to one of your other servers and use the script below to connect to your Mongo database running in a container on a different server.
#!/bin/sh
# connect to the mongo db from any weave enabled host

sudo docker rm -f db_connect; sudo weave run 10.100.1.99/24 --name db_connect -it mongo:2.6 /bin/sh -c "mongo 10.100.1.1"; sudo docker attach db_connect


Try it out on Digital Ocean using a couple of their smallest plans. The link will give you $10 credit - enough to run two of the smallest servers for a month.

Wednesday, June 18, 2014

CoreOS Floating IP

Using fleetctl to run Docker containers in CoreOS will start the container on a random member of the cluster and then restart it attomatically on another member if that one becomes unavailable.  This works great for most containers but you need a consistent way to reach an entry point whether it is a load balancer or an apache front end.

We can use the ExeStartPre and ExeStopPost commands from the fleetctl unit file to add and delete a virtual IP when our service is started and stopped.

[Unit]
Description=Example
After=docker.service
Requires=docker.service

[Service]
Restart=always
ExecStartPre=/usr/bin/sudo /usr/bin/ip addr add 172.17.8.100/24 dev enp0s8
ExecStart=/usr/bin/docker run --rm=true --name webapp -p 80:80 coreos/apache /usr/sbin/apache2ctl -D FOREGROUND

ExecStop=/usr/bin/docker stop webapp
ExecStopPost=/usr/bin/sudo /usr/bin/ip addr del 172.17.8.100/24 dev enp0s8

[Install]
WantedBy=multi-user.target

Sunday, August 21, 2011

Quick RESTful web services with the Camel cxfbean component

Trying to create a REST web service using the old Camel cxfrs component was somewhat a pain and not very intuitive in my opinion. It was such a pain I didn't even want to write an example post for it. I really wanted something more automated like Jersey.

Luckly the Apache Camel cxfbean component makes putting together a REST web service much easier. Just annotate your service class and use it as the resource for the cxfbean component.

#!/usr/bin/env groovy

@Grab(group="org.apache.camel", module="camel-core", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-jetty", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-cxf", version="2.8.0")
@Grab(group='ch.qos.logback', module='logback-classic', version='0.9.29')
import org.apache.camel.CamelContext
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.impl.SimpleRegistry
import org.apache.camel.builder.RouteBuilder

import javax.ws.rs.*


@Path("/example")
class MyExampleResource{

  @GET
  @Produces("text/plain")
  @Path("hello")
  public String hello(){
    "Hello World!"
  }
}

CamelContext cxt = new DefaultCamelContext()
cxt.registry = new SimpleRegistry()
cxt.registry.registry.put("myResources", [new MyExampleResource()])

cxt.addRoutes(new RouteBuilder(){
  void configure(){
    from("jetty:http://localhost:8080?matchOnUriPrefix=true")
    .to("cxfbean:myResources")
  }
})

cxt.start()

If you want to produce and consume a data format like JSON, you simply pass the necessary processor to the list of cxfbean processors. The CXF libaries include a JSON processor that you can quickly hook up - but I ran into a couple of issues. First you have to annotate your data classes with JAXB annotations (which you may not be able to do depending on your situation.) Second the processor tried to marshal all bean properties (methods that start with get.)

Since I was using Groovy I ran into an issue where Groovy automatically adds a getMetaClass method that returns an Interface and the CXF JSON processor didn't like that. Without an obvious way to tell the processor not to do that I decided to write my own quick processor using the GSON library. Here is a complete example which marshals data in and out from JSON format.

#!/usr/bin/env groovy

@Grab(group="org.apache.camel", module="camel-core", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-jetty", version="2.8.0")
@Grab(group="org.apache.camel", module="camel-cxf", version="2.8.0")
@Grab(group='ch.qos.logback', module='logback-classic', version='0.9.29')
@Grab(group='com.google.code.gson', module='gson', version='1.7.1')
import org.apache.camel.CamelContext
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.impl.SimpleRegistry
import org.apache.camel.builder.RouteBuilder

import javax.ws.rs.*

import javax.ws.rs.ext.MessageBodyWriter
import javax.ws.rs.ext.MessageBodyReader
import java.lang.reflect.Type
import java.lang.annotation.Annotation
import javax.ws.rs.core.MediaType
import javax.ws.rs.core.MultivaluedMap
import com.google.gson.Gson


class Person {
  Integer id
  String name

  public String toString(){
      return "$id $name"
  }
}

@Path("/example")
class MyExampleResource{

  @GET
  @Produces("text/plain")
  @Path("hello")
  public String hello(){
    "Hello World!"
  }

  @GET
  @Produces("application/json")
  @Path("person/{id}")
  public Person getPerson(@PathParam("id") Integer id){
    Person p = new Person()
    p.id = id
    p.name = "Bob"
    return p
  }

  @PUT
  @Consumes("application/json")
  @Produces("application/json")
  @Path("person")
  public Person addPerson(Person p){
    println "Received ${p.toString()}"
    p.id = 123
    return p
  }
}

class GsonProvider implements
    MessageBodyWriter<Object>,
    MessageBodyReader<Object>{

    Gson gson = new Gson()

    boolean isWriteable(Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType){
        mediaType.subtype == 'json'
    }

    long getSize(Object t, Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType){
        0
    }

    void writeTo(Object t, Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType,
        MultivaluedMap<String, Object> stringObjectMultivaluedMap,
        OutputStream outputStream)
        throws java.io.IOException, javax.ws.rs.WebApplicationException{
        outputStream.write(gson.toJson(t).bytes)
    }

    Object readFrom(Class<Object> tClass, Type type,
        Annotation[] annotations, MediaType mediaType,
        MultivaluedMap<String, String> stringStringMultivaluedMap,
        InputStream inputStream) {
        gson.fromJson(inputStream.text, tClass)
    }

    boolean isReadable(Class<?> aClass, Type type,
        Annotation[] annotations, MediaType mediaType) {
        mediaType.subtype == 'json'
    }
}


CamelContext cxt = new DefaultCamelContext()
cxt.registry = new SimpleRegistry()
cxt.registry.registry.put("myResources", [new MyExampleResource()])
cxt.registry.registry.put("jsonProvider", new GsonProvider())

cxt.addRoutes(new RouteBuilder(){
  void configure(){
    from("jetty:http://localhost:8080?matchOnUriPrefix=true")
    .to("cxfbean:myResources?providers=#jsonProvider")
  }
})

cxt.start()

Since originally writing this I discovered how to get the JSON parson that comes with CXF to work with Groovy classes. Include the XmlAccessorType annotation telling it to use FIELD.

...
import javax.xml.bind.annotation.XmlRootElement
import javax.xml.bind.annotation.XmlAccessType
import javax.xml.bind.annotation.XmlAccessorType
import org.apache.cxf.jaxrs.provider.JSONProvider
...
@XmlRootElement
@XmlAccessorType(XmlAccessType.FIELD)
class Person {
  Integer id
  String name

  public String toString(){
      return "$id $name"
  }
}
...
cxt.registry.registry.put("jsonProvider", new JSONProvider())
...

Sunday, November 21, 2010

Simple ESB with Camel and Groovy

I'm not sure if writing an ESB service can be considered fun - but when using Groovy and Camel together it comes pretty close. Camel provedes a nice Java based DSL which makes the routing very clean and easy to follow. When you mix in the power of Groovy you get a very elegant solution.

With Groovy you can say a closure implements an interface. This allows us to embed Groovy code inline as a Processor in a Camel route.

    from("jetty:http://localhost:8080/hello")
    .process({Exchange exchange ->
      exchange.out.setBody "Hello World"
    } as Processor)

We can go one step futher by taking advantage of Groovy's dynamic metaClass to create our own method which takes a closure directly. The following adds a new method 'x' to the RouteDefinition class.

RouteDefinition.metaClass.x = {Closure c -> ((RouteDefinition)delegate).process(c as Processor)}

Our route definition now becomes much more elegant

    from("jetty:http://localhost:8080/hello")
    .x {Exchange ex -> ex.out.setBody "Hello"}
    .x {Exchange ex -> ex.out.body = ex.in.body + " World"}

Add in some Grab annotations from Groovy's Grape extension to automatically download the Camel libraries and the whole thing becomes a single, small, self-contained script.

#!/usr/bin/env groovy

@Grab(group="org.apache.camel", module="camel-core", version="2.5.0")
@Grab(group="org.apache.camel", module="camel-jetty", version="2.5.0")
import org.apache.camel.CamelContext
import org.apache.camel.impl.DefaultCamelContext
import org.apache.camel.builder.RouteBuilder
import org.apache.camel.Processor
import org.apache.camel.Exchange
import org.apache.camel.model.RouteDefinition

RouteDefinition.metaClass.x = {Closure c -> ((RouteDefinition)delegate).process(c as Processor)}

CamelContext cxt = new DefaultCamelContext()

cxt.addRoutes(new RouteBuilder(){
  void configure(){
    from("jetty:http://localhost:8080/hello")
    .x {Exchange ex -> ex.out.setBody "Hello"}
    .x {Exchange ex -> ex.out.body = ex.in.body + " World"}
  }
})

cxt.start()