Share on TwitterDigg This

Introduction

Creating your own log analytics
I find loggly, one really cool project. But at it’s current price is kinda prohibitive for many developer teams. So I decided to get something similar done in house for my projects.
After browsing the web, I found some interesting solutions, that when combined, could really get me a great analytic tool.

From batch load to online analysis

Logging analysis have been pretty much a parse and load job for me over the last years. The task was pretty much load a huge Apache/Tomcat/App log, do some aggregation using a plethora of tools (from simple java to hadoop for really huge logs). But the task was pretty much the same, get a log file at the end of the day, run a batch job for a few hours, load into a SQL DB, and rely on DB aggregation and some Data Warehousing for checking the results.
I knew we could do better than that. The first thing that hit me was this great post [ log everything as json make your life easier ] that I found on Hacker News.
That post made me feel really stupid, how something so obvious had skip us for such a long time. So that was the first thing we’ve done. We moved from plain text logging into a json logging.

Changing your log layout into JSON

This could be easy as candy, but it really depends on your environment. Tomcat is a piece of crap when it comes to logging. I can’t count how many times we ran into trouble with Tomcat’s logging system. For application logging you are pretty much safe, as you would have control of it, and by using slf4j you can pretty much control everything. But when it comes to tomcat’s own access log, things are not that simple :(
So I went ahead and changed the AccessLogValve located at ${catalina.home}/conf/server.xml to look something like this:

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="{&quot;host&quot;:&quot;%h&quot;, &quot;user&quot;:&quot;%u&quot;, &quot;method&quot;:&quot;%m&quot;}" />
		 <!-- %h %l %u %t &quot;%r&quot; %s %b -->

This would produce a log on this format:

{
"host" : "localhost",
"user" : "-" ,
"method" : "GET"
}

Well, that’s even easier to read than the regular tomcat log format.

Here come the new tools

But so far we have not changed much, we are still saving the log into a file, and after that I had to bulk load and all the crap. But then I found cube an amazing node.js udp/tcp analytic tool that uses mongodb as its backend. AMAZING! So now we just need to change our log format to follow it’s convention and then send it via UDP to the server, that could be online, and this is the most exciting part. I just needed to change the AccessLog to output the correct format and send it over UDP. Should be easy. It was NOT!

Problems with Access Log Valve

Ok, now I just had to put my log into cube’s format:

{
  "type": "request",
  "time": "2012-04-28T15:28:20.212Z",
  "data": {
    "host": "localhost",
    "user": "-",
    "method": "GET"
  }
}

And this is where I bumped into the first issue with the AccessLogValve. According to the docs, you do have a datetime parameter the %t, but that is not the timestamp format expected by cube. On the javadoc shows an option to customize the date format by using %t{format}, well it did not work. Tomcat ignores that and just put the same format and to make it worst it adds the string format to the generated string. I’m probably doing something wrong here, but did not had the time to investigate, so let’s move on.

Fixing Tomcat access log with a custom log valve

I found a blog with a nice example on how to delegate AccessLog to a log4j, I’m only changing a bit to overcome the problem with the dateformat of tomcat valve. To overcome that I built the JSON string inside the Valve, and added only the data:{} part of the object as the data from the logging, it kinda makes sense right?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package com.fb.tomcat;
 
import java.text.DateFormat;
import java.text.MessageFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
 
import org.apache.catalina.valves.AccessLogValve;
import org.apache.log4j.Logger;
 
public class Log4JAccessLogValve extends AccessLogValve{
 
	private final Logger logger = Logger.getLogger("accessLog");
 
	protected static final String info1 ="com.fbt.tomcat.Log4JAccessLogValve";
	final DateFormat df = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.S'Z'");
 
	@Override
	public void log(String message) {
		String json = "'{'\"type\":\"request\", \"time\":\"{0}\", \"data\": '{' {1} '}' '}'";
		String text = MessageFormat.format(json, new Object[]{df.format(new Date()),message});
		logger.info(text);
	}
 
	@Override
    public String getInfo()
    {
        return info1;
    }
 
    @Override
    protected void open()
    {
    }
}

Well, even without the timestamp bug on the valve, we had the issue to send the logs over UDP instead of saving them into disk. At this point we have a few alternatives in hand:

  1. Modify the AccessLogValve to send the log message over UDP.
  2. Use the syslog Appender and make syslog forward the message to cube (this is what loggly does)
  3. Use an UDPAppender (part of log4j companions)

There’s no free cake folks. All of them come with a cost, We just need to pick one.

  1. In this option you’ll need to deal with possible connection problems, synchronization and low level stuff
  2. The easier one, but you will need to have a syslogd running and configured
  3. The UDPAppender is not part of ANY log4j official package. And the build is broken, you have to face some nasty Maven SHIT to get it working

So let’s go ahead and choose the second implementation.

Log4j configuration

og4j.logger.accessLog=INFO, SYSLOG, file
log4j.appender.file=org.apache.log4j.RollingFileAppender
log4j.appender.file.File=${catalina.home}/logs/access_log4j.log
log4j.appender.file.Append=true
log4j.appender.file.MaxFileSize=20MB
log4j.appender.file.MaxBackupIndex=2
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%m
 
log4j.appender.SYSLOG=org.apache.log4j.net.SyslogAppender
log4j.appender.SYSLOG.SyslogHost=localhost
log4j.appender.SYSLOG.Facility=local2
log4j.appender.SYSLOG.FacilityPrinting=true

XSyslog config

Now you need to configure your syslog to forward to cube own server. I’m putting here the config for my linux rsyslog daemon, please refer to your own syslog tool to get the job done. Just append the following line to your /etc/rsyslog.conf

local2.* @localhost:1180

Summing up

  1. Install mongoDB
  2. Install cube
  3. Copy log4j.jar to your tomcat lib folder, along with the log4j.properties shown here
  4. Copy your compiled custom AccessLog jar into bin folder of tomcat
  5. Modify server.xml to use the new AccessLogValve
  6. Modify rsyslog.conf to forward the logs
  7. Access localhost:1081 and enjoy some real time logging monitoring

Happy coding (and monitoring)

Share on TwitterDigg This

Introduction



I don’t get all the hate that new technology adopters have with existing frameworks. It impresses me the number of scala users out there thrashing everything java.

We all know the limitations of java platform. But you have to face it, most of the codebase today is java based. Most of your co-workers are happy with java, and the worst: so is management.

So going on a bang approach and getting rid of all things java, may not be the best approach if you want to use some new cool language like scala. I took this approach, introducing scala in a Spring based rest app. Spring is well known, developers are used it, your manager probably heard something about it :) and its a very stable and tested framework. So you are not adding some 0.0.2 version of a new REST service complete based on scala :)

Enough words, let’s get our hands dirty :)


Setup


If you followed my last post, there’s an example on how to create a HATEOAS based app using Spring there. Here we have lower expectation. All we want is to setup a basic SpringMVC Rest project using Scala.

All the setup is pretty much the same, except for the fact we will be using scala classes instead of java classes here. And a very important step: Jackson will not work as a JSON serializer. You’ll have to use Jerkson


So, to make things simpler here, I’ll just paste all the relevant code bellow, any question, just go to my previous post explaining the role of each of the classes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
class WebAppInitializer extends WebApplicationInitializer{
 
	def onStartup(container: ServletContext) : Unit = {
	  val rootContext = new AnnotationConfigWebApplicationContext()
	  rootContext.register(classOf[AppConfig])
	  container.addListener(new ContextLoaderListener(rootContext));
	  val dispatcherContext  = new AnnotationConfigWebApplicationContext()
	  dispatcherContext.register(classOf[DispatcherConfig])
	  val dispatcher = container.addServlet("dispatcher", new DispatcherServlet(dispatcherContext))
	  dispatcher.setLoadOnStartup(1)
	  dispatcher.addMapping("/")
	}
}
 
@Configuration
@EnableWebMvc
@ComponentScan(basePackages=Array("com.fb.mcare.controllers"))
class DispatcherConfig extends WebMvcConfigurerAdapter {
 
	@Bean
	def handleMapping() : DefaultAnnotationHandlerMapping  = {
		new DefaultAnnotationHandlerMapping
	}
 
	@Override
	override def configureMessageConverters(converters: java.util.List[HttpMessageConverter[_]]) {
		val converter = new MappingJerksonHttpMessageConverter(MediaType.APPLICATION_JSON)
		converters.add(converter)
	}
 
}
 
 
@Controller
@Autowired
@RequestMapping(Array("/patients/*"))
class PatientController(val service:PatientService) {
 
  def this() = {
    this(null)
  }
 
  @RequestMapping(produces=Array("application/json"), method=Array(RequestMethod.GET), value=Array("/{id}"))
  def list(@PathVariable id : String) : ResponseEntity[Person] = {
    println(id)
	val p = new Person("vinicius",32)
	val entity = new ResponseEntity[Person](p,HttpStatus.OK)
	return entity
  }
 
}

Ok so things start to get interesting here on line 25, were we actually setup our JerksonMapping.

Lines 34-51 represent our controller, note that we are using regular Spring annotations, and being a good scala class, we inject the PatientService in our controller, making it a immutable val :)

Note, that you can do pretty much everything you’d do with a regular spring controller, but this time you have the power of scala at your hands.

The most important part is the MappingJerksonMessageConverter. Using this converter instead of Jackson, makes possible the serialization/deserialization of Scala classes into Json:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
import scala.reflect.Manifest
import org.springframework.http.converter.HttpMessageConverter
import org.springframework.http.MediaType
import org.springframework.http.HttpInputMessage
import scala.collection.JavaConversions
import com.codahale.jerkson.Json
import java.io.IOException
import org.springframework.http.converter.HttpMessageNotReadableException
import org.springframework.http.HttpOutputMessage
import org.springframework.http.converter.HttpMessageNotWritableException
 
class MappingJerksonHttpMessageConverter(val supportedMediaType: MediaType) extends HttpMessageConverter[Any] {
 
 
  override def canRead(clazz : Class[_], mediaType: MediaType) : Boolean = {
 
    Json.canDeserialize(Manifest.classType(clazz))
  }
 
  override def canWrite(clazz : Class[_], mediaType: MediaType) : Boolean = {
    Json.canSerialize(Manifest.classType(clazz));
  }
 
  override def getSupportedMediaTypes() : java.util.List[MediaType] = {
    JavaConversions.asJavaList(List(supportedMediaType))
  }
 
  @throws(classOf[IOException])
  @throws(classOf[HttpMessageNotReadableException])
  override def read(clazz : Class[_], inputMessage : HttpInputMessage) : AnyRef ={
    Json.parse(inputMessage.getBody())
  } 
 
  @throws(classOf[IOException])
  @throws(classOf[HttpMessageNotWritableException])
  override def write(contentType : AnyRef, mediaType: MediaType, outputMessage: HttpOutputMessage) : Unit ={
    Json.generate(contentType,outputMessage.getBody())
  }
}

And that’s it :) You can now start using Spring with your scala project, and maybe even fool your manager (should not be a hard task anyways) that you are using Spring (but with the scala language power under the hood).

Happy coding!

Share on TwitterDigg This

Well, I know REST is an old topic, you find thousands of pages talking about it, so why another one?

Well, this post is about using Spring 3.1, some of it’s new features to accomplish at least part of the Hypermedia As The Engine Of Application State Architectural style. If you follow me till the end of this post, you’ll learn:

  • How to setup a Spring 3.1 project for REST endpoints without using 1 single XML file
  • Use SpringData mongodb as the persistence engine
  • Use Spring new Caching Feature for the caching of ETags
  • Use Spring AOP to add specific Headers to the Response of your REST endpoint
  • Some of my thoughts on REST in real world :)

I’ve being developing REST services for a while now, I would guess that my first REST endpoint dates back to 2008. But it occurs to me, that most of them (if not all) where not truly REST services as it is defined by . I was pretty much just sending JSON/XML over HTTP using the support for HTTP methods. That’s what most people is doing, and it’s what Jim Webber et al, call as Level two services on their great book Rest in practice (If you want to read one book on REST, that’s the one).

1 – Configuring your environment

I’m still playing with this project, this post contains only parts of it, please feel free to find the code at . So let’s start. The first thing you will need to use Spring 3.1 without any XML file, is to have a servlet 3.0 engine and use the WebApplicationInitializer class from Spring. What this class does is to bootstrap you spring context and servlet (for the Spring MVC).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
	public void onStartup(ServletContext container)
			throws ServletException {
		AnnotationConfigWebApplicationContext rootContext = new AnnotationConfigWebApplicationContext();
		rootContext.register(AppConfig.class);
 
		// Manage the lifecycle of the root application context
		container.addListener(new ContextLoaderListener(rootContext));
 
		// Create the dispatcher servlet's Spring application context
		AnnotationConfigWebApplicationContext dispatcherContext = new AnnotationConfigWebApplicationContext();
		dispatcherContext.register(DispatcherConfig.class);
 
		// Register and map the dispatcher servlet
		ServletRegistration.Dynamic dispatcher = container.addServlet(
				"dispatcher", new DispatcherServlet(dispatcherContext));
		dispatcher.setLoadOnStartup(1);
		dispatcher.addMapping("/");
 
	}

This class loads our appContext which is defined in a class called AppConfig, creates a secondary context for our SpringMVC dispatcher Servlet, and register that servlet within the container. The could is pretty self explanatory, I don’t think I need to go any further.

Next the two configuration classes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
@Configuration
@ComponentScan(basePackages={"com.fb.restbucks.services"},excludeFilters={@Filter(Configuration.class)})
@EnableCaching
public class AppConfig {
 
	@Bean
	public MongoDbFactory mongoDbFactory() throws Exception {
		return new SimpleMongoDbFactory(new Mongo(), "restbucks");
	}
 
	@Bean
	public MongoTemplate mongoTemplate() throws Exception {
		MongoTemplate template = new MongoTemplate(mongoDbFactory());
		template.setWriteConcern(WriteConcern.SAFE);
		return template;
	}
 
	@Bean
	public CacheManager cacheManager() throws Exception {
		SimpleCacheManager simpleCache = new SimpleCacheManager();
		simpleCache.setCaches(Arrays.asList(new ConcurrentMapCache("tags")));
		return simpleCache;
	}
 
@Configuration
@EnableWebMvc
@ComponentScan(basePackages={"com.fb.restbucks.controllers","com.fb.restbucks.aspects"},excludeFilters={@Filter(Configuration.class)})
@EnableAspectJAutoProxy(proxyTargetClass=true)
public class DispatcherConfig extends WebMvcConfigurerAdapter {
 
	@Override
	public void configureMessageConverters(
			List<HttpMessageConverter<?>> converters) {
		MappingJacksonHttpMessageConverter converter = new MappingJacksonHttpMessageConverter();
		List<MediaType> mediatypes = new ArrayList<MediaType>();
		mediatypes.add(MediaType.APPLICATION_JSON);
		converter.setSupportedMediaTypes(mediatypes);
		converters.add(converter);
	}
 
 
	@Bean
	public DefaultAnnotationHandlerMapping handlerMapping(){
		return new DefaultAnnotationHandlerMapping();
	}
 
 
}



The @ComponentScan annotation will inform spring to scan the classpath for classes annotated with @Component/@Service/@Controller/@Repository and add them as Spring beans, you can filter by packages and add some exclusions. This is the same of declaring a bean in a spring xml file.

On our first class AppConfig besides the mongoDB configuration, the interesting part is the configuration of a cachemanager. Spring’s new caching abstraction makes it really simple to have one umbrella interface to access different cache providers (Ehcache and Gemfire for instance). The @EnableCaching annotation will allow Spring’s post processors to look for some annotations on our code and add caching support. More on this later on…

The second class configures the SpringMVC framework. A few important things to notice here:

  1. This project uses Aspects, and as the aspect is weaving some controllers methods, it was required to have the @EnableAspectJAutoProxy declared in this configuration and not on AppConfig
  2. WebMvcConfigurerAdapter class is an Spring utility class that allows us to customize the SpringMVC framework, here we are setting two important features: the message converters to use Jackson and JSON and the DefaultAnnotationHandlerMapping. I found that without registering the DefaultAnnotationMapping, the controller’s mapping were not working.

That’s all you need to have your project configured. Now let’s move to some Convetions and finally some code.

2 – Conventions

Some conventions I’ve used in this post. First of all, I’m not implementing the full HATEOAS style, as I said this is a work in progress, just decided to share. For one thing I did not implement the MediaType part of HATEOAS, I just let it loose using APPLICATION/JSON as the format.

But I did follow this convention (and adapted some stuff as well).

2.1 How I’ve mapped HTTP Verbs

  • GET: Returns a Resource, or a collection of resources, always returns the latest Etag (more on this later) header. An example of a response is shown bellow

  • Response:
    status: 200 OK
    Date: Mon, 05 Dec 2011 19:07:52 GMT
    Transfer-Encoding: chunked
    Server: Apache-Coyote/1.1
    Etag: c0ca00e0665925adfcf5d5f846eac29c
    Content-Type: application/json
    {“entity”:{“id”:”4ed8ee2b67f86842b1edac2c”,”status”:”PENDING”,”items”:[{"milk":"SKIM","name":"mocha","quantity":1,"size":"LARGE"}]},”links”:[]}


  • HEAD: Same as get, but without a body, only returns the Etag of the resource

  • POST: Adds a new Resource, returns a HTTP 201 (CREATED) and the resource location in the header

  • Response:
    status: 201 Created
    Date: Mon, 07 Dec 2011 19:33:56 GMT
    Content-Length: 0
    Server: Apache-Coyote/1.1
    Location: http://localhost:8080/restbucks/orders/4eddfda485cdbad7de0dc685


  • PUT: Updates a resource. Validates the If-Match header (Etag) and if client has the lattest version updates and returns a 204, if client has an obsolete version, returns a PRE_CONDITION_FAILURE

  • DELETE: Same as put, same validation happens.

  • OPTIONS: Returns all the links for the resource


2.1.1 A special note on Etag



Etags are used mostly to allow content caching, but I really decided to use it for something else. If you ever used Hibernate in production you know that Optimistic locking is the best way to combine integrity and performance. Think on Etag as the same as the locking on Hibernate. By providing an identifier to your client for every GET/HEAD operation on a resource, you can guarantee0 that any modification operation such as DELETE/PUT will require the client to send the etag for the entity. If you stick to this, it’s pretty much guarantee (note I said pretty much not 100% for sure) that you will avoid concurrent modification scenarios.

I found this to be the nicer way to deal with versioning entities so far, all you have to do is guarantee that your client must send the Etag on every PUT/DELETE request, and that you will return an Etag on every GET/HEAD operation.

2.1.2 Mapping your domain to resources


Over the years I’ve used JAX-RS and either CXF or RestEasy to handle my RESTful webservices. Well, as pretty much everything JEE, traditional JAX-RS services are just plain WRONG. I know this will raise the fury on some JEE fans out there, and I’m sure there will be someone posting on how one could use JAX-RS properly, but JAX-RS promotes the usage of direct mapping from your domain to a Resouce, this is wrong. And I’ll tell you why: You are dealing with a WEB (HTTP ppl, Headers, links, media type) endpoint here. For most part your resource will have “protocol” stuff that you should not care on your resource. Spring’s ResponseEntity class is the perfect fit to customize the return method of your endpoint. It allows easy manipulation of HTTP Headers, and Status (Again, someone will shout in fury here, that same could be done on JAX-RS, my point is: It does not promote it, like most JEE stuff it leads developers to use the spec on a wrong way).


I found that ResponseEntity gives you a good degree of freedom, and if you want to keep your architecture clean, your domain services should return domain entities only, and your endpoint (controllers) are responsible to map this to a ResponseEntity. Accept the fact that you have a service that deals with HTTP protocol, hiding too much complexity sometimes can back fire at you.

But even with Spring ResponseEntity, I found that another layer of indirection was required to represent my domain classes as resources. For instance, a true HATEOAS RESTful service promotes links as first class citizens, so I’ve decided to add a Link class, and a Resource class to wrap my Domain classes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
 
public class Link {
 
	private String rel;
	private String href;
	private HttpMethod method;
 
	public String getRel() {
		return rel;
	}
	public void setRel(String rel) {
		this.rel = rel;
	}
	public String getHref() {
		return href;
	}
	public void setHref(String href) {
		this.href = href;
	}
	public HttpMethod getMethod() {
		return method;
	}
	public void setMethod(HttpMethod method) {
		this.method = method;
	}
 
}
 
public class Resource<T> {
 
	private T entity;
	private List<Link> links;
 
	public Resource(T entity){
		this.entity = entity;
		this.links  = new LinkedList<Link>();
	}
 
	public T getEntity() {
		return entity;
	}
 
	public void setEntity(T entity) {
		this.entity = entity;
	}
 
	public void addLink(Link link){
		this.links.add(link);
	}
 
	public List<Link> getLinks() {
		return links;
	}
 
	public void setLinks(List<Link> links) {
		this.links = links;
	}
 
}



Using this Resource class, you can now encapsulate your domain object inside it, and add the links that a resource may have. Our final controller method would look something like this:

public ResponseEntitity<Resource<Order>> getOrder(String id){}

Now ResponseEntity wraps all the protocol related information (Headers + Body + Status) and Resource wraps the domain object + links.


3 – Implementation


Now onto the final piece which is the implementation itself. To reduce the amount of code, I’m only going to show the Controllers and relevant classes here, hiding some services and domain objects, you can find them on the git repository anyway.

3.1 Controllers


Easy part, RESTful webservices are no different than a regular controller on SpringMVC, maybe besides the fact of you probably won’t return a ModelAndView but an entity. As with most things in Spring, you are free to choose the way to implement them. Here’s my implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
public abstract class BaseController<T> {
 
 
	@Autowired
	protected ETagService eTagService;
 
	@SuppressWarnings("rawtypes")
	@ExceptionHandler(EntityNotFoundException.class)
	protected ResponseEntity entityNotFound(){
		return new ResponseEntity(HttpStatus.NOT_FOUND);
	}
 
	@RequestMapping(consumes="application/json", produces="application/json", method=RequestMethod.HEAD, value="/{id}")
	public ResponseEntity head(HttpServletRequest request){
		String url = ServletUriComponentsBuilder.fromRequest(request).build().toString();
		try {
			String tag = eTagService.get(url);
			HttpHeaders headers = new HttpHeaders();
			headers.add("Etag", tag);
			return new ResponseEntity(null, headers, HttpStatus.NO_CONTENT);
		} catch (InvalidTagException e) {
			return new ResponseEntity(HttpStatus.NOT_FOUND);
		}
	}
 
	@RequestMapping(consumes="application/json", produces="application/json", method=RequestMethod.OPTIONS, value="/")
	public abstract ResponseEntity<T> describe(HttpServletRequest request);
}



Remember when I’ve said on the conventions that I want all HEAD methods only need to return the latest version of the Etag? Well, that’s why I’ve put a method on a base class, so all my controllers would benefit from it. Note that I use an EtagService (will explain latter) but basically all I do is get the tag for the url and return it. Also, all resources should implement an Options Method, this method should describe all the links for a given resource. Each implementation should fill the links, as this is coupled to the business requirements. And last, there’s an Exception Mapper here. SpringMVC allows you to map Exception handlers, in my case I’ve decided to create an Exception named EntityNotFoundException (yeah common name I know), and my data layer throws this whenever an entity is not found on the repository. I allow this Exception to leak and Spring handles this, as a resource not found should map to a HTTP 404 response.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
@Controller
@RequestMapping("/orders/*")
public class OrderController extends BaseController<Resource<Order>>{
 
	@Autowired
	private OrderService orderService;
 
	@RequestMapping(consumes="application/json", produces="application/json", method=RequestMethod.POST, value="/")
	public ResponseEntity<Resource<Order>> dynamicTest(@RequestBody Order order){
		orderService.create(order);
		Resource<Order> resource = new Resource<Order>(order);
		return new ResponseEntity<Resource<Order>>(resource,HttpStatus.CREATED);
	}
 
	@RequestMapping(consumes="application/json", produces="application/json", method=RequestMethod.GET, value="/{id}")
	public ResponseEntity<Resource<Order>> getOrder(@PathVariable("id") String id){
		Order order = orderService.find(id);
		Resource<Order> resource = new Resource<Order>(order);
		return new ResponseEntity<Resource<Order>>(resource,HttpStatus.OK);
	}
 
	@RequestMapping(consumes="application/json", produces="application/json", method=RequestMethod.PUT, value="/{id}")
	public ResponseEntity<Resource<Order>> update(@RequestBody Order order){
		order = orderService.update(order);
		Resource<Order> resource = new Resource<Order>(order);
		return new ResponseEntity<Resource<Order>>(resource,HttpStatus.NO_CONTENT);
	}
 
	@Override
	public ResponseEntity<Resource<Order>> describe(HttpServletRequest request) {
		//TODO describe this resource links
		return null;
	}
 
}


@RequestMapping is how SpringMVC will handle your requests (douh). You can define the media contract (accept and content-type) by setting consumes and produces values. The value is the relative path of your url, and you can use identifiers {id} for instance, pretty much like on JAX-RS. You also define the method to handle (GET,POST,PUT).

As explained earlier I return a ResponseEntity> as the representation of my resource, and besides that is just invocation on the OrderService. This is everything for the controllers, besides some conventions there’s absolutely nothing different than any other SpringMVC REST application out there. But it’s now that things should get at least different from most applications.

3.1.1 EtagService

I’ve already gave an explanation on what an ETag means to me, you can find a general explanation here. The ETagService, constructs given an url and the entity associated. The tag is then persisted into MongoDB on a collection {url:tag}. This service also relies on Spring amazing caching abstraction. So let me explain it’s interface and caching abstraction before diving into implementation.

1
2
3
4
5
6
7
8
9
10
public interface ETagService {
	@Cacheable(value="tags", key="#url")
        public String generate(String url, Object entity);
        @Cacheable(value="tags", key="#url")
	public String get(String url);
	@CacheEvict(value="tags",key="#url")
	public void remove(String url);
        @CacheEvict(value="tags",key="#url")
	public String update(String url, Object entity);
}

Line 2-5: Those methods generate a tag or get a tag given an url. By using @Cacheable we are instructing spring to cache the results (the tag) of this method invocation using the provided url parameter as the key.
Lines 6-9: Both update and remove methods need to evict an entry from the cache, so the next time a tag is requested, the generate or get method will gets called. The eviction only happens on the cache item determined by the url key.

The actual implementation of the service is shown bellow, there are probably 1.000 ways of doing the ETag generation better than this one (hashcodes for example) but for sake of simplicity this will suffice.

@Service
public class MongoETagService implements ETagService {
 
	@Autowired
	private MongoOperations mongoOps;
 
	private ObjectMapper mapper = new ObjectMapper();
 
	public String generate(String url, Object entity) {
		String tag = createTag(entity);
		BasicDBObject dbo = new BasicDBObject().append("url", url).append("tag", tag);
		mongoOps.getCollection("etags").save(dbo);
		return tag;
	}
 
	public void remove(String url) {
		BasicDBObject dbo = new BasicDBObject().append("url", url);
		mongoOps.getCollection("etags").remove(dbo);
	}
 
	public String update(String url, Object entity) {
		String tag = createTag(entity);
		BasicDBObject dbo = new BasicDBObject().append("url", url);
		mongoOps.getCollection("etags").update(dbo, new BasicDBObject().append("$set", new BasicDBObject().append("tag", tag)));
		return tag;
	}
 
 
	public String get(String url) {
		BasicDBObject dbo = new BasicDBObject().append("url", url);
		List<DBObject> results = mongoOps.getCollection("etags").find(dbo).toArray();
		if(results.size() == 0 ){
			throw new InvalidTagException("No tag found for url: " + url);
		}
		return String.valueOf(results.get(0).get("tag"));
	}
 
 
	private String createTag(Object entity){
		StringWriter writer = new StringWriter();
		try {
			mapper.writeValue(writer, entity);
		} catch (JsonGenerationException e) {
			e.printStackTrace();
		} catch (JsonMappingException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}
		return DigestUtils.md5DigestAsHex(writer.toString().getBytes());
	}
 
 
 
}


3.1.2 Implementing Conventions using AOP

Well, for the last part. On section 2 I’ve talked about some conventions to rule the general state machine navigation right. Things like checking for an Etag header, adding an Etag header, adding a location. Those conventions maps to HTTP Methods, so having an Interceptor to modify the response seemed to me the best alternative. Again, many ways to skin a cat, I could use a ServletFilter, a SpringMVC Interceptor, but I decided to use an Aspect for that.
The following aspect will change the response to add the etags, location or check etags, and thus leaving the implementation of the controllers away from this plumbing code.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
 
@Component
@Aspect
public class ResponseEnhancerAspect {
 
 
	@Autowired
	private ETagService eTagService;
 
	@SuppressWarnings({ "rawtypes", "unchecked" })
	private Object handlePOST(ProceedingJoinPoint pjp) throws Throwable{
		HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.getRequestAttributes()).getRequest();
		Object retVal = pjp.proceed();
		ResponseEntity entity = (ResponseEntity)retVal;
		HttpHeaders headers = new HttpHeaders();
		Resource resource = (Resource) entity.getBody();
		Object id = ReflectionUtils.findMethod(resource.getEntity().getClass(), "getId").invoke(resource.getEntity());
		headers.add("Location", ServletUriComponentsBuilder.fromRequest(request).path("{id}").build().expand(id).toString());
		return new ResponseEntity(null,headers,entity.getStatusCode());
	}
 
 
	@SuppressWarnings({ "rawtypes", "unchecked" })
	private Object handleGET(ProceedingJoinPoint pjp) throws Throwable {
		HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.getRequestAttributes()).getRequest();
		Object retVal = pjp.proceed();
		ResponseEntity entity = (ResponseEntity)retVal;
		HttpHeaders headers = new HttpHeaders();
		String url = ServletUriComponentsBuilder.fromRequest(request).build().toString();
		String tag = null;
		try{
			tag = eTagService.get(url);
		}catch (InvalidTagException e) {
			tag = eTagService.generate(url, entity.getBody());
		}
		headers.add("Etag", tag);
		return new ResponseEntity(entity.getBody(),headers,entity.getStatusCode());
	}
 
	@SuppressWarnings({ "rawtypes", "unchecked" })
	private Object handlePUT(ProceedingJoinPoint pjp) throws Throwable {
		HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.getRequestAttributes()).getRequest();
		ResponseEntity responseEntity = (ResponseEntity) pjp.proceed();
		String url = ServletUriComponentsBuilder.fromRequest(request).build().toString();
		HttpHeaders headers = new HttpHeaders();
		Resource resource = (Resource)responseEntity.getBody();
		String newTag = eTagService.update(url, resource.getEntity());
		headers.add("Etag", newTag);
		return new ResponseEntity(responseEntity.getBody(),headers,responseEntity.getStatusCode());
	}
 
	private boolean checkEtag() {
		HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.getRequestAttributes()).getRequest();
		String url = ServletUriComponentsBuilder.fromRequest(request).build().toString();
		String storedTag = null;
		try {
			storedTag = eTagService.get(url);
			String providedTag = request.getHeader("If-Match");
			if(providedTag == null || !storedTag.equals(providedTag)){
				return false;
			}else{
				return true;
			}
		} catch (Exception e) {
			return false;
		}
	}
 
 
	private Object handleDELETE(ProceedingJoinPoint pjp) throws Throwable{
		HttpServletRequest request = ((ServletRequestAttributes) RequestContextHolder.getRequestAttributes()).getRequest();
		String url = ServletUriComponentsBuilder.fromRequest(request).build().toString();
		pjp.proceed();
		eTagService.remove(url);
		return new ResponseEntity(HttpStatus.NO_CONTENT);
	}
 
	@Around("execution(public org.springframework.http.ResponseEntity com.fb.restbucks.controllers.*.*(..)) && @annotation(requestMapping)")
	@SuppressWarnings({ "rawtypes", "unchecked" })
	public Object handleMethod(ProceedingJoinPoint pjp, RequestMapping requestMapping) throws Throwable{
		Object retVal = null;
		switch (requestMapping.method()[0]) {
		case GET :
			retVal = handleGET(pjp);
		break;
 
		case POST:
			retVal = handlePOST(pjp);
		break;	
 
		case PUT:
			if(checkEtag()){
				retVal = handlePUT(pjp);
			}else{
				retVal = new ResponseEntity(HttpStatus.PRECONDITION_FAILED);
			}
		break;
 
		case DELETE:
			if(checkEtag()){
				retVal =  handleDELETE(pjp);
			}else{
				retVal = new ResponseEntity(HttpStatus.PRECONDITION_FAILED);
			}
		break;
 
 
		}
		return retVal;
	}



Only a comment here. At first I wanted to have one advice per method, but since Spring does not uses annotations like JAX-RS where you have one annotation per method (@GET, @POST), I could not create the pointcut for the aspect based on a parameter. I tried, asked on forums, but nothing, if you know how to do it, please send me the answer, I’d gladly update this post.
This class has one method per HTTP verb, plus a checkEtag that should occurs before PUT/DELETE methods. All it does is to implement the rules defined at section 2 of this post.


Conclusion

I really enjoyed using SpringMVC and Spring new features to implement a RESTful service. I don’t want to compare apples and oranges and saying that SpringMVC would be a better alternative to others like Resteasy or RESTfulie, it all depends on your project architecture, and your team skills.
My intention with this post was just to show a quick way to get a Spring 3.1 REST based project, and add some of my takes on how to build a (at least close to) HATEOAS implementation of a RESTful service without too much pain.

Happy Coding