Jackson annotations with MongoDB

Posted on by

At VZ we are currently busy testing a new storage backend for the VZ feed.  We’ve pushed the existing backend to its limits and while so far it has served us well, we are finding that for what we want to do going forward, it just isn’t the right match for our requirements.  So we’ve spent some time investigating what the best backend will be, and we are now in the testing stage with MongoDB, loading it with our feed data and hammering it with load tests.  So far so good.

The backend of our feed service is implemented in Java, and talks JSON back to clients.  If you log in to VZ and use your favourite browsers developer console to see XHR traffic, you can see the JSON that the feed service returns.  This JSON is generated from POJOs using Jackson, a very simple yet powerful, performant and flexible framework for serialising POJOs to JSON.

Looking carefully at the JSON, you’ll notice that the feed data contains more than just a list of messages, there is a mixture of objects whose data is clearly generated on the fly by the feed service, and other objects that the feed service doesn’t really care about, it just receives them from the services that generate them, and passes them to the client as is.  For example, information about a particular photo, or a gadget, or a status update, or a new friend.  This data is represented by many different POJOs in the feed, and in some cases specific processing is done on them, but mostly the feed receives them from the various services that generates them, and passes them back to the client untouched.

All these POJOs have Jackson annotations on them, and we know that they work well with Jackson.  With our old storage backend, we simply serialised them to JSON to store them.  One of the reasons for going with MongoDB is that we wanted to be able to easily query our data, to get statistics and a better understanding of how the service was being used.  What we wanted to avoid though was having to either rewrite the POJOs, or make MongoDB equivalent copies of them, in order to store them in MongoDB.  What we really wanted to do is reuse the Jackson annotations on them so that we could store them as is in MongoDB.

A google search revealed that there already was a mapper out there by Grzegorz Godlewski that did this, however this mapper had a few problems in our eyes.  For one, it required a fork of the Mongo Java Driver, and there was no indication that this fork would ever be pulled back into the stable driver.  It also said that it was experimental and not production ready, and we weren’t about to trust our feed with an experimental technology that didn’t have a clear future.  It is worth saying though that Grzegorz’s mapper is quite innovative, it serialises/deserialises objects directly to/from the BSON that MongoDB speaks with the client.  This would make it the most performant mapper out there.  But we didn’t feel that the technology was mature enough for our use case.

However, as Grzegorz pointed out, plugging in a custom parser/generator to Jackson is a simple thing to do, so we decided to implement our own mapper that parsed/generated the MongoDB map like DBObject‘s.  In addition to this, we implemented a very lightweight interface that wraps the MongoDB DBCollection, called JacksonDBCollection.  This provides all the same methods that DBCollection provides, except that where appropriate, it replaces DBObject method parameters and return types with strongly typed POJO versions.  DBObject‘s still get used for querying, because you can’t express all queries using POJOs, however when you just want to use equality in your queries, you can use your POJO as the query.

It’s open source!

We’re pleased to announce that we’ve made this library available as an open source library. You can download the source code, and contribute to it yourself, at GitHub.  If you don’t want the source code, but just want to start using it in your project now, you can get it from the central maven repository:

<dependency>
    <groupId>net.vz.mongodb.jackson</groupId>
    <artifactId>mongo-jackson-mapper</artifactId>
    <version>1.0</version>
</dependency>

The details

To give you a taste of what this framework looks like, I’ll show some coding examples.  Here is how to create a JacksonDBCollection:

JacksonDBCollection<MyPojo, String> coll = JacksonDBCollection.wrap(
    dbCollection, MyPojo.class, String.class);

The two type parameters are the type of the POJO that is being mapped, and the type of the ID of the POJO.  The reason we require the type of the ID of the POJO is for strongly typed querying by ID, and strongly typed retrieval of generated IDs.  Here is what my POJO looks like:

public class MyPojo {
  @ObjectId @Id public String id;
  public Integer someNumber;
  public List<String> someList;
}

Maybe not the best Java coding style with public mutable fields, Jackson is flexible enough to support almost anything, this keeps it simple for this blog post.  In the feed, we’re using immutable objects with private final fields, and @JsonCreator annotated constructors.

One issue that we encountered while implementing this is how to handle ObjectId‘s.  If you declare an object to have a type of ObjectId, it works without a problem.  But this is a POJO mapper, and some people may not want to have an ObjectId type in their POJO, particularly if they are reusing the POJO for the web.  So we added an annotation, @ObjectId, that can be put on any String or byte[] property, that tells the mapper to serialise/deserialise this property to/from ObjectId.

You’ll also see the use of the @javax.persistence.Id annotation.  This is really just a short hand that we implemented for @JsonProperty("_id"), which has the added advantage of not having to use the Jackson views feature if you want to use the same POJO for the web and don’t want the name of the property to be _id. You could just as easily make the field name _id with no extra annotations.

Inserting an object is simple:

MyPojo pojo = new MyPojo();
pojo.someNumber = 10;
pojo.someList = Arrays.asList("foo", "bar");
WriteResult<MyPojo, String> result = coll.insert(pojo);

or with write concerns:

WriteResult<MyPojo, String> result = coll.insert(pojo, WriteConcern.MAJORITY);

We are letting MongoDB generate the ID for us here, which begs an important question, how do we find out what the ID it generated was? With the MongoDB Java Mapper, it sets the ID on the object you passed in. This is not practical with Jackson, because you may be using a custom serialiser, or @Creator annotated factory method/constructor, in the case of the VZ feed our objects were immutable, making it impossible to do that. We supported this by implementing our own WriteResult class, which aside from wrapping all the methods from the MongoDB Java Driver WriteResult, provides a few methods such as getSavedId() and getSavedObject(), which deserialises the id/object so you may obtain the ID:

String id = result.getSavedId();

Now using that ID we can load the object again:

MyPojo saved = coll.findOneById(id);

Equality based querying can be done using the POJO as a template:

MyPojo query = new MyPojo();
query.someNumber = 10;
for (MyPojo item: coll.find(query)) {
    System.out.println(item.id);
}

And loading partial objects can also be done using the POJO as a template, by setting any fields you want loaded to be something that isn’t null:

MyPojo query = new MyPojo();
query.someNumber = 10;
MyPojo template = new MyPojo();
template.someNumber = 1;
template.id = "not null"
for (MyPojo item: coll.find(query, template)) {
    System.out.println(item.id);
    assert(item.someList == null);
}

And when this isn’t enough, you can fall back to normal DBObject‘s for both the query and the template:

for (MyPop item: coll.find(new DBObject().add("someNumber", new DBObject().add("$gt", 5))) {
    System.out.println(item.id);
}

There are more examples for some more advanced use cases, for example using custom views, on the GitHub project page.

Summary

So here we have a new POJO mapper for MongoDB that you can use, and due to the fact that it uses Jackson, it has a head start in being very powerful, flexible and performant. Some of our plans for future features include:

  • @Reference annotation support, loading dehydrated referenced objects containing just the ID, with convenience methods for hydrating these objects
  • Schema migration features, where the JacksonDBCollection can return option tuples containing either the old or the new object type based on what was detected

If you have any other features you’d like to see, please raise an issue in the GitHub project!

10 thoughts on “Jackson annotations with MongoDB

  1. Awesome, James! Love that you guys decided to open source it.

    However, your examples would be much easier to follow if you used some real world objects instead of generic MyPojo’s with foo and bar properties.

    You know like:

    public class BlogPost {
    @ObjectId @Id String id;
    String author;
    List tags;
    }

  2. Hi guys. Thanks for good words about my forked driver. I’ve been using it in one project for few months and it’s working quite good.
    I’m going to contact 10gen and try to pull it together into main project. I hope they will help me and we’ll get good suport for Jackson mapping out of box.

  3. Hi Grzegorz, what I’d really like to see is a pluggable mechanism for handling BSON in the driver. This would help more than just mapping with Jackson, other frameworks could take advantage of it too to do their own mappings.

  4. Good job James. This looks really good. Good to see you’re pushing the feed at vz even further.

  5. I usually do not leave a response, however after looking at a few of the comments on
    this page Jackson annotations with MongoDB | developer.
    vz.net. I actually do have a few questions for you if it’s allright. Is it simply me or does it look like some of these responses come across as if they are written by brain dead individuals? :-P And, if you are posting on other sites, I would like to keep up with everything fresh you have to post. Would you list of every one of all your social pages like your twitter feed, Facebook page or linkedin profile?

  6. This is really attention-grabbing, You are a very
    skilled blogger. I’ve joined your feed and look forward to
    in the hunt for more of your magnificent post.
    Also, I have shared your site in my social networks

  7. Most of these refinancing options are gained through commercial lending institutions Bret Brightly if you can find any mistakes it really is
    imperative that you just get hold of your bank immediately and also the
    bureaus to rectify the mistakes.

  8. Hey there Your entire web page starts up really slow for
    my situation, I don’t know who’s issue is that however facebook starts up pretty immediate.
    However thanks for posting awesome articles. Almost everyone who discovered this site must have found
    this short article literally beneficial. I
    hope I will be able to find a lot more remarkable stuff and I also should really flatter your site simply by stating you have carried out
    awesome writing. Just after checking out your article,
    I have bookmarked your web page.

Leave a Reply

Your email address will not be published. Required fields are marked *


*