ElasticSearch With Grails and Mongo

Setting Up ElasticSearch and Best Practices:

Let’s talk about elastic search based upon Lucene. There’s already wealth of information about elastic search in various blogs but when I was trying to incorporate it into our Grails + Mongo application, I’d to navigate through uncharted territory. Since, I care so much about you guys, I don’t want you to wander into that dark path. So, here’s all you need to know.  

We going to talk about the 4 pillars for setting up elastic search on grails server with mongoDB:

  1. interactive connectionElasticSearch
  2. html cleanerMongo River
  3. Word to htmlGrails Elastic Search Plugin
  4. replace textElastic Search Queries

 

Elastic Search:

Despite grails being a great framework, it hasn’t reached the elastic search community. This might be the reason why elastic search doesn’t has a simple solution for grails.

So, what is elastic search? Simple, elastic search is  multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents, yada yada… You would’ve probably googled a bit and already know this and other stuff. To be honest, I don’t really care about the nitty-gritty of this because we’ve a plugin to take care, most of it.

 

Mongo River:

TL; DR: Mongo river was one of the most popular solution for using elastic search with mongo database but we’ve a much better solution in form of grails plugin.

Mongo River is/was a cool plugin that uses mongoDB as datasource to store data in elastic search. Running a query is as simple as:

JSON result = HTTPBuilderService.performRestRequest("http://localhost:9200/_search",

   "GET", [q: "Jeans"])

Whoa, look how elegant it is, you just need to ran one HTTP query and you’re done. And that’s not the only good thing about it.

Good Parts:

  • Production Tested: There are hundreds of rivers in production. Some large companies like Mashable are using the river
  • No extra grails plugin required: A simple HTTP GET request is sufficient
  • Normal Mongo Document Implementation: mongo document is stored within elastic search without transformation. The new document stored in elastic search will use the same id as mongoDB

Too good to be true, right? No wait, before you run that apt-get command there already some disturbance on the dark side of the River.

Not so good parts:

  • Extra grails plugins: Plugins are required, example lang-groovy is required to work with grails and that too doesn’t provide grails domain friendly methods
  • Replicas: Overhead of creating mongoDB replicas The current implementation monitors oplog collection from the local database. So, Database replica is a must have. It does not support master/slave replication. Monitoring can be done using tailable cursor
  • Multiple Rivers: MongoDB river does not allow to fetch content from more than one collection in a single river. Hence, we need to create multiple rivers where one river is mapped to one collection

 

If you’re still not convinced and want give it shot then use Stackoverflow-link but you’ve been warned, it’s like running rm -rf /* command, it’s tempting but you’re better off not doing it 🙂

Elastic Search Grails Plugin:

TL; DR: Can’t make it any shorter, must read.

The elastic search plugin intends to implement a simple integration with Grails of the elastic search engine. The plugin’s focus is on exposing Grails domain classes for the moment. It is inspired by the existing searchable plugin as reference for its syntax and behavior. I’ll try to keep it as simple as possible, you know, for young padawans.

Setting it up:

elasticSearch {
     datastoreImpl = "mongoDatastore"
     client.mode = "node"
}

 After configuration we only need one line in domain to get started:

          static searchable = true

And if plugin starts throwing exceptions because of unindexable fields like your enum etc. Then don’t panic, you just need to limit the fields that you want to be searchable since you don’t want all the fields in the domain to be searchable, do you?

static searchable = {
    only = ["name", "birthdate", "age"]
} 

Now, we can either use public search method for cross-domain searching or we use the injected dynamic method in the domain for domain-specific searching.

Map globalSearchResult = elasticSearchService.search("Donald Duck")

Map domainSearchResult  = MyDomain.search("Donald Trump")

You can easily guess which query is going to be faster. And that’s it, you can start using your kickass search server. But the path to become a Search Jedi is not that easy my young padawan. Alright, that’s cool and all, but how am I going to do complex queries, I hear you, we’ll get to that. Alright. First let’s look at the feature table.

Feature Comparison:

FeatureGrails PluginMongo River
Mongo Version3.0.0 (tested on)Independent
Implementation TimeFasterSlower
Add only one collectionNoYes
Additional PluginNoYes
Needs Mongo ReplicaNoYes
Limit Domain fields to be indexedYesYes

 

Setting Up Grails Plugin:

Elastic search provides support for primitive data types like String, Numeric datatypes (int, double etc), Date, Boolean, Map, then suddenly there is an Object datatype and almost nothing in between. But in grails, our domain’s fields are not restricted to these datatypes. So, the two issues with datatypes are:

1. Unsupported data types (like mongo-objectIDs, enums) are stored as an object datatype.

2. Object type is used for storing embedded classes.

Unsupported domain fields issue:

Unsupported fields needs to be handled by adding transient fields which are proxy values to those fields that cannot be indexed directly to elastic search and returning allowed datatypes from them.

Class myDomain {     
    Status myStatus          // Status is an enum
    ObjectID id
    int weight
    static searchable = [ "elasticStatus", "weight", "elasticID"]
    static transients = [ "elasticStatus", "elasticID"]

    String getElasticStatus() {
        return myStatus?.id ?: ""
    }

    String getElasticID() {
        return id.toString()
    }
}

In this case the enum ID would be indexed that can later be queried easily using elastic search plugin.

Embedded classes issue:

Embedded classes could be indexed in a similar way as unsupported data type but this time the class properties need to converted to map of simple datatypes.

class Country {

int population
String continent // Method to convert all class properties to a map.  Map asMap() {  this.class.declaredFields.findAll {!it.synthetic }.collectEntries {  [(it.name):this."$it.name"]  }} } class MyDomain {     String name Country country       // nullable constraint static searchable = [ "name", "elasticCountry"] static transients = [ "elasticCountry"] String getElasticCountry() { return country.toMap() ?: [population: 0, continent: ""] } }

That’s all the configuration you need in your domain. You’re welcome.

Lower-level Queries to Elastic Search Server:

For complicated queries we cannot rely on plugin’s default query method. We have to use lower-level queries which are just like grails criteria so we can divide our queries into two parts: AND Queries and OR Queries

List result = []
List andQuery = []
List notQuery = []
AndFilterBuilder andFilterBuilder = new AndFilterBuilder()
andQuery = ["elasticCountry.continent: Asia",  "name: John"]
orQuery = ["weight: 50",  "name: Leo"]  
BoolQueryBuilder boolQueryBuilder = boolQuery()
    .must (queryStringQuery(elastifyList(andQuery, "AND")))
    .mustNot (queryStringQuery(elastifyList(notQuery, "OR")))
FilteredQueryBuilder filteredQueryBuilder = filteredQuery(boolQueryBuilder, andFilter(someFilter))

Querying the elastic search server:

elasticSearchHelper.withElasticSearch { client ->
     SearchResponse searchResult  = client.prepareSearch("my.domain.package.domainName_v0")
        .setQuery(filteredQueryBuilder)
        .execute()
        .actionGet()
    for (SearchHit hit : searchResult.hits.hits) {
        result << [name: hit.source.name, status: hit.source.elasticStatus]
    }
}

 

Let’s dig deeper into it. We’re using the ElasticSearchHelper bean and encapsulating our code within a withElasticSearch block. Then we’re passing the name of the index which is a combination of package, domain name and version. You can check for all your indexes in data directory at root of your project. Getting back to the code, we’re iterating over the search results to get all the required data. Using a separate method for processing search result would be more logical depending upon the number of indexed items.

And that’s all you need know for performing all kinds of queries to elastic search server. For other queries like regex etc,you can always look into the elastic search documentation. If you’ve any query, feel free to comment below.

Useful Links:

1. Grails Plugin Documentation

2. Elastic Search Documentation

3. Mongo River Repository

4. Elastic Search tutorial video

Enjoy!

About CauseCode: We are a technology company specializing in Healthtech related Web and Mobile application development. We collaborate with passionate companies looking to change health and wellness tech for good. If you are a startup, enterprise or generally interested in digital health, we would love to hear from you! Let's connect at bootstrap@causecode.com

Leave a Reply

Your email address will not be published. Required fields are marked *

STAY UPDATED!

Do you want to get articles like these in your inbox?

Email *

Interested groups *
Healthtech
Business
Technical articles

Archives