Matt on Software

When you create in index in RavenDB you are able to write a Map statement that controls how your source Json document is stored in the Lucene index.

So, given the following POCO’s

public class TShirt {
    public String Id { get; set; }
    public String Name { get; set; }
    public int BarcodeNumber { get; set; }
    public List<TShirtType> Types { get; set; }
}

public class TShirtType {
    public String Colour { get; set; }
    public String Size { get; set; }
}

You have 2 main options, which are shown below as a code sample and a diagram illustrating how the index looks

Option 1

Flatten out the nested items and index every combination of TShirt/TShirtType as a single Lucene document

from shirt in docs.TShirts
from type in shirt.Types
    select new { 
        shirt.Name, 
        type.Colour, 
        type.Size,      
    }

Option 2

Leave the nested TShirtType items inside the parent TShirt

from shirt in docs.TShirts
    select new { 
        shirt.Name, 
        Shirt_Types = shirt.Types.Select(t => t.Colour)
        Shirt_Sizes = shirt.Types.Select(t => t.Size)
    }

Note: There are a few things that RavenDB is doing for you here

• If your Map statement contains an item that is an IEnumerable, then it will “flatten” out the items for you and index each one as a separate field inside the Lucene document (see Option 2)
• Imaging if you do a query that matches several Lucene documents, but only 1 RavenDB document, you will only get each RavenDB doc once. This means that if you have a query with “Take(10)”, you will get 10 matching documents as expected. To see the code that handles this, take a look at Ayende’s blog post.

A while ago an interesting scenario came up on the RavenDB group, that the current query mechanism couldn’t handle. The full details are in the documentation, so I’m not going to repeat them here, but the issue is due to the way that RavenDB has to index relational data in Lucene.

The solution was to allow users to Intersect queries on the server-side and then only get back the documents that match all the sub-queries. The code to do this is shown below:

session.Query<TShirt>("TShirtNested")
      .OrderBy(x => x.BarcodeNumber)
      .Where(x => x.Name == "Wolf")
      .Intersect()
      .Where(x => x.Types.Any(t => t.Colour == "Blue" && t.Size == "Small"))
      .Intersect()
      .Where(x => x.Types.Any(t => t.Colour == "Gray" && t.Size == "Large"))
      .ToList();

However implementing this wasn’t straight forward because RavenDB allows paging via Take()/Skip(). To see why this is an issue, lets take a simple example, where the user expects 3 docs back, i.e. Take(3).

Internally each sub-query is processed in turn, collecting the matching Raven doc ID’s each time. So firstly, “Where(x => x.Types.Any(t => t.Colour == “Blue” && t.Size == “Small”))” is applied, giving the following matches. Note: each row represents 1 RavenDB document, but several Lucene documents. Therefore, to be a match the entire row must satisfy all the sub-queries.

So far, so good, we’ve got 3 RavenDB docs that match, “tshirt/1”, “tshirt/2” & “tshirt/3”. The problem comes when we perform the 2nd sub-query as well, “Where(x => x.Types.Any(t => t.Colour == “Gray” && t.Size == “Large”))”:

We’ve now had to discard “tshirt/4” as a match, because it’s doesn’t satisfy both sub-queries, so we only have 2 matches in total. The solution to this is to perform the entire search again, but this time ask Lucene for twice as many matches. This gives us more candidates that match each sub-query, so by the time they’re all applied we can give the user the 3 matching documents they asked for.

In reality, the 1st query starts by getting twice as many docs as you ask for anyway, so Take(4) causes it to get 8 docs, in the hope that this will leave 4 by the end. But every time it ends up with less than it needs, it doubles the amount and repeats the process.

The reason for going to all this effort, is that it’s more efficient to get the results from Lucene in small pages, asking for 1000’s of results when we only need 100 is expensive. In reality most queries are limited to 128 matches, due to RavenDB being safe-by-default. So this mechanism is meant to be a cheap way of getting the amount of docs we need, but still allowing for the scenario where after collecting the sub-queries we don’t have enough results and have to start again.