Approach To Incrementally Index Database Data From Multi-table Join In Lucene With No Unique Key

I have a particular SQL join such that: select DISTINCT ... 100 columns from ... 10 tabes, some left joins Currently I export the result of this query to XML using Toad (I'll quer

Solution 1:

Lucene has no concept of "updating" a document. So an update or an add is essentially a delete + add.

YOu can track the progress here - https://issues.apache.org/jira/browse/LUCENE-4258

So you will need to keep the logic of doc.hashCode() in your app i.e Do not ask lucene to index a document if you know that no values have changed ( You can have a set of hashCode values and if it matches it then that document has not changed ) . You might want to have a logic for tracking deletes also

Solution 2:

If you increment an id on each relevant update of your source DB tables and if you log these ids on record deletion,

you should then be able to list deleted, updated and new records of your data being indexed.

This step might be performed within a transitory table, itself extracted into the xml file used as input to lucene.