Hi we are experiencing an issue with our FTS Indexer/Queue processor. We are trying to run indexing on a database after a fresh migration of a large set of data. We have run the indexer on all entities and the queue has been filled up. We are now running the Process Queue task as a scheduled task every minute or so processing in batches of 300 records at a time.
This task will run for a few hours but eventually throws the attached error and cannot seem to recover. We are forced to restart Tomcat to get the queue processor to continue processing. Looking at the error it looks like it is looking for the write.lock file and cannot find it. It is almost like something gets out of sync and it has already released the lock but doesn’t know it.
FTS creates a single index writer and a lock for a Tomcat process and keeps and doesn’t close it. So index writer is recreated when you restart a Tomcat server.
Possible errors:
The lock file is being changed externally by antivirus or another external process.
The lock file is placed on a network resource (\\gmpvt.net\apps\WorkManagment\uat\FTS\write.lock).
A network error may occur while accessing the lock.
Application is running in a cluster and several nodes try to update an index at the same time.
We are running in a cluster and have setup middleware communication using the JGroup settings provided by the system. I have the scheduled task running as a singleton. Is there other configuration that I need to do to make sure this task can run without error in a cluster?
Also what is the issue with placing the file on a network resource? We are doing this as well in order to allow both application servers access to the same index files. Is there a reason this does not work and is there a work around?
what is the issue with placing the file on a network resource?
FTS/Lucene checks for the lock file when adding a document. Perhaps, the lock file isn’t available due to network reasons and the error described above happens. Are there any other errors in the logs related to FTS?
We are running in a cluster and have setup middleware communication using the JGroup settings provided by the system
I would recommend checking that FTS indexing is not performed in parallel on different cluster nodes.
Please turn on debug level for FTS logs (slf4j category: com.haulmont.fts.core) on both nodes.
FTS writes the following messages in a log file while indexing:
So I enabled the logging and watched the results for a few hours. It didn’t look like there was any case when both applications tried running the process queue task at the same time.
This time around the task never failed and ran all weekend without issue. However I tried using the FTS in the front end and received an error stating the file was locked:
I checked the Scheduled Task results again and it was still running every 30 seconds without fail. I disabled the Process Queue Scheduled Task to see if that would unlock the FTS lock so I could use the search functionality and it did not.
Upon restarting the Process Queue Scheduled Task it started failing with the attached error. I disabled the task once more and now the search functionality is working again but I cannot start the task without getting the attached error.
Does the search function create its own lock or is the lock only used by Process Queue task? It seems like the Process Queue task is not exiting properly and leaving the lock in place.