Improve Solr indexing performance

Description

Solr indexering performance can (hopefully) be improved, for queuing as well as indexing

Technical tasks

None

Deployment actions

None

Activity

Show:
Marijn Otte
July 21, 2015, 8:55 PM

Added a better index for the queue table. Improves 0.5 seconds per run

Marijn Otte
July 21, 2015, 9:31 PM

Change created to keep retry to start an indexer if another is busy

Marijn Otte
July 21, 2015, 9:37 PM

Batch size changed to 100 (15%-20% performance improvement). Queue size changed to 5000 to take advantage of every cron minute (50% performance improvement for background indexing)

Marijn Otte
July 22, 2015, 7:43 PM

200 documents:
Get queue records from database 0.9s 16,67%
Deserialize documents (incl get from DB) 1.3s 24,07%
Converter 2.7s 50,00%
Send to Solr 0.5 9,26%

Total 5.4s

Marijn Otte
July 23, 2015, 12:44 PM

Pull request comment fixed

Assignee

Unassigned

Client

None

Fix versions

Configure