Fixed

• ALTO indexing is now more tolerant of whitespace inside of element
  declarations, fixing e.g. parsing of the Transkribus default ALTO formatting

Changed

• Added compatibility with Solr 10
• Automated tagged releases: CI now validates release metadata, publishes
  GitHub releases from the changelog, deploys versioned documentation and
  updates the Solr plugin repository metadata

Dependencies

• Updated Solr to 10.0.0 and Lucene to 10.3.2
• Updated Commons Text to 1.15.0, Commons IO to 2.22.0 and Guava to
  33.6.0-jre

Deprecated

• WIP/pre-release builds from main are no longer published to GitHub
  Releases, the documentation site or the Solr plugin repository; use tagged
  releases instead

Fixed

• Highlighting of hyphenated words when context was set to 0 lines, in this
  case we now include the line with the second part of the hyphenation on the
  next line.
• Sometimes the second part of a hyphenated word in a fragment would get
  skipped

Dependencies

• Updated Solr to 9.10.0 and Lucene to 9.12.3
• Updated other dependencies to their latest versions

Fixed

• Highlighting when a highlighted span was crossing multiple vertical columns on
  the page

Chores

• Use JSONC for Renovate config instead of JSON5

Changed

• It is now possible to use ExternalUtf8Filter in a OCR field definition and still use inline OCR, mixed with external OCR.
• Improve performance when highlighting with hl.ocr.scorePassages=false, we now only analyze as many passages as the number of snippets that were requested

Fixed

• Version constraints in the Solr plugin repository have been corrected, these were previously pinned to Solr 9.0, which was wrong.
• When highlighting with hl.ocr.scorePassages=false, the snippets returned would be the last snippets in the document, not the first, as intended. This has now been fixed and we only return the first snippets in the document.

Changed

• Respect global request limits (time, memory, cpu) from the [*allowed Solr
  query parameters][1] in addition to our own OCR-specific limit

Fixed

• Highlighting responses that are partial by the time they hit the highlighting stage
  during distributed request processing resulted in errors, this has been fixed.

[1] https://solr.apache.org/guide/solr/latest/query-guide/common-query-parameters.html#timeallowed-parameter

Breaking Changes

• When using Solr >= 9.8 and the the plugin is included via the directive, the Solr JVM needs to be launched with -Dsolr.config.lib.enabled=true for the plugin to work

Changed

• MiniOCR page fragment parsing is now more robust regarding the order of attributes and additional whitespace
• Added a small command-line tool to convert ALTO and hOCR to MiniOCR in util/miniocr.py
• Removed usage of deprecated API for retrieving stored field values for Solr versions that support it

Changed

• During indexing, we now only need a single pass through the input files, instead of
  two, this is in preparation for the S3 storage backend, where we don't have the luxury
  of relying on a page cache to paper over our inefficencies.

Fixed

• Fix bug that resulted in missed matches during highlighting (#442, thanks @schmika!)
• Fix bug that resulted in incomplete reads from the input file under some circumstances (#441, thanks @schmika!)
• Compatibility with Solr 9.7

Major performance and stability improvements in this release, upgrading is highly recommended.

Changed:

• Add support for multithreaded highlighting. Uses all available logical CPU cores by default and can be tweaked with the numHighlightingThreads and maxQueuedPerThread attributes on the OcrHighlightComponent in solrconfig.xml.
• Removed PageCacheWarmer, no longer needed due to multithreading support.
• Completely refactored, simplified and optimized I/O stack to reduce number of file system reads and allocations/data copies during highlighting, accounting for a significant performance improvement over previous versions (4-8 times faster in a synthetic benchmark that was not I/O-bound)
• We no longer memory-map files for reading. Benchmarking revealed that it did not improve performance with the new I/O stack (probably due to the reduced amount of actual reads), on the contrary, performance was improved for many concurrent queries. A huge drawback of the memory-mapped approach was that in the presence of I/O errors like disappearing mounts, truncated files, etc, the JVM could simply crash (due to the kernel sending a SIGBUS signal when encountering an I/O error).
• When locating breaks in the forward direction, we used to put the break point at the end of the limiting element opening tag. With the new implementation, the break point is now at the start of the limiting tag open element, i.e. no part of the limiting element is contained in the created section. This leads to a small change in the scores assigned to passages (since BM25 uses the length of the scored content in its calculations).

Fixed:

• When using source pointers with multiple files, the plugin no longer leaks file descriptors. We previously didn't close the currently opened file when opening the next one.

Changed:

• Add support for Solr 9.6
• Removed unused classes
• Refactored timeout logic to match new approach used in Solr >= 9.5
• Dependency Updates

Changed:

• Missing files no longer fail the complete search request, instead the OCR
  highlighting for the document is skipped
• Add support for Solr 9.5
• Updated documentation with warning for Solr 9 users to disable security sandboxing
  when using pointers to external files

Fixed:

• Regular highlighting in case no hl field can be determined works again (#404)
• Passage building across more than two concatenated files works now (#422)