Remove needsDOM() in ParseFilter

[jnioche]

This was provided as a way of avoiding building the DOM structure for when no ParseFilters were requiring it and is used only in the Tika parser.

The RobotsTag extractor requires such a DOM so unless we are able to extract the robots directives differently e.g. via SAX and a custom ContentHandler we should reconsider whether this method is really needed.

The JSoupParseFilter could avoid building the DOM too and extract the values via its org.jsoup.nodes.Document prior to building the DOM (if required).

Related to #154

[jnioche]

The SitemapParserBolt calls the ParseFilters too but does not generate a DOM which can result in a NPE. This has been addressed in [b0d6f59] but relies on needsDOM()

[jnioche]

The fix in [b0d6f59] is actually quite problematic. Originally needsDOM() was merely an optimisation aimed at avoiding the cost of building the DOM if no parse filters needed it. This meant 'the parsefilter might need a DOM so please provide one' which is not the same as 'this parsefilter needs a DOM' as implied by [b0d6f59].

We could remove the whole thing and instead let the users specify via config whether a DOM should be generated or not. This means that each individual parsefilter should check whether the DOM is not null before trying to process it with Xpath.

[AsifQureshi04]

Hi @jnioche , I'd like to work on this issue.

After exploring the codebase, I found that needsDOM() is used in two places:

1. JSoupParserBolt — to skip building the W3C DocumentFragment from the JSoup Document
2. Tika's ParserBolt — to skip adding a DOMBuilder to the handler chain

Three production filters override it to return true (XPathFilter, LDJsonParseFilter, DebugParseFilter), and ParseFilters.filter() uses it to skip filters that need DOM when none is available (e.g., from SiteMapParserBolt).

Could you confirm the preferred approach?

1. Remove needsDOM() entirely and always build the DOM in both parsers
2. Remove only the method from ParseFilter's public API but keep the lazy DOM construction internally (always build it when the parser has the content available)

Happy to implement whichever direction you prefer.

[jnioche]

thanks @AsifQureshi04
Building the DOM is costly and makes the parsing a lot slower, we definitely don't want to do it systematically.
Some of the parsefilters that needed it have since been ported to JSoupParseFilters, the purpose of which was in part to remove the need for the DOM.
Would you mind checking which of the implementations that use it do not have a JSoupParseFilter equivalent? Ignoring DebugParseFilter which is there just to debug the actual DOM.
Thanks!