[testlib] fix page validator selection. Closes #2869456
* use a validator that checks XML well-formness for the html5 doctype
(as a convenient side-effect, it does not choke on e.g. a canvas tag
like the HTMLParser does)
* use a DTD validator if there is an xml declaration
* else use the HTMLValidator