[testlib] fix page validator selection. Closes #2869456 * use a validator that checks XML well-formness for the html5 doctype (as a convenient side-effect, it does not choke on e.g. a canvas tag like the HTMLParser does) * use a DTD validator if there is an xml declaration * else use the HTMLValidator