[robots.txt] Root facet navigation URLs must not be excluded, but their subpaths

We now automagically exclude facet navigation URLs. That may render a too large part of the content to be excluded from indexing. E.g. if /products points to the general products facets overview without any filters, then none of the products are indexed.

What we should exclude is its subpaths instead, since those lead to the same resultsets and documents being available under multiple URLs, and those should be excluded from indexing. E.g. /products/price, /products/categories etc.