Hi
I've reached out to Google with this one, but I've had no success engaging them yet. So here we go.
We created a new Agent in DialogFlow ES. This has been used in the flows without any issues. We have now tried to add a FAQ page to the DialogFlow Knowledge section and when submitting the document we keep getting an error in the browser console of "Failed to crawl https://somesite/faq. Please verify that your URL is publicly accessible and is hosted on a site that can be indexed by Google Search."
So that has lead us down the first rabbit hole. The site is publicly accessible. You can browse it from a normal browser. Next is Google Search Indexer. We added the property and verified the domain. I verified that robots.txt is allowing all search engine. The noindex tag is also not present in any of the website pages. Then using URL Inspector, I had to request the FAQ be added to the crawler and then index. This completed without any problem. I verified some other pages within the site as well and we are getting the same successful result in those pages. So from Search/Index and Crawling it all looks fine.
I've eliminated or ruled out both the reason for the failed crawl error, but when trying to add the FAQ page to the Knowledge section in DialogFlow ES, we still getting the same error.
Any ideas on what else to try?
I raised this on this forum to help anyone else out that might have the same problems.
TIA