Google’s Mysterious New Chrome Web Store User Agent: What Does It Mean for Your Browser?
Ever stumble upon a mysterious visitor in your crawl logs called Google-CWS and wonder, “Who invited this guest?” Well, it turns out Google has rolled out a new user agent dubbed Google-CWS—think of it as Chrome Web Store’s personal fetcher, acting only when you ask it to. This nifty bot fetches URLs tucked away in the metadata of Chrome extensions and themes, bypassing the usual robots.txt rules because, hey, it’s user-triggered! It’s kind of like having your own personal assistant running errands only when you say so, ignoring “Do Not Disturb” signs. For developers and site owners scratching their heads over unexpected crawl activity, this clears up the mystery and helps you understand exactly what’s poking around your site. Curious to dive deeper? LEARN MORE.
Google has added a new user agent to its help documentation named Google-CWS. This is the Chrome Web Store user agent that is a user-triggered fetchers.
More details. Google posted about the new user agent over here, it reads; “The Chrome Web Store fetcher requests URLs that developers provide in the metadata of their Chrome extensions and themes.”
What are user-triggered fetchers. A user-triggered fetchers are initiated by users to perform a fetching function within a Google product.
The example provided by Google was “Google Site Verifier acts on a user’s request, or a site hosted on Google Cloud (GCP) has a feature that allows the site’s users to retrieve an external RSS feed. Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. The general technical properties of Google’s crawlers also apply to the user-triggered fetchers.”
Why we care. If you see this user agent in your crawl logs, you now know where it is from. The Chrome Web Store fetcher requests URLs that developers provide in the metadata of their Chrome extensions and themes.
Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.
About the Author
