I don’t know. Google is a search engine for web content. I don’t think it should discriminate based on the content being freely available or not.
Adding a feature for an end user to say restrict a search in some manner might be nice, but the point, at least in my opinion, of a web search engine is to find any and all matching content on the web. If some of that is paid or otherwise blocked content it’s still doing its job properly.
Once upon a time, I believe Google had a policy that they would only index content that was actually visible when you went to the URL without cookies. Anything else was deemed improper SEO. In particular, it was not allowed to serve a special version of a page to Google.
This was nondiscriminatory, and I think it was a great policy.
I agree with your point, but as a user, I would want Google to rank results lower if the content served to the Googlebot isn't the same as the content served to the general public.
I'd expect this behavior to be a red flag in Google's ranking algorithm. Don't drop Quora, but just let them slide below the fold on results.
It may be in the guidelines somewhere, but if they actually cared at all, LinkedIn and Pinterest would have disappeared from all search results years ago.
Yes, that's an unfair practice. Ans they disable Google cache at the same time. Why do I see it in the results when I can't read it without making an account, or worse paying some subscription. It should be hidden then.
They already did something similar for Experts Exchange a long time ago. They threatened to delist EE because of their deceptive SEO (including answers in search results, but requiring accounts when going to their page from Google). Google insisted the actual answers be listed. Deceptive search results made their service worse. Arguably Quora, Pinterest, et. al. are doing the same.
Maybe a flag for the pages that require accounts would be in order. But for the shills on Quora, well, why would Google show ads for businesses' services as top results for free - especially since that's the vast majority of their revenue?
Uh, Google's main job is to rank results according to what other people find helpful. There is no truly objective way to do this. The closest you can get is to define "objective" the way pollsters do, as a way of understanding average opinions.
What does "average" mean here, though? Do you rank restricted sites higher if more people have accounts?
They could probably do a better job if they knew what you're subscribed to. For example, people with a Quora account or Washington Post subscription should get that ranked higher. But this would mean telling Google more about you, which seems to be unpopular around here?
It's not that complicated. Google indexes webpages. Webpages are identified by a URL. If, for any client, without any context or previous interaction, the URL results in anything other than the indexed content, it should not appear in search results.
If a webpage requires login credentials, which the crawler obviously does not have, the crawler should have received a login page.
No, the point of the search engine is to help the searcher find what they are looking for. It should be easy for Google to figure out that searchers are NOT looking for what Quora and Pinterest are providing.
> but the point, at least in my opinion, of a web search engine is to find any and all matching content on the web
This is where the early search engines started out at. Turns out there is a LOT of content found for many searches, and the real value to most people is in finding _relevant_ content.
The devil is in the detail of how do you define relevance? This is why I think it is so important to have multiple search engine choices, each with their own algorithms and different sets of user options.
> I don’t think it should discriminate based on the content being freely available or not.
I feel at least a little weird about giving crawlers a carte blanche to paid content. Creating relevant pages, showing them to the crawler to get ranked higher, and then not showing them to the user but requiring something in return first... I don't know.
I do see your point: it would be good to have everything indexed, even paid content or content available only in certain regions or whatever, just so you know what your options are if you're willing to pay for it. But then I think there should be a button to hide those results (either opt-in or opt-out, to show paid content).
As it is, it just weirds me out to show the crawler different content than you show the user and not be seen as fraudulent and get delisted altogether.
If I create a site that serves a world of unique, useful content to Google, but locks out everyone else completely, do you want that content indexed? Is it really on the web? The search result only serves to annoy you into thinking there's anything to find.
Agreed, but Google does have UX Signals. In a perfect world, you'd think Google would be able to identify the "wall" as part of their signals they measure from a site & rank accordingly.
Adding a feature for an end user to say restrict a search in some manner might be nice, but the point, at least in my opinion, of a web search engine is to find any and all matching content on the web. If some of that is paid or otherwise blocked content it’s still doing its job properly.