This doesn't seem like it's going to work. Given a choice between search engines pretty much everyone is going to choose Google. I think there are definitely areas that Google acts anti-competitively. But search? The competitors are a URL away. And few use them because Google is just flat out better.
There are structural issues that make it very difficult for new competitors to gain traction in search. For example, many website owners are now hostile to new web crawlers, but they're happy to allow Google to hammer their servers because they want that sweet, sweet search traffic.
Mandating a commonly accessible crawl, with cached versions of the pages, would help new entrants a lot.
Also, there're large network effects with ad networks. It seems unlikely that many marketing managers are going to take the time to do targeted keyword queries on your search engine with 1/1,000,000 the traffic of Google.
I don't know a single website owner that is hostile to any search engine web crawler, unless that web crawler is slamming them with so many requests they're effectively getting DDoS'd.
Reddit, twitter, facebook are just three to start. There are plenty that disallow crawlers except google. We've crawled a significant amount now and just because you are unaware of them doesn't mean they don't exist. I can attest they're there.
I'll also add plenty of sites don't block any engine but confer special privileges to google bot which depending on the site and their size are almost the same thing.
Edit: And I'll add to limit confusion Reddit hides the sitemap and denies access there's is not an outright ban -- it just makes it a lot harder.
Content kingdoms have their own reasons to be hostile to Everyone searching, including Google. Even when their content is "searchable" by Google, they'll tease you with something and gate almost all content.
They are part of the story of why Google is degrading, not why they're doing well.
So unless you have a google domain your sol, it's also just generally frowned upon. We have our own UA WhizeBot with an email contact so you can let us know if our crawler is doing anything you'd rather it not.
There have been a few legal cases that protect scraping publicly available information on the web but we'd rather follow robots.txt to avoid the potential for shenanigans in any case.
That's "poor man's cloaking". Most people that genuinely care if it's Googlebot or not will verify it appropriately by doing a DNS lookup and a reverse DNS of that IP to ensure it's one of Google's IPs.
'Faking' the UA is much more likely to annoy site owners and end up with a permanent block.
I do, some third party search crawlers are just badly programmed, and after you get burned a bunch of times you just want to deny anyone who isn't one of the main players. I think they are basically startups with a lot of money to spend on crawl compute, but who haven't really figured out their crawl engine, and it can go wild on your site.
You also have bots that seem to be credential stuffing, bots that seem to be content scrapping (stuff with same typos shows up elsewhere after their visits, really obvious on new / fresh articles, bots that seem to be exploring for copyright claims, rando bots (maybe comment sentiment analysis for stock trading) etc.
Right, but that’s not really anti-search-engine, that’s anti-people-who-want-our-actual-data.
Like I said in another comment, if you’re someone that just wants to surface Amazon results they love you. It’s the people that want to take advantage of amazon data in some other way they’re trying to stop.
Oh, good to know, thanks. Are you running relatively complete crawls of larger sites at similar rates to Googlebot?
I've definitely seen my share of robots.txt that give special permission to Googlebot, but maybe my corner of the web was unusually aggressive toward crawlers.
No, not complete crawls of any sites, we've aimed to get a wider coverage in place of deeper crawls. This means we have most sites indexed and are now gradually going deeper. We do encounter robots.txt blocks, but we don't see it as a major issue right now.
It’s a bit like saying, in 1911, that given the choice between petroleum companies, everyone is going to choose Standard Oil, to which the answer is a resounding yes, of course they will.
Ask anybody in much of the Midwest where they will choose to buy groceries and they will answer Walmart.
This kind of reasoning is not an argument against the philosophy behind antitrust regulations. Rather, it is emblematic of the situation that monopolies create for themselves, by using their economies of scale to create pricing that cannot reasonably be expected to lose any competition over customers.
The same economies of scale benefit consumers and the economy as a whole as long as the business behaves in a manner consistent with the shared values of society. As soon as they decide to behave differently, though, society is at the mercy of a monopoly because they have consumed the entire market.
It's not illegal to be a monopoly - especially by providing a superior product or service.
The case hinges on the no-compete, tying, and default placement agreements. If Google didn't pay anyone to be the default search engine, didn't require Android Phone makers to pre-install Google apps and make Google the default search in order to get access to Google services and the store, etc then there's no problem.
The complaint also says that one of the reasons Google's results are better is the network and scale effects that come from owning 80-90% of search. A competitor has a very difficult time no matter how good their algorithm because you need insights you can only get when people use your service at scale. I don't personally know if that is true but that's part of the complaint.
The proposed antitrust violation is likely the combination: using their dominant position to outspend any other players when paying for default placement to ensure they're the only one with the scale to have the best search results. That forms a self-reinforcing cycle: users who do choose pick you because you have the best results and you use that fact to fund no-compete and default placement agreements to ensure you have unmatched scale which further reinforces your ability to deliver the best results.
Again: I don't know if this is true or a valid argument, but it seems to be what the complaint claims.
Agree (responded in an earlier comment). If DOJ are going to look anywhere, it should be the advertising side of things and placement in search results (as opposed to sidebar ads). Even then, I'm not sure there is any case.
The question I would posit is: Do the actions from Google that the Justice Department consider to be anti-competitive prevent competitors from becoming better?
For example, let's say I could be curious about DuckDuckGo if I had to choose a default search engine when I got an Apple device instead of Google being the default, then that could be revenue to DuckDuckGo for them to improve their search engine.
So what are you going to do, force Apple to sell their search default option for less money (whatever DuckDuckGo can afford to pay) in order to support more competition? It's really Apple's choice and interest what it does with the default search, Apple could decide tomorrow to point it to its own search engine and there's nothing Google, the DOJ or anyone else can do about it.
Not really sure what the DOJ expect Apple or Google to do in this situation, it seems to me that 2 companies entered a mutually benefiting contract. You can't argue that Google "colluded" with Apple, there's a lot of throwing punches between each other (all the privacy oriented moves Apple is doing are hurting Google's business) and again Apple could be making its own search engine anytime they wanted, they already replaced Google Maps with their own thing.
The monopolized market in this case would be the market for default search engines on mobile.
If the only two players (Google and Apple) both use Google Search by default, then Google has effectively captured 100% of the mobile ad market. (And since Google is paying Apple et al for that default state, it is indeed a market.)
The court could require device makers (including Google itself) to prompt users to select their search engine provider, or potentially ban Google from buying search engine defaults from other companies (Apple, Mozilla, etc.) as an anticompetitive practice.
That's an interesting scenario. Let's say Google is banned from buying search engine defaults. Then Mozilla would no longer be able to sell the Firefox default to Google.
Sure, they could sell it to someone else, but without the biggest player bidding up the price, it'd probably sell for a fraction of what it does today -- which could be devastating to Mozilla, given that almost all of their funding comes from the search deal. What would that do to competition in the browser space?
Competition in the browser space is over and Chromium won. If this had been regulated earlier by preventing Google from preinstalling Chrome on all Android devices, we wouldn't be in this situation where Firefox is on life support, but here we are.
Using Firefox as a reason not to break up some of Google's hold on search seems extremely short-sighted. Besides, Edge using Chrome changed the whole browser market dynamic as more and more people are using it. Once Microsoft reaches a somewhat decent percentage of install we will be back at a two (three?) player situation.
Apple could actually make more money. Say Google is split into two renamed search engines: elgooGA and elgooGB, with evenly divided assets. They would immediately be two search titans competing for the important Apple contract and would try to outbid one another. Perhaps elgooGA wins Apple, but then elgooGB wins Firefox. Every alternative platform to Apple would stand to get more money. And the users would then become more accustomed to having different default search providers and usage might be split. That is a win for DuckDuckGo and all other competitors.
Regarding DuckDuckGo, my understanding is that though they have their own web crawler, most of the information is coming from Bing and Yahoo, who are the only direct competitors to Google.
Before asking "should we" we need to answer "can we". If forcing Google to give users a choice results in them picking Google anyways then whether we should do that or not is irrelevant. Since it won't achieve it's stated purpose there's no point in forcing it.
The question is: does any of the competitors stand a chance against Google if they can't acquire more users and therefore improve their engine with the added revenue? Seems like a catch-22, and having Google as the default search engine everywhere, including in the web browser they make and that has the highest marketshare only exacerbate this.
>few use them because Google is just flat out better.
I completely disagree with this part, but you're totally right on people continuing to use it.
Search is highly dependent on the query. At first DDG's results seemed bad to me, but after a while I think I've changed how I write my searches. It's hard to explain, but I guess I'm putting more thought into understanding what it is that I hope to find.
Now, Google's results seem to just be a listing of whoever did the best SEO targeting on the subject, and ultimately that means worse results for me. It's less about what I'm looking for and more about what Google has to show me... and Google always has something relevant to show me. When DDG doesn't, I'm forced to re-consider my query and try again, ultimately reaching a better destination.
However, "change the way you search" is niche at best. Google is satisfying because it's so easy to use, that you almost don't even have to write a query. It's like an automatic "I'm feeling lucky" based on it's knowledge of you and your location and time of day, etc...
good god is this overdramatic. This whole thread is. Google is a good citizen, but people on hacker news just love a good story about tearing up big tech. How is society better off with google torn up? Arguments I expect and don't accept:
1. It allows alt search engine or alt browser to succeed. So what? The alt search engines and browsers are worse than the google ones. It doesn't benefit any actual people to have them rise up.
2. Google screwed my friend one time. Every single company (and person) on earth is guilty of this. One swallow does not a summer make.
3. Google is restricting this tech I like, or google is sometimes wrong about their bans. See (2)
And so on. I haven't seen anything compelling in this thread, just a whole lot of hyperventilating.
And to your point. No, search is not a wall of ads. But if you feel that way, literally change your search engine. It's easier than typing this comment you just posted.
Google does have a great service, but calling them a 'good citizen' is a stretch, IMO.
The events of celebritynetworth.com are a good example of why they've not been behaving as a good citizen (there is a thread about that on this forum).
A good number of people have took up your suggestion and changed their search engine, DDG's traffic count continues to rise. The reality is that many users will use whatever is put in front of them, i.e. how many people use the URL bar of their browser for searches, or use a search toolbar for a specific URL? Google knows this well, hence their paying to be the default search engine choice and paying an amount that no other engine can match.
Google's knows they are the best in class when it comes to general search, but to be sure they have been aggressive in ensuring that no one else gets a look in.
The UK's CMA report estimates that an entrant to the market would require around £20 billion of capital to be a credible alternative, as in, a proper crawling search engine, not a meta search engine like DuckDuckGo. That's before paying the likes of Apple billions of dollars simply so that people will use "what is there".
Having such a dominant position comes with responsibility, which is not solely Google's. The social networks along with Google are essentially the gateways to the web and how they present that information is important- and essentially only one point of view. Having room for more competition, choices, algorithms surely is a good thing.
I was just searching for "quill", a javascript editor. I clicked the first result without really looking (after all, i m searching name, right? ) - and it took me to some competing product called fro-something. I wasted my time, their money , yet google still made money.
I just tried search for "quill" in incognito, and I got quill.com as the first hit (both ads as well as organic result) - apparently there's a company called Quill Corporation (quill.com). It seems reasonable to return that as the first hit. quilljs.com is 5th hit on the first page. "quill javascript" has quilljs.com as the first hit.
This seems reasonable. What's your suggestion on how to make this better ?
just checked my history, my first result was "froalla", happened more than once
(the query must have been "quill js" or "quill editor" cant remember -- it's an example of the infamous 'google® tax' - an obviously keyword-targeted ad)
It is a search for "express-http-proxy" "logging" but at leat I got maybe one relevant result and a bunch of wildly unrelated ones, despite doublequotes.
There was one among the top 10 when I did the search.
As for most of the rest of the first 10, null would have been a valid, more correct and generally better result!
I actually wasted several minutes on one of them right away since I haven't still gotten into the habit of verifying every single Google result to se that it actually is what I asked for and not whatever filler Google decided to use this time. Not that it should be necessary, - they had this sorted back before 2010!
Seriously: why can't Google or DDG get this right anymore?
This sounds like exactly what the grandfather post (by wasdfff) was discussing. A competitor advertised their product based on the keywords you searched for.
You didn't buy their product, but Google helped them get your gaze briefly.
What is search but a list of adverts associated with your keywords?
Google can do without this "google tax", and imho they should. It's unethical , and it's not like a competitor putting a sign next to yours. The user is literally searching for a brand and is instead driven to click another. At best the competitor should be an ad on the side in this case. Considering how (especially in mobiles) google is often used as a kind of DNS-autocorrect in the omnibox, this behavior is unethical on the same level as websites with popups. A rich company like google would not normally allow it to itself, they can easily dispense with such sleaziness. The fact that they can do it unpunished is indicative of a monopoly position.
It will work most likely. We aren't in 2000 any more and both parties are mad Big Tech. There will definitely be some heads rolling in FAANG in the next few years.
