Google a monopoly?
Dominant position Google on search market, which the US Department of Justice has deemed an illegal monopoly, provides the company with a decisive advantage in the war for artificial intelligence. A company spokesman said that Googlebot manages AI Overviews because AI and the company’s search engine are deeply intertwinedThe spokesperson added that search results display information in a variety of formats, including images and graphics.
Google also said that publishers can blockdisplaying specific pages or parts of the AI Overviews pages, but probably This would also prevent blocked snippets from appearing in all other Google search features.including in lists of web links. Therefore, most publishers who get at least half of their traffic from Google search enginedoes not want to take the risk associated with limiting the range.
Google’s Competition: New AI Search Engines
The development of generative AI has spawned a new wave of startups that want to offer search engines based on artificial intelligence modelsto provide concise answers to user questions. The popularity of chatbots has left Google worried about the future of its search engine, which has had virtually no competition for years. But before new search engines can truly threaten the giant’s business, they will have to build their own search indexes by searching the web yourself. And that’s no easy task.
Crawling costs site owners money, processing power, and storage, so many publishers include a file that sets the rules for how bots can visit their sites. The companies that are given the most freedom are usually Google and Microsoft’s BingOnly they can drive enough traffic to sites to make it profitable for their publishers.
Big can do more
However, search engine startups they can’t promise the same traffic as the big playersuntil they gain popularity. This is one of the reasons why young companies started pay publishers to license contentsaid Alex Rosenberg, CEO of Tako, an artificial intelligence startup. “Now you have a ton of tech companies that are paying for content, paying for access to it, because they need it to compete in any serious way,” Rosenberg said. “Whereas Google doesn’t have to do that.”
AI Learning Content Usage Agreements
The wave of deals between startups and publishers has apparently bypassed Google. The exception is a $60 million deal the American giant has signed with Reddit. According to anonymous sources to Bloomberg, Google has unofficially signaled to other publishers that it is not interested in any negotiations.
Google’s Deal with Redditwhere millions of users engage in heated debates on niche topicsoffers the company a treasure trove of information for AI modelsThe deal coincided with changes by Google that increased the presence of results from forums like Reddit in search results, which resulted in a huge increase in traffic to the social network. A Reddit spokesperson said that improvements in the quality and speed of the product contributed to the increase in traffic.
OpenAI Troubles
Small startups are not alone in their struggles. OpenAI recently launched SearchGPTa test version of its extremely popular chatbot tailored to search. However, popular websites including Amazon, Goodreads and Uniqlo have blocked GPT crawler on their sites, potentially spelling trouble for OpenAI’s search engine growth. OpenAI said sites could appear in search results even if they choose to exclude their content from AI training.
The useful features of the search engine
If a search engine is to offer users detailed insight into internet contentmust have access to detailed network indexor the table of contents of the pages analyzed by the search engine. It most often includes information about keywords, links, headings and content. But for companies that want to answer user questions directly using AI, the data takes on a different level of importance, said Vladimir Prelovac, founder of Kagi, a search engine startup.
“Generative AI models aren’t very intelligent on their own,” Prelovac says. “To get any high-quality output from generative AI, you need to have access to the same search index”. This index contains a collection of web pages read by search engine robotsaved and available in a given search engine. Google has the largest search index.
How to bypass the blockade?
Omnipresence robots.txt fileswhich set indexing guidelines, forces startups to make complex decisions, said Richard Socher, founder of search startup You.com. The files have not been recognized as legally binding in the U.S., so companies can index public dataas long as it’s not login information or personally identifiable information, Socher said. He said “any site that has a robots.txt file that only allows Google to index it” is essentially supporting Google’s monopoly.
Neeva, a search startup founded by former Google employeeswhich was acquired by Snowflake last year, advocated for “index neutrality” to make it easier for startups to build search indexes. Following the breakthrough court rulings stating that Google has monopolized the online search market, Department of Justice is considering seeking remedies, including forcing the search giant to sharing more data with competitorsor even breaking up the company, Bloomberg reported. One proposal that has attracted significant attention is for Google to share data it collects through Googlebot or give its famed search index to rivals. European Union Digital Markets Act already requires Google to share some data about search queries.
A crushing advantage
For Wiens, CEO of iFixit, Google’s advantage over other AI companies, gained from his search empire is at the heart of the antitrust issues for the company. “Separating Google search from their AI work would resolve the conflicts,” Wiens added.
According to representatives DuckDuckGo search engines, technological changes in search make “Google’s antitrust index even more problematic.” “Search indexes are incredibly important in an era of generative artificial intelligence” – said Kamyl Bazbaz, Senior Vice President of Public Relations at DuckDuckGo.
Regardless of the outcome of the antitrust case, the ongoing changes in the search market underscore how important it is so that publishers can decide their own fate and not become overly dependent on any one technology platform, including Google,” said Joe Ragazzo, publisher of the news site Talking Points Memo. “We believe that you have to build real relationships with your readers,” Ragazzo said, “and that’s how you create a publication that will endure across time.”
