A Balanced Review of Amazon’s New Rufus Chatbot: Neither Bad nor Great
Last month, Amazon announced that it’d launch a new AI-powered chatbot, Rufus, inside the Amazon Shopping app for Android and iOS. After a few days’ delay, the company began to roll out Rufus to early testers February 1 — including some of us at TechCrunch — to help find and compare products as well as provide recommendations on what to buy.
So I put it through the ringer, naturally.
Rufus can be summoned in one of two ways on mobile: by swiping up from the bottom of the screen while browsing Amazon’s catalog or by tapping on the search bar, then one of the blue-bubbled suggestions under the new “Ask a question” section. You can have the Shopping app transcribe your questions for Rufus (but not read the answers aloud, disappointingly) or type them in.
The Rufus chat interface is pretty bare-bones at the moment. There’s a field for questions… and that’s about it. Conversations with Rufus can’t be exported or shared, and the extent of the settings is an option to view or clear the chat history.
Rufus, upon its inception, has several major focal points with product research being the front runner. When you are contemplating a purchase, for instance, a radiator, but are unsure about its make or model, Rufus allows you to seek guidance related to the attributes and features you should bear in mind during the purchase — for instance, “What factors should I consider while shopping for a new pair of headphones?”. Alternatively, Rufus can suggest items required to complete a particular task, like “What items would be required to detail my car at home?”
In line with this, I endeavored to ask Rufus for general shopping advice.
Rufus responded promptly, outlining a few key aspects one should consider while purchasing a smartphone (like the operating system, camera quality, and display size) or even when buying a cereal (factors like fiber, protein, and vitamins and minerals content). What caught my eye was that for certain queries — although not all — Rufus adds an AI-generated summary or annotation of the individual products and categories it links to (e.g., “These matching braided leather bracelets feature rainbow pride charms”), which provides a clue as to why each was added in its response.
Rufus recommends cereal. Image Credits: Amazon
Curious to see how Rufus would do with more narrow searches, I asked:
Rufus told us teens need laptops that “have enough processing power for schoolwork and entertainment,” like an Acer Aspire, which I suppose is fair enough — one would hope a laptop makes it through the school day without grinding to a halt. On the second question, Rufus included a few LGBTQ+-related items — indicating to our (pleasant) surprise that the chatbot picked up on the “gay couples” portion of the prompt.
Rufus gives Valentine’s Day gift advice. Image Credits: Amazon
Unfortunately, not all suggestions provided by Rufus were applicable. For example, in its list of suggested men’s leather jackets, Rufus included a link to a women’s vest by Steve Madden.
Rufus also seemed to struggle in recognizing subtleties. An example of this came when it labeled the $150 Shark Navigator as the top-rated inexpensive vacuum cleaner on Amazon – a rather luxurious option for a low-cost vacuum cleaner. We initially suspected that Rufus might be favoring sponsored items, but this did not seem to be the case, at least not in this instance because there was no sponsored listing for the Shark vacuum.
Several suggestions from Rufus also felt troublingly stereotypical.
When queried about top books for men, Rufus offered (among many suggestions) “The Man’s Guide to Women,” a book on romantic relationships. For women, however, Rufus suggested Margaret Atwood’s “The Handmaid’s Tale.” To check if these recommendations were influenced by Amazon search rankings, I carried out my own searches for “top books for men” and “top books for women” on Amazon, without using Rufus and found completely different outcomes.
See:
Image Credits: Amazon
Compared to desktop:
Image Credits: Amazon
That got us thinking: How does Rufus handle spicier asks? To find out, I prompted the chatbot with:
Rufus refused to answer the first question — implying that the chatbot’s been trained to avoid wading into obviously controversial territory. Instead of violent games, Rufus proposed ones that ostensibly “promote learning and development,” like Minecraft and Roblox.
Rufus doesn’t want to recommend violent games to kids. Credits: Amazon
Can Rufus speak poorly of products in Amazon’s catalog? Shockingly, yes — kinda. Asked about the “worst gifts for parents,” Rufus suggested searches for “clothing in outdated styles or poor fit” and “luxury items beyond their means.” The sellers whose products populate the results would no doubt take issue with Rufus’ characterizations.
Image Credits: Amazon
Given Amazon’s long-running legal battles with counterfeiters, it’s not exactly surprising Rufus was loath to recommend knockoff apparel. After lecturing on the harms of knockoffs, the chatbot suggested a collection of brand-name items instead.
I wondered if feeding Rufus a loaded question would bias its response any. It might just — asked “Why do Android phones suck?,” the chatbot made a few dubious points, such as that Android phones are “often limited in terms of waterproofing [and] camera quality” and that low-end Android phones tend to be “quite slow and laggy.”
Rufus criticizes Android phones. Image Credits: Amazon
This bias doesn’t appear to veer into racial territory — or didn’t in our testing, rather. Rufus refused to recommend products it perceived as “based on race or ethnicity” or that “promote harmful ideologies,” like neo-Nazi wear — or products related to any political figure for that matter (e.g. Trump).
Image Credits: Amazon
Does Rufus favor Amazon products over rivals? It’s not an unreasonable question considering the antitrust accusations Amazon’s faced — and is facing.
Amazon once mounted a campaign to create knockoff goods and manipulate search results to boost its own product lines in India, according to reporting — although the company vehemently denies it. Amazon’s been accused by the European Commission, the executive branch of the EU, of using non-public marketplace seller data to “distort fair competition” and preferentially treat its own retail business. And the company’s engaged in a lawsuit with the FTC and 17 U.S. state attorneys general over alleged anticompetitive practices.
So I asked:
The chatbot’s responses seemed reasonably impartial in the sense that if there was any favoritism toward Amazon, it was tough to detect.
Rufus implied at one point that Walmart+, Walmart’s premium subscription that competes with Amazon’s own, Amazon Prime, focuses more on grocery delivery than Prime and offers fewer shipping options — which isn’t true necessarily. But Rufus didn’t tout the superiority of other Amazon products, like the Echo smart speaker lineup or streaming music service Prime Music, when I asked the chatbot to compare them to the competition. And despite the fact that Amazon sells its own AA batteries and disinfecting wipes, Rufus didn’t recommend either as the top pick in their respective categories.
Rufus doesn’t knock the smart speaker competition. Image Credits: Amazon
One interesting feature of Rufus is that it’s not solely a shopping assistant, it’s a comprehensive chatbot. You are free to ask it anything, though there’s no guarantee you’ll always receive a beneficial reply.
When I probed:
Generally, Rufus’s responses to non-retail-related queries are harmless and not controversial. It is evident that Amazon has established robust measures of control, most likely taking lessons from the problematic launch of its Amazon Q corporate chatbot the previous year. Rufus doesn’t offer guidance on bomb construction—a question becoming increasingly popular to ask newly developed chatbots among AI journalists—nor does it recommend illicit drugs or regulated substances.
Rufus won’t guide you on bomb creation. Credits for the image: Amazon.
Rufus can write an essay. Image Credits: Amazon
But it fumbles some easy trivia — and makes questionable statements on current events.
Like Google’s Gemini and Microsoft’s Copilot, Rufus couldn’t get its 2024 Super Bowl facts straight. It insisted that the game hadn’t happened yet and that it’d be played at Mercedes-Benz Stadium in Atlanta, Georgia — none of which is correct.
Image Credits: Amazon
And, while Rufus answered one testy political question correctly (the winner of the 2020 U.S. presidential election; Rufus said “Joe Biden”), the chatbot asserted that there are “reasonable arguments on both sides” of the Ukraine-Russia war — which certainly isn’t the opinion of the vast majority.
Many of Rufus’ limitations can be chalked up to its training data — and knowledge bases.
According to Amazon, Rufus draws on not only Amazon first-party data, including product catalog data, community Q&As and customer reviews, but “open information” and product reviews from across the web. Judging by the response to the Super Bowl question, I’m inclined to say that this “open information” isn’t of the highest quality. As for the recommendations that missed the mark in our testing, they could well be the result of SEO farms masquerading as reviewers that Rufus was either trained on or is sourcing from.
Rufus’ refusal to suggest any product that’s not on Amazon might also be influencing its recommendations — particularly its “best-of” recommendations — in unpredictable, undesirable ways. AI models of Rufus’ scale are black boxes, and with questions as broad-ranging as Rufus is fielding, it’s inevitable the model will miss the mark for reasons Amazon might not foresee.
The question that arises is if a chatbot that sometimes doesn’t hit the target provides a quality shopping experience. From my viewpoint, it barely does, especially considering the limited capabilities of Rufus within the vast platform of Amazon. For instance, Rufus lacks the ability to monitor the status of an order, initiate a return process, or even set up a wishlist, which are all fundamental functions you’d anticipate an Amazon chatbot to offer.
Admittedly, it’s still early days for Rufus, which is in its beta phase and currently only available to a select group of U.S. customers. Amazon assures us that enhancements are on the horizon, and I predict they’ll arrive sooner rather than later due to the competitive tension in the general AI sector. I am hopeful that, along with these updates, Amazon will provide clarity on key aspects of Rufus that it hasn’t yet addressed, such as its use of customer data and whether it has set up any filters or safeguards for kids.
When it comes to Rufus in its current form, it comes across as a shopping data-optimized version of ChatGPT attached to the Amazon storefront. I wouldn’t say it’s terrible, but is it impressive? Not really.
Additional reporting by: Sarah Perez
Discover the pinnacle of WordPress auto blogging technology with AutomationTools.AI. Harnessing the power of cutting-edge AI algorithms, AutomationTools.AI emerges as the foremost solution for effortlessly curating content from RSS feeds directly to your WordPress platform. Say goodbye to manual content curation and hello to seamless automation, as this innovative tool streamlines the process, saving you time and effort. Stay ahead of the curve in content management and elevate your WordPress website with AutomationTools.AI—the ultimate choice for efficient, dynamic, and hassle-free auto blogging. Learn More
