Gemini is behaving exactly as instructed. Asking it to draw different groups of people (e.g. "Vikings" or "NHL players") is the base case, not an edge case. The questions are all about how it got greenlit by a $1.8T market cap company despite this incredibly predictable behavior.

— Nate Silver (@NateSilver538) February 23, 2024

</div>

In fact, I think there is a precedent for Gemini; like many comparison points for modern-day Google, it comes from Microsoft.

Microsoft and The Curse of Culture

From Neowin, in 2010:¹

Microsoft workers celebrated the release to manufacturing of Windows Phone 7 by parading through their Redmond campus on Friday with iPhone and BlackBerry hearses. Employees dressed up in fancy dress and also modified cars to include Windows Phone branding. Aside from the crazy outfits the workers made fake hearses for giant BlackBerry and iPhone devices. Employees cheekily claimed they had buried the competition with Windows Phone 7.

This was, to be clear, insane. I wrote about the episode in 2013’s The Curse of Culture; it’s been eight years, so I hope you’ll allow me a particularly long excerpt:

As with most such things, culture is one of a company’s most powerful assets right until it isn’t: the same underlying assumptions that permit an organization to scale massively constrain the ability of that same organization to change direction. More distressingly, culture prevents organizations from even knowing they need to do so. From Edgar Schein’s Organizational Culture and Leadership:

Basic assumptions, like theories-in-use, tend to be nonconfrontable and nondebatable, and hence are extremely difficult to change. To learn something new in this realm requires us to resurrect, reexamine, and possibly change some of the more stable portions of our cognitive structure…Such learning is intrinsically difficult because the reexamination of basic assumptions temporarily destabilizes our cognitive and interpersonal world, releasing large quantities of basic anxiety. Rather than tolerating such anxiety levels, we tend to want to perceive the events around us as congruent with our assumptions, even if that means distorting, denying, projecting, or in other ways falsifying to ourselves what may be going on around us. It is in this psychological process that culture has its ultimate power.

Probably the canonical example of this mindset was Microsoft after the launch of the iPhone. It’s hard to remember now, but no company today comes close to matching the stranglehold Microsoft had on the computing industry from 1985 to 2005 or so. The company had audacious goals — “A computer on every desk and in every home, running Microsoft software” — which it accomplished and then surpassed: the company owned enterprise back offices as well. This unprecedented success changed that goal — originally an espoused belief — into an unquestioned assumption that of course all computers should be Microsoft-powered. Given this, the real shock would have been then-CEO Steve Ballmer not laughing at the iPhone.

A year-and-a-half later, Microsoft realized that Windows Mobile, their current phone OS, was not competitive with the iPhone and work began on what became Windows Phone. Still, unacknowledged cultural assumptions remained: one, that Microsoft had the time to bring to bear its unmatched resources to make something that might be worse at the beginning but inevitably superior over time, and two, that the company could leverage Windows’ dominance and their Office business. Both assumptions had become cemented in Microsoft’s victory in the browser wars and their slow-motion takeover of corporate data centers; in truth, though, Microsofts’ mobile efforts were already doomed, and nearly everyone realized it before Windows Phone even launched with a funeral for the iPhone.

Steve Ballmer never figured it out; his last acts were to reorganize the company around a “One Microsoft” strategy centered on Windows, and to buy Nokia to prop up Windows Phone. It fell to Satya Nadella, his successor, to change the culture, and it’s why the fact his first public event was to announce Office for iPad was so critical. I wrote at the time:

This is the power CEOs have. They cannot do all the work, and they cannot impact industry trends beyond their control. But they can choose whether or not to accept reality, and in so doing, impact the worldview of all those they lead.

Microsoft under Nadella’s leadership has, over the last three years, undergone a tremendous transformation, embracing its destiny as a device-agnostic service provider; still, it is fighting the headwinds of Amazon’s cloud, open source tooling, and the fact that mobile users had six years to get used to a world without Microsoft software. How much stronger might the company have been had it faced reality in 2007, but the culture made that impossible.

Google is not in nearly as bad of shape as Microsoft was when it held that funeral. The company’s revenue and profits are as high as ever, and the release of Gemini 1.5 in particular demonstrated how well-placed the company is for the AI era: the company not only has leading research, it also has unmatched infrastructure that enables entirely new and valuable use cases. That, though, makes the Gemini fiasco all the more notable.

Don’t Be Evil

The questions around Google and AI have, to date, been mostly about business model. In last year’s AI and the Big Five I talked about how Kodak invented the digital camera, but didn’t pursue it because of business model reasons, and made the obvious analogy to Google’s seeming inability to ship:

Google has long been a leader in using machine learning to make its search and other consumer-facing products better (and has offered that technology as a service through Google Cloud). Search, though, has always depended on humans as the ultimate arbiter: Google will provide links, but it is the user that decides which one is the correct one by clicking on it. This extended to ads: Google’s offering was revolutionary because instead of charging advertisers for impressions — the value of which was very difficult to ascertain, particularly 20 years ago — it charged for clicks; the very people the advertisers were trying to reach would decide whether their ads were good enough…

That, though, ought only increase the concern for Google’s management that generative AI may, in the specific context of search, represent a disruptive innovation instead of a sustaining one. Disruptive innovation is, at least in the beginning, not as good as what already exists; that’s why it is easily dismissed by managers who can avoid thinking about the business model challenges by (correctly!) telling themselves that their current product is better. The problem, of course, is that the disruptive product gets better, even as the incumbent’s product becomes ever more bloated and hard to use — and that certainly sounds a lot like Google Search’s current trajectory.

Google has started shipping, and again, Gemini 1.5 is an incredible breakthrough; the controversy over Gemini, though, is a reminder that culture can restrict success as well. Google has its own unofficial motto — “Don’t Be Evil” — that founder Larry Page explained in the company’s S-1:

Don’t be evil. We believe strongly that in the long term, we will be better served — as shareholders and in all other ways — by a company that does good things for the world even if we forgo some short term gains. This is an important aspect of our culture and is broadly shared within the company.

Google users trust our systems to help them with important decisions: medical, financial and many others. Our search results are the best we know how to produce. They are unbiased and objective, and we do not accept payment for them or for inclusion or more frequent updating. We also display advertising, which we work hard to make relevant, and we label it clearly. This is similar to a newspaper, where the advertisements are clear and the articles are not influenced by the advertisers’ payments. We believe it is important for everyone to have access to the best information and research, not only to the information people pay for you to see.

Google has by-and-large held to that promise, at least as defined by Page: the company does not sell search result placement. Of course the company has made ads look more and more like organic results, and crammed ever more into the search results page, and squeezed more and more verticals, but while there are always whispers about what is or isn’t included in search, or the decisions made by the algorithm, most people still trust the product, and use it countless times every day.

One does wonder, though, if the sanctity of search felt limiting to some inside of Google. In 2018 a video leaked of an all-hands meeting after the 2016 election where Google executives expressed dismay over the results; the footage was damaging enough that Google felt compelled to issue a statement:

At a regularly scheduled all hands meeting, some Google employees and executives expressed their own personal views in the aftermath of a long and divisive election season. For over 20 years, everyone at Google has been able to freely express their opinions at these meetings. Nothing was said at that meeting, or any other meeting, to suggest that any political bias ever influences the way we build or operate our products. To the contrary, our products are built for everyone, and we design them with extraordinary care to be a trustworthy source of information for everyone, without regard to political viewpoint.

Perhaps this seemed to some employees to be an outdated view of the world; I’m reminded of that quote from Angela Y Davis: “In a racist society it is not enough to be non-racist, we must be anti-racist.” In this view calls for color-blindness in terms of opportunity are insufficient; the only acceptable outcome is one in which outcomes are equal as well. The equivalent in the case of Google would be that it is not enough to not be evil; one must be “anti-evil” as well.

The end result is that just as Microsoft could, shielded by years of a Windows monopoly, delude themselves into thinking they had an iPhone killer, Google could, shielded by years of a search monopoly, delude themselves into thinking they had not just the right but the obligation to tell users what they ought to believe.

After Gemini

As I noted in the excerpt, I very much try to avoid politics on Stratechery; I want to talk about business models and societal impact, and while that has political implications, it doesn’t need to be partisan (for example, I think this piece about the 2016 election holds up very well, and isn’t partisan in the slightest). AI, though, is increasingly giving all of us no choice in the matter.

To that end, my Article last fall about the Biden executive order, Attenuating Innovation, was clearly incomplete: not only must we keep in mind the potential benefits of AI — which are massive — but it is clearly essential that we allow open source models to flourish as well. It is Google or OpenAI’s prerogative to train their models to have whatever viewpoint they want; any meaningful conception of freedom should make space for an open market of alternatives, and that means open source.

Secondly, it behooves me, and everyone else in tech, to write Articles like the one you are reading; “the politics of the median member of the San Francisco Board of Supervisors” has had by far the loudest voice in tech because most people just want to build cool new things, or write about them, without being fired or yelled at on social media. This does, though, give the perception that tech is out of touch, or actively authoritarian; I don’t think that’s true, but those of us who don’t want to tell everyone else what to think, do, paradoxically, need to say so.

The biggest question of all, though, is Google. Again, this is a company that should dominate AI, thanks to their research and their infrastructure. The biggest obstacle, though, above and beyond business model, is clearly culture. To that end, the nicest thing you can say about Google’s management is to assume that they, like me and everyone else, just want to build products and not be yelled at; that, though, is not leadership. Schein writes:

When we examine culture and leadership closely, we see that they are two sides of the same coin; neither can really be understood by itself. On the one hand, cultural norms define how a given nation or organizations will define leadership — who will get promoted, who will get the attention of followers. On the other hand, it can be argued that the only thing of real importance that leaders do is to create and manage culture; that the unique talent of leaders is their ability to understand and work with culture; and that it is an ultimate act of leadership to destroy culture when it is viewed as dysfunctional.

That is exactly what Nadella did at Microsoft. I recounted in The End of Windows how Nadella changed the company’s relationship to Windows, unlocking the astronomical growth that has happened under his watch, including the company’s position in AI.

Google, quite clearly, needs a similar transformation: the point of the company ought not be to tell users what to think, but to help them make important decisions, as Page once promised. That means, first and foremost, excising the company of employees attracted to Google’s power and its potential to help them execute their political program, and return decision-making to those who actually want to make a good product. That, by extension, must mean removing those who let the former run amok, up to and including CEO Sundar Pichai. The stakes, for Google specifically and society broadly, are too high to simply keep one’s head down and hope that the San Francisco Board of Supervisors magically comes to its senses.

1. Image credit Carl J on Flickr ↩