It was in August 2009, two months after the release of the iPhone 3GS, that the barbarians crashed the gates; from The Online Photographer:

The leading photo sharing site, flickr.com, charts the popularity of the cameras used by its membership. Recently the Apple iPhone has jumped into a virtual tie for first place with the Canon XTi. Furthermore, flickr states on its “Camera Finder” page that it can only detect the camera used about 2/3rds of the time, and that, therefore, cameraphones are under-represented on the graphs. Yikes.

The iPhone would go on to not only be the number one camera on Flickr, but in a far more compelling measure of its impact, create the conditions for Instagram, the photo-sharing network that eclipsed Flickr like Google once eclipsed Yahoo (Yahoo, of course, owned Flickr). Instagram itself was soon acquired by Facebook, which itself had benefited tremendously from the iPhone camera; having a good and ever-improving camera with you everywhere, paired with constant connectivity, transformed photos from memorials for special occasions to, in the words of Snap CEO Evan Spiegel, “part of the communication fabric of our daily lives.”

How is it, then, that a company like Facebook, which is mostly used on mobile — i.e. Android or iOS — made 20.3 million reports of Child Sexual Abuse Material (CSAM) in 2020, while Apple made only 265?¹ After all, there are almost certainly more photos on smartphones than there are on social networks — the former is in large part a superset of the latter.

18 USC § 2258A

U.S. Code Title 18 Part 1 Chapter 110 Section 2258A lays out how companies must handle CSAM (complete text here):

(a)Duty To Report.—

(1)In general.—

(A) Duty.—In order to reduce the proliferation of online child sexual exploitation and to prevent the online sexual exploitation of children, a provider—

(i) shall, as soon as reasonably possible after obtaining actual knowledge of any [CSAM], take the actions described in subparagraph (B); and
(ii) may, after obtaining actual knowledge of any facts or circumstances [suggesting imminent child abuse], take the actions described in subparagraph (B).

(B)I Actions described.—The actions described in this subparagraph are—

(i) providing to the CyberTipline of NCMEC [National Center for Missing & Exploited Children], or any successor to the CyberTipline operated by NCMEC, the mailing address, telephone number, facsimile number, electronic mailing address of, and individual point of contact for, such provider; and
(ii) making a report of such facts or circumstances to the CyberTipline, or any successor to the CyberTipline operated by NCMEC.

There is no escaping this responsibility when and if CSAM is discovered:

(e)Failure To Report.—A provider that knowingly and willfully fails to make a report required under subsection (a)(1) shall be fined—

(1) in the case of an initial knowing and willful failure to make a report, not more than $150,000; and
(2) in the case of any second or subsequent knowing and willful failure to make a report, not more than $300,000.

What is not required is that companies actively seek out CSAM on their services:

(f)Protection of Privacy.—Nothing in this section shall be construed to require a provider to—

(1) monitor any user, subscriber, or customer of that provider;
(2) monitor the content of any communication of any person described in paragraph (1); or
(3) affirmatively search, screen, or scan for facts or circumstances described in sections (a) and (b).

These two provisions get at why Facebook and Apple’s reported numbers have historically been so different: it’s not because there is somehow more CSAM on Facebook than exists on Apple devices, but rather that Facebook is scanning all of the images sent to and over its service, while Apple is not looking at what is in your phone, or on their cloud. From there the numbers make much more sense: Facebook is reporting what it finds, while Apple is, as the title of Section (3) suggests, protecting privacy and simply not looking at images at all.

Apple Protects Children

Last week Apple put up a special page on their website entitled Expanded Protections for Children:

At Apple, our goal is to create technology that empowers people and enriches their lives — while helping them stay safe. We want to help protect children from predators who use communication tools to recruit and exploit them, and limit the spread of Child Sexual Abuse Material (CSAM).

Apple is introducing new child safety features in three areas, developed in collaboration with child safety experts. First, new communication tools will enable parents to play a more informed role in helping their children navigate communication online. The Messages app will use on-device machine learning to warn about sensitive content, while keeping private communications unreadable by Apple.

Next, iOS and iPadOS will use new applications of cryptography to help limit the spread of CSAM online, while designing for user privacy. CSAM detection will help Apple provide valuable information to law enforcement on collections of CSAM in iCloud Photos.

Finally, updates to Siri and Search provide parents and children expanded information and help if they encounter unsafe situations. Siri and Search will also intervene when users try to search for CSAM-related topics.

John Gruber at Daring Fireball has a good overview of what are in fact three very different initiatives; what unites, them, though, and continues to differentiate Apple’s approach from Facebook’s, is that Apple is scanning content on your device, while Facebook is doing it in the cloud. Apple emphasized repeatedly that this ensured that Apple does not get access to your content. From the “Communications Safety in Messages” section:

Messages uses on-device machine learning to analyze image attachments and determine if a photo is sexually explicit. The feature is designed so that Apple does not get access to the messages.

From the “CSAM Detection” section:

Apple’s method of detecting known CSAM is designed with user privacy in mind. Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child safety organizations…This innovative new technology allows Apple to provide valuable and actionable information to NCMEC and law enforcement regarding the proliferation of known CSAM. And it does so while providing significant privacy benefits over existing techniques since Apple only learns about users’ photos if they have a collection of known CSAM in their iCloud Photos account. Even in these cases, Apple only learns about images that match known CSAM.

There are three ways to think about Apple’s approach, both in isolation and relative to a service like Facebook:² the idealized outcome, the worst case outcome, and the likely driver.

Capability Versus Policy

Apple’s idealized outcome solves a lot of seemingly intractable problems. On one hand, CSAM is horrific and Apple hasn’t been doing anything about it; on the other hand, the company has a longstanding commitment to ever increasing amounts of encryption, ideally end-to-end. Apple’s system, if it works precisely as designed, preserves both goals: the company can not only keep end-to-end encryption in Messages, but also add it to iCloud Photos (which is not currently encrypted end-to-end), secure in the knowledge that it is doing its part to not only report CSAM but also help parents look after their children. And, from a business perspective, it means that Apple can continue to not make the massive investments that companies like Facebook have in trust-and-safety teams; the algorithm will take care of it.

That, of course, is the rub: Apple controls the algorithm, both in terms of what it looks for, what bugs it may or may not have, and also the inputs, which in the case of CSAM scanning is the database from NCMEC. Apple has certainly worked hard to be a company that users trust, but we already know that that trust doesn’t extend everywhere: Apple has, under Chinese government pressure, put Chinese user iCloud data on state-owned enterprise servers, along with the encryption keys necessary to access it. What happens when China announces its version of the NCMEC, which not only includes the horrific imagery Apple’s system is meant to capture, but also images and memes the government deems illegal?

The fundamental issue — and the first reason why I think Apple made a mistake here — is that there is a meaningful difference between capability and policy. One of the most powerful arguments in Apple’s favor in the 2016 San Bernardino case is that the company didn’t even have the means to break into the iPhone in question, and that to build the capability would open the company up to a multitude of requests that were far less pressing in nature, and weaken the company’s ability to stand up to foreign governments. In this case, though, Apple is building the capability, and the only thing holding the company back is policy.

Then again, Apple’s policy isn’t the only one that matters: both the UK and the EU are moving forward on bills that mandate online service companies proactively look for and report CSAM. Indeed, I wouldn’t be surprised if this were the most important factor behind Apple’s move: the company doesn’t want to give up on end-to-end encryption — and likely wants to expand it — which leaves on-device scanning as the only way to satisfy governments not (just) in China but also the West.

Cloud Versus Device

I think that there is another solution to Apple’s conundrum; what is frustrating from my perspective is that I think the company is already mostly there. Consider the status quo: back in 2020 Reuters reported that Apple decided to not encrypt iCloud backups at the FBI’s request:

Apple Inc. dropped plans to let iPhone users fully encrypt backups of their devices in the company’s iCloud service after the FBI complained that the move would harm investigations, six sources familiar with the matter told Reuters. The tech giant’s reversal, about two years ago, has not previously been reported. It shows how much Apple has been willing to help U.S. law enforcement and intelligence agencies, despite taking a harder line in high-profile legal disputes with the government and casting itself as a defender of its customers’ information.

This has a number of significant implications on Apple’s security claims, and is why earlier this year I ranked iMessage as being less secure than Signal, WhatsApp, Telegram, and Facebook Messenger:

iMessage encrypts messages end-to-end by default; however, if you have iCloud backup turned on, your messages can be accessed by Apple (who has the keys for iCloud backups) and, by extension, law enforcement with a warrant. Unlike WhatsApp, though, this is both on by default and cannot be turned off on a granular basis.

This caveat applies to almost everything on your iPhone: if you give in to the never-ending prompts to sign-in to iCloud and its on-by-default backup solution, your data is accessible to Apple and, by extension, law enforcement with a warrant. I actually think this is reasonable! I wrote this when that Reuters report came out:

Go back to what I said above: determined actors will have access to encryption and facial recognition. Anyone trying to argue whether or not these technologies should exist is not living in reality. It follows then, that we should take care to ensure that good actors have access to these technologies too. That means not making them illegal.

Second, though, legitimate societal concerns about the needs of law enforcement and the radicalizing nature of the Internet should be taken seriously. That means we should think very carefully about making encryption the default…This also splits the difference when it comes to principles: users have agency — they can ensure that everything they do is encrypted — while total privacy is available but not given by default.

I actually think that Apple does an excellent job of striking that balance today. When it comes to the iPhone itself, Apple is the only entity that can make it truly secure; no individual can build their own secure enclave that sits at the root of iPhone security. Therefore, they are right to do so: everyone has access to encryption.

From there it is possible to build a fully secure environment: use only encrypted communications, use encrypted backups to a computer secured by its own hardware-based authentication scheme, etc. Taking the slightly easier route, though — iCloud backups, Facebook messaging, etc. — means some degree of vulnerability that, let’s not forget, is sometimes justifiably leveraged. Law enforcement can get a warrant for those backups or chat logs, just as they can install a wire tap.

Again, this isn’t going to stop determined bad actors, but as I noted, nothing is. The question is what of the rest, those that get swept up by the worst sort of communities, and who commit legitimate crimes: what should their defaults be?

I made a similar argument about Facebook’s plans to encrypt-by-default Facebook Messenger conversations, which I opposed, even as I supported encryption-by-choice: I am not anti-encryption, and am in fact very much against mandated backdoors. Every user should have the capability to lock down their devices and their communications; bad actors surely will. At the same time, it’s fair to argue about defaults and the easiest path for users: I think the iPhone being fundamentally secure and iCloud backups being subject to the law is a reasonable compromise.

Apple’s choices in this case, though, go in the opposite direction: instead of adding CSAM-scanning to iCloud Photos in the cloud that they own-and-operate, Apple is compromising the phone that you and I own-and-operate, without any of us having a say in the matter. Yes, you can turn off iCloud Photos to disable Apple’s scanning, but that is a policy decision; the capability to reach into a user’s phone now exists, and there is nothing an iPhone user can do to get rid of it.

A far better solution to the “Flickr problem” I started with is to recognize that the proper point of comparison is not the iPhone and Facebook, but rather Facebook and iCloud. One’s device ought be one’s property, with all of the expectations of ownership and privacy that entails; cloud services, meanwhile, are the property of their owners as well, with all of the expectations of societal responsibility and law-abiding which that entails. It’s truly disappointing that Apple got so hung up on its particular vision of privacy that it ended up betraying the fulcrum of user control: being able to trust that your device is truly yours.

1. Google made 547k ↩

2. Beyond the fact that it may be illegal  ↩