Dropbox built Magic Pocket and took the files home

Dropbox did not “leave the cloud” in the 37signals sense. They moved the majority of file content off Amazon S3 onto a system they designed, in data-centre space they operated, and they said so in 2016–2017 with unusual specificity. They also said they kept AWS where it still paid.

No dollar saving is quoted below that Dropbox did not publish. They did not put a net-present-value figure on Magic Pocket in these sources.

Hybrid, then a pocket

Akhil Gupta’s March 2016 Dropbox Tech post is the primary construction account. Dropbox stores two kinds of data: file content and metadata about files and users. Metadata and web servers had long lived in data centres Dropbox managed. File content sat on Amazon S3. They call themselves an early S3 adopter; Gupta writes that they could not have grown as fast without a service like AWS, and that Amazon “continues to be an invaluable partner.”

Scale, as they published it then: 500 million signups, 500 petabytes of user data (about 40 PB when Gupta joined in 2012). WIRED, the same week, frames the first eight years the same way: billions of files on Amazon’s machines, not Dropbox’s.

Magic Pocket is the in-house store for file content. Gupta: two and a half years on, they were storing and serving over 90% of users’ data on custom-built infrastructure. WIRED repeats the 90% figure and the two-and-a-half-year build.

Why they built it

Gupta gives two reasons, not a press-kit dozen.

1. Performance. “One of our key product differentiators is performance.” Bringing storage in-house let them customise the stack end-to-end for their workload.
2. Unit economics. Their block-storage use case is unusual at their scale. Custom hardware and software, they wrote, produce better unit economics than renting a general store.

WIRED quotes VP of engineering Aditya Agarwal on the same economic point: cloud providers are not a charity; “there is some margin somewhere.” COO Dennis Woodside: the company gets “substantial economic value” running its own operation. Those are qualitative. They are not a published dollar delta.

TechCrunch (2017), after the move had settled: Dan Williams, then head of production engineering, said the decision was about quality, control, and management — third parties can be solid; they believed they could be equal or better because they know the system. A spokesperson told TechCrunch that cost is always considered, but the goal of the network expansion was performance, reliability, flexibility, and control, “which we have succeeded in doing.”

What they actually shipped

Gupta’s timeline:

• Prototype work; Magic Pocket a major initiative in summer 2013.
• Dark launch, August 2014 — mirroring between two regional locations; extra backups kept another six months.
• 27 February 2015 — first time Dropbox stored and served user files exclusively in-house.
• BASE Jump, from 30 April 2015 — install servers in three regional locations fast enough to match the data flow; peak transfer “over half a terabit per second.”
• Target: 90% of data from in-house infra by 30 October 2015. They say they hit it 7 October 2015, without major service disruption or data loss.

Design claims from that post: encrypted at rest; designed for annual data durability over 99.9999999999% and availability over 99.99%. James Cowling’s follow-up (“Inside the Magic Pocket”) describes an immutable block store: files split into blocks up to 4 MB, encrypted and compressed, named by hash, aggregated into ~1 GB buckets, replicated then erasure-coded, stored independently in at least two US zones (west / central / east). Cells of around 50 PB raw; OSDs they describe as over a petabyte per machine, over 8 PB per rack, on spinning disks, SSDs reserved for databases and caches.

TechCrunch adds the human scale they were willing to state: about 1,500 employees, “just around a dozen” on the infrastructure team; three US data centres plus a backbone they compared to something you might find at Google or Facebook. They still used AWS for some workloads.

They did not empty AWS

Gupta, 2016: they will “continue to invest in our own infrastructure as well as partner with Amazon where it makes sense for our users, particularly globally.” Later that year they planned to store data in Germany on AWS for European business customers who requested it.

WIRED: Dropbox “continues to use the Amazon cloud in Europe — just because the business is growing in a less predictable way in Europe.”

That is a hybrid on purpose: steady, understood bytes on owned metal; geographic and demand uncertainty left on a rented region. It is the opposite of a slogan.

Custody, not romance

Magic Pocket is still someone else’s cloud if you are a Dropbox user. The useful fact for an institution that is the data controller is the split Gupta drew: they would not leave the bulk of file content on a general-purpose object store once they could design for their durability, locality, and cost curve.

OpenXperiments is not Dropbox and does not run exabyte cells. The pattern we take is the one they documented: put the crown-jewel bytes where you can change the hardware, the encoding, and the failure domain — and be honest about the slice you still rent. Customer data on our projects is meant to stay in the customer data centre. We are not attaching their 90% figure to anyone we work with.