Storage
Three kinds of storage for three ways of reaching data
Object, block and file storage differ in how data is reached: by name over the web, as a disk, or as a shared folder.
Cloud storage comes in three kinds, and what separates them is how you reach the data. Object storage holds whole files in a bucket, and you fetch each one by its name over the web. Block storage is a disk. It is attached to one machine, and that machine’s operating system uses it like any hard drive. File storage is a shared folder. Many machines open it at once, the way an office shares a network drive. Each provider has one service of each kind. Once you know how a piece of data will be reached, the choice is nearly made.
Object storage
Whole files under names, with no limit on how many
An object is data plus a key plus metadata, kept in a bucket and reached through a web address.
Object storage is the workhorse, so start there. A bucket is a flat list of names. It looks like folders, but the slashes are just characters in the name. The list can hold four objects or four billion; you never set a size. Each entry is an object: the name, called a key; the data itself, stored whole; some metadata, such as its type and your own labels; and, if you ask, every earlier version. And each object has a web address. Anything you permit can fetch it from anywhere. The one thing you cannot do is change part of an object. You replace the whole thing.
Object storage · Going deeper
Buckets, containers and the switches that matter
S3 buckets, Azure containers in a storage account and Cloud Storage buckets do the same job. Versioning, replication, lifecycle rules and blocked public access are the settings to know.
Durability and availability
Eleven nines: every object is copied across three zones
Durability is the chance data survives; availability is the chance you can reach it now. Object storage offers 11 nines of the first.
How safe is data in a bucket? When you store one object, the service writes copies onto several disks in each of three zones, and only then tells you it is saved. If a disk dies, the copy is rebuilt from the others. The result is a durability of ninety-nine point nine, nine more nines, percent. In plain terms: store ten million objects and you might lose one in ten thousand years. Availability is a separate number. It says how often you can reach the data: here, all but about fifty-three minutes a year. During those minutes the data is unreachable. It is not gone.
Durability and availability · Going deeper
Azure’s four redundancy options, drawn as copies
LRS keeps three copies in one data centre, ZRS one in each of three zones, GRS and GZRS add three more in a paired region.
Storage classes
Pay less to store, and more to read
A storage class trades a lower price per gigabyte for slower or costlier retrieval. Pick by how often data is read.
Not all data deserves the same price. Standard is for data in daily use: about two cents per gigabyte a month, and instant to read. Infrequent access costs about half, with a small fee each time you read. It suits data touched perhaps once a month. Glacier Instant Retrieval drops the price again and keeps the instant read, at a higher fee. Glacier Flexible Retrieval is cheaper still, but you wait minutes or hours for your data. And Deep Archive costs a tenth of a cent, one twenty-third of Standard, with a wait of up to twelve hours. All five are equally safe. You are choosing how quickly, and how cheaply, you can get the data back.
Storage classes · Going deeper · 1 of 2
The same ladder on Azure and Google Cloud
Hot, Cool, Cold and Archive on Azure; Standard, Nearline, Coldline and Archive on Google Cloud. Intelligent-Tiering and Autoclass choose for you.
Storage classes · Going deeper · 2 of 2
Lifecycle rules: data cools as it ages
A lifecycle rule moves objects to colder classes on a schedule and deletes them at the end. Here it cuts seven years of storage from about $1,930 to about $150.
Block storage
A disk attached to one machine
A volume is a virtual disk of fixed size for one machine in the same zone, backed up with snapshots.
A virtual machine needs a disk, and block storage is that disk. It is called a volume. You choose a size, here a hundred gigabytes, and attach it to one machine in the same zone. The operating system formats it and uses it exactly like a physical drive. It can change any small piece of any file in under a millisecond, which is what databases need. Notice the billing. Thirty gigabytes are in use, but you pay for all hundred, because that is the size you reserved. To protect it, you take snapshots: copies of the whole disk at one moment, kept in object storage, from which a new disk can be made anywhere.
Block storage · Going deeper · 1 of 2
Kinds of disk: general purpose, fast, cheap and temporary
Disks are sold by IOPS. General purpose SSDs suit most work; local disks are fastest and are wiped when the machine stops.
Block storage · Going deeper · 2 of 2
Snapshots copy only what changed
After the first full copy, each snapshot stores only changed blocks, so days of history cost little.
File storage
One folder tree, mounted by many machines at once
A managed file system is a shared network folder with real paths and permissions, open to many machines across zones.
The third kind exists because a disk belongs to one machine, and sometimes many machines need the same files. File storage is a folder tree that lives on the network. It has paths, folders and permissions, like the shared drive in an office. Machines in every zone of the region mount it, and all of them can read and write the same files at the same moment. It grows as you add files and shrinks as you remove them, and you pay for what is in it. It is the most expensive kind per gigabyte. Use it when a program expects a shared folder and cannot be changed.
Choosing storage
How the data is reached decides, and the price agrees
Object storage by default, block storage for disks and databases, file storage only when software needs a shared folder.
Put the three side by side. Object storage is reached by name through a web interface, by anything, from anywhere. You pay only for what is stored: about twenty-three dollars for a thousand gigabytes. Block storage is reached by one machine’s operating system as a disk. You pay for the size you reserved: about eighty dollars. File storage is reached as a folder by many machines at once: about three hundred dollars. So the rule is simple. Put everything you can in object storage. Use block storage for operating systems and databases. Keep file storage for the software that demands it.
Data transfer
Moving a mountain of data: sometimes a lorry beats the wire
Large migrations use shipped storage devices; smaller or changing data uses online transfer services.
One practical problem remains: getting existing data into the cloud. Say you have a hundred terabytes. Over a hundred megabit line, running flat out day and night, that takes ninety-three days. Over a gigabit line, nine days. Only at ten gigabits does it fall under a day, and few offices have that to spare. So providers offer something old-fashioned. They post you a rugged storage device. You fill it, send it back, and they load it into your bucket. About a week, door to door, however slow your internet is.
Data transfer · Going deeper
Backups, and keeping storage on site in step with the cloud
Storage in the cloud is durable, and a durable mistake is still a mistake. Backups protect against deletion, corruption and ransomware; hybrid services connect what you still keep on site.
Storage in the cloud is durable, and a durable mistake is still a mistake. Backups protect against deletion, corruption and ransomware; hybrid services connect what you still keep on site. The 3-2-1 rule. One place to schedule backups. AWS Backup sets schedules and retention for disks, databases and file systems across an account. Azure Backup does the same from a Recovery Services vault, including on-premises servers. Google Cloud Backup and DR covers virtual machines, databases and file systems. Replication is not a backup. A replica copies your mistakes within seconds. A backup is a copy from an earlier time that a mistake cannot reach: versioned, and ideally locked against deletion (S3 Object Lock, Azure immutable storage, Google bucket lock). Hybrid storage. AWS Storage Gateway presents cloud storage inside your building as an ordinary file share or disk, caching recent data locally. Azure File Sync keeps Windows file servers on site in step with an Azure file share. Online transfer AWS DataSync, AzCopy and Google’s Storage Transfer Service copy over the network on a schedule and verify what they copied.
The 3-2-1 rule
One place to schedule backups
- AWS Backup sets schedules and retention for disks, databases and file systems across an account.
- Azure Backup does the same from a Recovery Services vault, including on-premises servers.
- Google Cloud Backup and DR covers virtual machines, databases and file systems.
Hybrid storage
- AWS Storage Gateway presents cloud storage inside your building as an ordinary file share or disk, caching recent data locally.
- Azure File Sync keeps Windows file servers on site in step with an Azure file share.
- Online transfer AWS DataSync, AzCopy and Google’s Storage Transfer Service copy over the network on a schedule and verify what they copied.