Archive for the ‘Amazon S3’ Category

Build 4299: change to Amazon S3 backup format

16 January 2012

When a database is backed up automatically to Amazon S3, it is stored as a number of S3 objects. The rules for constructing the object names associated with a given database have changed slightly: they are documented here. Existing backups are unaffected.

New Amazon service agreement

3 July 2007

Amazon have at last revised the service agreement for the use of the Amazon S3 storage service. The new service agreement is a great improvement its predecessor, which we criticised in detail in the S3 in Business article. Using the “backup to S3” features of Cardbox now seems legally as well as technically practicable.

Amazon’s EC2 on-demand computing facility still has unacceptable restrictions placed on its use, but that is a separate service from Amazon S3 and is not relevant to Cardbox users.

S3 in Business: 13 – Conclusion

28 August 2006

[Complete series]

I am writing the first draft of this posting less than 24 hours after the announcement of the public beta release of Amazon EC2 (Elastic Computing Cloud) and it is making me relive the emotions that I and many other developers felt when Amazon S3 was announced. A shared culture is clearly at work. In each case, Amazon create and offer a service that has very few features: but those that it has, it implements consistently, cleanly and robustly. The documentation has the forceful simplicity of a 1970s IBM manual, saying exactly what needs to be said, straightforwardly and unambiguously: it has been written by someone with a mind with the intention that it should be read by someone with a mind. The developer forums are monitored by intelligent people who get to the bottom of the questions that are asked – sometimes deeper than the questioner himself – and give answers uncontaminated by considerations of marketing or public relations. Nothing like that has been seen since the Microsoft Windows 3.1 beta test programme.

Outsiders do not realise how deeply technological development, like scientific research, is shaped and driven by emotion. If you have ever looked at an iPod and wished that you needed one, you will have had an inkling of it. The reason that half of us are in computing at all is that we see computers as things that we can make beautiful things out of: and S3 and EC2 arouse the same emotion.
Some people have criticized S3 and EC2 for being bony, but that is the point of them. You cannot go wrong if you have good, strong bones to build on. If the foundation is right, things just go on getting better. A woman with good bones is six times as beautiful at 60 as she is at 20.

To take one example: when an EC2 computer is shut down, it evaporates. Completely. It is as if it had never been. In particular, any data that it stored are lost. This sounds ridiculous, but it keeps the EC2 architecture simple and beautiful. Anyone building a system on EC2 already has access to data storage of great robustness and infinite capacity: it’s called S3. So each system developer can put together the bones – instantly expandable computing capacity and bulletproof data storage – to make the body he needs. There is no need to contaminate EC2 by building into it assumptions about whether persistent storage is needed, and how much of it is needed, and how it should be administered. Each developer can make the decisions that are right for him and not be constrained by someone else’s idea of what he ought to want.

Conclusion

This series has been about applying the excitement generated by S3 (and now, by EC2) to doing real work in the real world. It has been about reconciling the romantic thrill of “darling, we’re going to have a baby” with the reality of coping with something damp that yells.

For server-based services such as tunesafe.com, S3 is ideal. The user interacts with the web server but all big data transfers are shunted directly into and out of S3. S3 costs money, of course; but it costs so little money that in many cases (such as probably tunesafe.com) it is possible to charge a flat fee because the wholesale data storage and data transfer costs can be bundled up into a single retail offering that still has an attractive price.

For startups that want a million users but don’t want to issue a million invoices a month, the availability of premium-rate storage (“golden buckets”) would be a major enabling technology. The profit from the golden buckets would pay for software development and the maintenance of the necessary servers; Amazon would take their cut; the user would have just one monthly bill to pay; and again, because S3 is so cheap, even a premium-rated version of it would be competitive and affordable. Whole new business models will be possible as a result. The demand is certainly there from developers: we all hope that Amazon will get round to providing this option.

No business plan is complete without some assessment of risk. For a start, Amazon need to find a way of convincingly reassuring us that we are not betting our companies on Amazon’s continuing interest and benevolence. Possible reassurances include the provision of limited guarantees of continuity or the availability of rival services that behave identically to S3. It might even be in Amazon’s interest to help set up such services.

S3 is claimed to be reliable but there are no figures to support this and users of S3 are thus taking on an unquantified risk. Quantifying the risk by publishing data is one way forward. A more interesting approach is to offer insured data storage for a small extra premium. Apart from being potentially extremely profitable, this could catalyse the transformation of the computing industry into one that takes responsibility for its products.

The contract under which S3 is currently offered is entirely unsuited to its purpose. It is not only unacceptable to anyone who plans to depend on S3, it also contradicts everything we know about Amazon’s intentions and corporate philosophy. It needs to be torn up and rewritten from scratch.

Amazon intend to spread their data centres worldwide so that computing capacity (EC2) and storage capacity (S3) are available at equally high speeds all round the world. They must pay serious attention to the legal consequences of storing data in various countries because there is no way for a user to control where, physically, data are stored in the S3 network. There is no point in virtualising away the technical differences between storage technologies if you cannot similarly virtualise away the legal differences between various jurisdictions. It is impossible to expect every Amazon S3 user to ensure that his data comply with the laws of the United States, Britain, Spain, China, and so on. In a world where every country tries to extend its jurisdiction over the whole of the Internet, we need Amazon to work on our behalf to obtain some sort of extraterritorial status for our data.

Let me finish by referring back to the start of this posting. With S3 and EC2, Amazon are revitalising computing by bringing it back to its roots. They have started a revolution. If some of my criticisms have sounded harsh, it is because I am applying some of Amazon’s own perfectionism to the services they have created and because I think that they deserve a large share of the profits from the revolution that they have started.

S3 in Business: 12 – A miscellaneous interlude

26 August 2006

[Complete series]

Every good filing system has a section called “M for miscellaneous”. So does this blog series.

Anti-Internet software

It is possible for an attacker to provoke Norton Internet Security into blocking all access to Amazon S3 from your computer. So if a Tunesafe user visits a site that contains a specially designed page – or views a forum entry containing a specially designed image – his Tunesafe backups will cease to be accessible.

This does not simply affect Norton Internet Security and Amazon S3: it is potentially a problem for all Web services when used with any anti-Internet software. There is a detailed blog posting here and an executive summary here.

Slashdotting

Slashdot is a site that offers “news for nerds”. To appear on its front page is to be read by hundreds of thousands of people. If the news story contains a link to your web site, very many of those readers will visit your site in a very short time, probably leading to the complete collapse of your server under the load. Your site has been “slashdotted”.

A malicious analogue of slashdotting is also possible. One of the more vicious Internet worms of the last few years aimed to recruit a network of millions of computers all of which would attack one site (such as a US government one) at the same moment. That particular attack failed because the worm was not well designed; but it is quite common to find high-profile sites such as Worldpay having to operate at reduced capacity for several days as a result of a denial-of-service attack.

One of the attractions of putting your publicly accessible data on Amazon S3 is that you are immune to slashdotting.

One of the dangers of putting your publicly accessible data on Amazon S3 is that you are immune to slashdotting.

Why is it a danger? Because a normal slashdotted site becomes slower and slower and eventually crashes or grinds to a halt. This limits the size of the attack. But if you have a 2MB photograph stored on Amazon S3, access to it will be as fast with a million visitors as it is with one; and at $0.0004 per visit, a million visitors will cost you $400. You are safe from a denial-of-service attack but open to a draining-of-bank-account attack.

I haven’t gone too far into this risk because Tunesafe is immune to it: nothing is publicly accessible at all, and each user’s backups are stored in his own private data space on S3. But if you are planning a public service, assess the risks carefully. Various S3 users are lobbying Amazon for automatic limits and cutouts so that access to their data is suspended in case of attack. Amazon are quite reasonably resisting their requests for now. For a start, it would imply real-time gathering of billing information across Amazon’s network, whereas at present they can get away with consolidating the data at relatively long intervals. This may end up as one of those risks that just have to be budgeted for.

Doshslatting

Doshslatting is slashdotting backwards. Amazon charge for uploads as well as downloads: so an attacker could wreck your budget by uploading vast amounts of data. (Again, because it doesn’t have publicly accessible data spaces, Tunesafe is immune to this.)

Doshslatting can only be malicious, which makes it rather easier to deal with. Given the cheapness of S3 and the slowness of data transfer, an attacker would have to have access to a “botnet” of thousands of subverted PCs in order to cost you very much money. The motives for doshslatting could only be vandalism and extortion. Given that a botnet can be used for many lucrative purposes, vandalism isn’t worth the bother; and extortion is not only criminal but necessarily involves contact between the extortioner and his victim. There are easier ways to make money out of botnets.

Amazon S3 already contains partial protection against doshslatting. Rather than having a publicly writeable area of your storage space, you can make all of the space private and accessible only by signed requests from a client. You then make your web application work as follows:

  1. The web browser asks your own server for a signed write request.
  2. Your server considers the identity of the user, and if it thinks the user is legitimate it creates a signed request and sends it to the browser.
  3. The web browser sends the signed request to Amazon S3 along with the data it wants to upload.

This is good, rational protection, and it works. If Amazon allowed the signed request to include a size limit – “please write a data object, the object to be no more than 4MB in size” – things would be better still (technically, the modification would be very easy indeed). What is even better is that signed requests in S3 can have time limits, so your server can issue a request that will only work for 15 seconds after it was created. This will prevent most kinds of botnet attack.

Summary

The anti-Internet problem
This will become more and more of a problem as the use of web services grows. The problem will be stopped eventually, because the makers of anti-Internet software will have to make their products work rationally or people will stop buying them.
Slashdotting
There are technical impediments to Amazon providing precise rate-limiting mechanisms, but an imprecise one may eventually become available (eg. limiting starts when the traffic has reached somewhere between 1 and 5 times the chosen limit). Even so, this is a risk that has to be borne by the business that uses Amazon S3. Methods of monitoring and mitigation have to be considered and a budget set aside to pay Amazon’s fees if a surge in demand does occur. Not every risk can be prevented or pushed onto someone else’s shoulders!
Doshslatting
Methods exist to reduce the risk to the point where only a really dedicated attacker could have a measurable impact; and such attackers will have many more lucrative things to do with their time and resources.

Next time, this series is brought to a conclusion.

S3 in Business: 11 – Political risk

25 August 2006

[Complete series]

Last time, I covered the legal risks of entering into a contract with Amazon for S3 or its other web services. The complete loss of your service and data at any time, for any reason or for no reason at all, may seem a bad enough risk; but there may be worse to come.

Amazon boast a network of data centres. At present this “worldwide” network only covers countries that send teams to the World Series; but at some time in the future Amazon plan to put data centres in Europe and elsewhere to increase access speed, which is not really adequate outside North America (see A Slow Interlude).

Let’s imagine that Amazon place a data centre in the UK. This means fast access for UK users, who will no longer have to rely on their ISPs’ transatlantic links.

As part of Amazon’s worldwide network of data centres, the UK centre will receive replicated copies of data from all over the place – not only data originating or being used in Europe – as Amazon’s proprietary algorithms ensure maximum data security through maximum geographical diversity. If you scan a document in Peoria for use in Chicago, and store it in S3, it is as likely to end up in England as anywhere else.

You will probably be prudent and encrypt your data.

In England and Wales, the Regulation of Investigatory Powers (RIP) Act 2000 says that as part of an investigation “in the interests of national security, for the purpose of preventing or detecting serious crime, or for the purpose of safeguarding the economic well-being of the United Kingdom” the police may demand the keys to any encrypted data. It is an offence (with sentence of up to two years in jail) to fail to provide your encryption keys, and it can also be an offence to reveal the fact that the keys have even been demanded. Ross Anderson, the Professor of Security Engineering at Cambridge University has been forced to cover himself like this:

Here is my PGP key. If I revoke this key, I will always be willing to explain why I have done so provided that the giving of such an explanation is lawful.

So, suddenly, simply because Amazon have replicated some of your data into the UK without your knowledge, you open yourself to police demands to reveal your encryption keys with the threat of a prison sentence if you don’t.

A great deal of work is being done on this: see, for example, this blog entry on Cambridge’s Scrambling for Safety conference, which was devoted entirely to the RIP Act (there is BBC coverage here). On the government side, the draft code of practice is all emollience, containing safeguard after reassurance after safeguard in the best manner of the British Civil Service. But these reassurances ultimately come from the government of a country where a Labour Party member (an 82-year-old Holocaust survivor) was detained under anti-terrorism legislation for daring to heckle a minister at his own party’s annual conference (BBC report here), and so we may be forgiven for not being entirely reassured. How long before “the economic well-being of the United Kingdom” is interpreted as “the profits of a company in which the ruling party owns shares”?

I have mentioned one country but there will be others. The restrictions and penalties on S3 data will be a combination of all the possible ones in all the countries where Amazon choose to have a data centre. Do you really want to investigate the law and politics of every country where Amazon might decide to put a computer? This kind of international political risk can only get worse as Amazon’s data centres spread round the world.

Perhaps now we know why Jeff Bezos is so interested in space flight. S3 servers in low earth orbit could in the end be cheaper to buy than politicians. Googling for “low earth orbit routing protocol” already returns 12,800 pages

Next time I’ll mention a collection of risks that I’ve filed under “M for miscellaneous”: denial of service by anti-virus software, and cost inflation through slashdotting and doshslatting. It’ll break all my previous promises by being quite a technical posting, but be patient because the time after next will bring the conclusion to this whole series. Thank you for reading so far.

S3 in Business: 10 – Contractual risk

24 August 2006

[Complete series]

Last time, I covered the technical risks associated with using Amazon S3, investigating how sure we could be that S3 was reliable enough for our purposes.

Now (rather late) let’s turn to the simplest question of all: will Amazon let you use S3 to back up your music?

You will not use Amazon S3 in … any manner that might be discriminatory based on race, sex, religion, nationality, disability, sexual orientation or age.

So before you use Tunesafe, you must erase Debussy from your iPod (“The Golliwog’s Cakewalk”), all of Wagner (a known racist), and anything by NWA – to name but a few.

Yes, you say, but surely I could encrypt everything so that Amazon don’t notice?

You agree to provide such additional information and/or other materials related to your Application as reasonably requested by us or our affiliates to verify your compliance with this Agreement.

Well, no, actually. Amazon require the right to listen to all your music to make sure that it is the sort of thing they want to have on S3. This is reasonable. How can they tell whether your encrypted confidential data are illegal or not, if you don’t let them read your data?

What if we ignore this and go ahead anyway?

If your Application is determined (for any reason or no reason at all, in our sole discretion) to be unsuitable for Amazon Web Services, we may suspend your access to Amazon Web Services or terminate this Agreement at any time, without notice.

How thoroughly will Tunesafe have to censor its users’ content? Would it be OK to have music by Tom Lehrer on your iPod, or is poisoning pigeons in the park an illegal activity? (after all, the Audubon Society objects to it). To answer this kind of question definitively you will have to have a detailed knowledge of the criminal law of the state of Washington.

Your lawyers will not be idle. Apart from keeping tabs on Washington law and Washington politics they will have to check the Amazon Web Services Agreement hourly, since

We may modify any of the terms and conditions contained in this Agreement, at any time and in our sole discretion, by posting a change notice or a new agreement on the Amazon Website. Your continued use of Amazon S3 following our posting of a change notice or new agreement on our site will constitute your binding acceptance of the change.

…so that even if possessing Poisoning Pigeons in the Park isn’t grounds for erasure today, a quick addition of “cruelty to animals” to Amazon’s list of Banned Things could easily make it so tomorrow.

Committing crimes with electricity

Amazon S3 is meant to make data storage into a commodity, just like electricity. So I called my local electricity company to ask them if I was allowed to use electricity for criminal purposes. If they knew I was committing a crime, could they cut me off? Could they monitor my activities in order to check whether they needed to cut me off?

No, they said. I can use electricity to make defamatory statements, or to publish racist and genocidal propaganda, or to be cruel to badgers or bats. I could even use it to grow cannabis plants (except that British cannabis growers steal their electricity by bypassing the meter). As long as I pay my bills, the electricity company will continue to supply me with power to continue my illegal activities.

The contractual basis of doing business with Amazon

Amazon will claim that their licence agreement was drafted by a moron in a hurry and that they don’t really mean any of it. “We don’t mean any of these nasty things, really. You know we’re nice guys so just go ahead and don’t worry about the contract”.

This is a widespread modern trend in both contracts and legislation. It puts the customer (or the citizen) permanently in the wrong but spared the consequences of his wrongdoing by magnanimous corporations and governments. It essentially abolishes the rule of law and replaces it with administrative fiat.

The Amazon licence agreement is not a suitable basis for any serious commercial relationship and anyone who bases a business on this contract is failing in his duty of care to himself and his investors.

Why the contract is the way it is

Internet law is a shambles – international, national and state law all cemented together into a legal breccia, with no-one knowing quite where the lines of fracture will go when the whole thing is stressed. There is no universal consensus as to where criminal liability lies for the storage or transmission of data. Different jurisdictions take contradictory approaches. Some of them enact absurd laws and trust that they won’t be used: in the UK, for example, if I ring up a friend and encourage him to have a cigarette to get over an emotional crisis, my telephone company has committed an offence simply by transmitting my voice.

It is natural that Amazon’s lawyers (aware that their company is a large target with deep pockets) try to shrug the entire liability onto their users by banning any activity that might get Amazon into trouble. It is also natural that the users, reading the contract carefully, will avoid any activity that might get Amazon into trouble by avoiding any activity at all involving Amazon.

For the health of the entire Internet services market, someone has to establish the extent to which data storage and transmission are to be treated, like electricity, as an opaque commodity without a legal weight of their own. I can’t do it and Tunesafe is too small. It is the responsibility of large corporations such as Amazon to buy a few legislators (preferably, ones with international connections) and get the job done properly, to the benefit of us all.

Next time, how Amazon’s future actions could mean S3 users being liable to arrest and imprisonment if they ever travel to the UK.

S3 in Business: 9 – Technical risk

23 August 2006

[Complete series]

Last time, I covered the commercial risks associated with Amazon’s continued existence and continued offering of Amazon S3. Now it’s time to turn to technical risks. How reliable is the Amazon S3 service? Can we trust our backups to it? Can we build a business on it?

Computers go wrong. That is what they do. The job of a service provider is to build a reliable service on top of unreliable machinery. Amazon don’t say much about how they achieve reliability except that they have multiple data centres in different locations and S3 doesn’t even acknowledge receipt of a data object until it has been successfully stored in at least two geographically separate places. The idea is that a failure in one place is improbable but a failure in two places is inconceivable.

One can measure reliability in two ways: the percentage of time that the service is available, and the probability of a stored data item being lost or corrupted. In the case of Tunesafe, an occasional loss of service doesn’t matter much (the Tunesafe program can just go to sleep for a bit and try again later) but the loss or corruption of data is very serious indeed. It is in the nature of backups that no-one looks at them for years and years, and this means that any errors that have been introduced into the backup are discovered far too late for anything to be done about it.

Amazon make a claim about service availability (at least 99.99%, which means one hour’s downtime per year) but they make no claims about the integrity of the stored data. And they make no contractual promises at all about anything.

Amazon make a great thing of the fact that they use Amazon S3 themselves for their own business. This is a great recommendation but it isn’t actually a guarantee. Quite apart from anything else, we can’t compare Amazon’s tolerance for data loss with our own. If we use S3 as our only backup we may be far more sensitive to losses than Amazon are. Amazon with one or two books missing is still Amazon: a backup with one or two files missing is practically useless.

Why data loss is an easy risk

Data loss has one encouraging property: it is easy to tell whether it has happened. With S3 it’s easy to prove whose fault it is, as well: if Amazon maintain a tamper-proof log of every PUT and DELETE request to S3 then anyone can tell, objectively and beyond dispute, whether any object stored on S3 has (a) disappeared without a DELETE request (b) changed since the PUT request that created it.

So we have something that

  1. is a rare event
  2. causes harm
  3. can be identified with certainty.

An event like that is a prime candidate for insurance.

Suppose that Amazon paid compensation for each lost or corrupted S3 object – perhaps a thousand dollars per object, perhaps one cent per byte – how much would that cost them? We don’t know, because we don’t know the quantity of data that S3 loses; but Amazon do know, or they can get a mathematically inclined intern to find out.

Suppose that Amazon divide that total cost of compensation by the number of S3 requests received. Does the total come out at one cent per gigabyte? More? Less? Again, we don’t know but Amazon do.

A revolution in insurance

Traditional IT insurance is cumbersome and expensive. It requires the insurers to assess the degree of risk beforehand, with experts going through every bit of software, hardware and operating procedure. Handling claims is expensive as well, since the insurer has to assess exactly how much harm has been done to the business by an IT failure.

Things don’t have to be this way. To see this, let’s take the second point first. The “purest” form of insurance is pluvius insurance – you are having an open-air event on a certain date and you pay a premium so that you can be compensated if it’s rained off. Pluvius insurance is pure (a) because the insured person has no influence over whether the loss happens and (b) consequently the insurance can be for any amount that the insured person feels like. If I insure my local school’s fund-raising fair for $10m, no-one cares as long as I pay the premium: I can’t make it rain. So assessing the risk is as simple as reading weather records, and paying out the claim takes no time at all.

The kinds of insurance we’re familar with are “impure” by comparison, because the insured person has a strong influence over the probability of loss. My car insurers will want to know in advance what sort of driver I am and whether I live in a rough neighbourhood; and if my car is wrecked or stolen they won’t pay me any more than they say it’s worth. If it were otherwise, I could turn my car into money a lot more easily than by selling it second-hand. (I am told this used to happen in parts of Australia where they had fixed-value car insurance: you’d park your car in a certain spot with A$200 on the front seat and the keys inside the exhaust pipe, go for a drink and come back to find the car gone.)

Our risk of data loss and corruption is more like rain than like car theft. It is easy to assess in advance (if you know the error rates) and because we cannot influence the outcome, it can be insured for any amount that we feel like insuring it for, without any need for tedious calculation of the exact value of the loss suffered. All this would make S3-based data loss insurance very, very cheap to administer. It would lead a revolution in insurance just as S3 itself is leading a revolution in data storage.

Who should offer this insurance? The obvious answer is Amazon. They have the exact figures for the probabilities of data loss and no-one else has. It is like offering life insurance when you are the only one with accurate mortality tables. How they would do this would follow the pattern of many other industries around the world: they set up a wholly-owned insurance subsidiary, which offers data loss insurance for data stored on Amazon S3. That subsidiary underwrites part of the risk and reinsures the rest of it in the wholesale insurance market.

Amazon can thus give us a choice: the current S3 storage service at the current low cost, complete with all the empty guarantees about “we use our own service so it must be safe”; or an insured service for a few cents extra.

A cautious user of S3 could pay the extra premium; a braver user could use the size of the premium as an indication of how risky S3 storage actually is.

A revolution in IT responsibility

There is a Peanuts cartoon in which Lucy goes round getting everyone to sign a piece of paper that absolves her from all blame. No matter what happens any place or any time in the world, this absolves me from all blame!

When Lucy grew up she went into IT.

Every software and hardware supplier has contracts absolving itself from all blame if its products go wrong. As a result, IT remains an infantile industry. Because blame is disclaimed, there is no need for anyone to take responsibility. Moreover, where there is no blame there is no apportionment of blame. For example, if someone has difficulty printing, there is no way of finding out whether the cause is (a) the application program, (b) the operating system, (c) the printer driver, (d) the print spooler, (e) the printer firmware, (f) one of the fonts. And that is assuming that you’re not printing across a network!

As suppliers of Cardbox we spend a lot of time pinning down the causes of problems suffered by our users. We always hope that the cause is a bug in Cardbox, because then we can cure it; but all too often the cause is somewhere in the thicket of letters listed above. And there is no quick way of pinning it down, because none of the components have been designed to do a proper audit trail to help with troubleshooting. So the user suffers; and every day computing becomes less like engineering and more like medicine.

Suppose that Tunesafe wanted to offer insured backups to its users. It could use insured S3 storage, which accounts for most of the risk. The remaining risk would be that data loss or corruption was being caused by bugs in Tunesafe. Tunesafe being small and simple, that risk wouldn’t be too hard to assess and the company could decide to carry it itself. Of course there is also the risk of a greedy user trying to make Tunesafe go wrong so that he can get his hands on the compensation, so the Tunesafe program will have to have a secure log so that it can be proved whether (for example) a backup has been deleted because the user asked for it to be deleted rather than because something went wrong with the program.

So now blame can be reliably apportioned between “Amazon S3”, “Tunesafe”, and “elsewhere”; and the logs that Tunesafe keeps in self-defence will also be a valuable tool for debugging the rest of the complete system. Thus, one slice at a time, IT can be de-Lucified, with each component in a complex system able to identify whether it is to be blamed for an error or whether the error is someone else’s fault – and, if so, whose. It will take a long time, but S3 is a very good place to start.

Next time, contractual risk: will reading Amazon’s contract with us cure insomnia, or cause it?

S3 in Business: 8 – Commercial risk

22 August 2006

[Complete series]

It is risky to offer a business that depends entirely on a service provided by someone else. Amazon S3 might stop tomorrow, or it might become too expensive for us to use. It’s only been going for a few months, so we don’t even have a track record to go by. Here are some of the things that might happen to S3 in the next few years:

  • Amazon go bust.
  • Amazon are taken over by someone with no interest in continuing Amazon Web Services as an external product offering.
  • The stock market decides that Amazon ought to stick to selling books, and, faced with a steep decline in its share price, the company obeys.
  • At the next scheduled review of the project, Amazon decide that Amazon Web Services hasn’t been the success they hoped. They get tired of it and pull the plug.
  • Amazon decide that it’s too much bother dealing with all those tiny customers with tiny payments, and impose a $100-per-month minimum spend on S3.
  • Amazon realise they got their sums wrong and double their prices.

We are like a fly hitching a ride on a racehorse: it is taking us where we want to go, fast and with no effort, but we could be swatted off at any moment. How do we reassure our backers that we are safe? How can Amazon reassure us that we are safe?

Contractual reassurance

Amazon need to reassure us that S3 will not be switched off tomorrow. How do they do this while keeping their commercial freedom of manoeuvre? It is for them to think of something and for the market to decide whether it is good enough. Perhaps they could write a notice period of a few months into their contract, long enough for users of S3 to retrieve their data and store them somewhere else. They might also ring-fence S3 financially in such a way that it would continue for at least the notice period (possibly with reduced performance) whatever else might be happening at Amazon as a whole.

S3 without Amazon?

When the Cardbox Server does its automatic backup to S3, it sends S3 commands to a server and waits to receive S3 responses back from the server. Tunesafe would work in the same way. The server used is Amazon’s S3 server at s3.amazonaws.com, but it doesn’t have to be that particular server. Cardbox (and Tunesafe) would work just as well with any other server, as long as it also understood S3 commands and sent S3 responses in return.

If a different data storage supplier also supported the S3 protocol then we could choose whether to use Amazon’s service or its rival’s, and if one company failed then we could switch to the other one. There is nothing new about second-sourcing, which has long been a standard method of risk limitation in the defence and aerospace industries. There is an art to it: you want the second source of your designs to be big and efficient enough to be credible but not so big and efficient that they will take too much business away from you.

Amazon claim that the big value in S3 is not the S3 protocol itself – anyone could have designed that – but the enormous investment that they have made to ensure that information stored in S3 is always available, available fast, and never lost. That being the case, it would not be a threat to their business if someone else also offered an S3-protocol service. It would encourage customers to commit themselves to S3, knowing that they would not be completely dependent on Amazon’s continuing benevolence.

One could even imagine Amazon publishing a simple open-source program that turns any single computer into an S3 server. A single computer (or even a single data centre full of computers) would not be a rival to the full S3 service, but it would mean that no matter what happened – even if every contractual assurance offered by Amazon failed – a business dependent on S3 would have something to fall back on.

If this discussion seems a bit abstract, let’s compare Amazon’s data storage service with the electricity supply. Defining the S3 protocol is the equivalent of deciding that electricity should be provided at 220 volts and 50 cycles per second. Once that decision has been made, the market for electrical appliances blossoms because everyone can build appliances to use just one kind of supply. If some people get themselves diesel generators that generate 220V/50Hz as a backup in case the mains supply fails, that is no threat to the electricity company – it’s actually a benefit, because it means that those customers won’t look for alternative kinds of power. Even giving away diesel generators would make sense, if they were cheap enough, because no-one would rely exclusively on generator power when the mains supply was cheaper than diesel, more reliable, and of unlimited capacity. (If you are in America, please read “220” as “110” and “50” as “60”.)

Allowing widespread use of a protocol one has developed is not quite risk-free and Amazon will have to be careful how they do it. But if they can state publicly that they have no problem with certain kinds of services that use the S3 protocol then that is a perfect additional reassurance for us to give to doubting investors. “We use Amazon S3,” we can say. “We hope that we’ll always use Amazon S3. But if ever we can’t, for whatever reason, then we do have an alternative that will work without any of our systems having to be changed”.

Next time, technical risk. What if Amazon S3 doesn’t work exactly as advertised? What if it starts forgetting things from time to time?

S3 in Business: 7 – An introduction to risk

21 August 2006

[Complete series]

Let’s imagine that Amazon have thought a lot about it and eventually taken up the Golden Buckets idea and let’s think forward a few months, to the moment when we’ve done the planning, done some programming, and are starting to approach possible financial backers.

As such people do, the backers appreciate our excitement but see it as their job to contribute the prudence that we evidently lack. Alternatively, they are a bunch of cynical pessimists looking to talk down the value of our project so they can get more of it for their money. It doesn’t really matter which description you choose: the fact remains that they come up with a list of possible things that might go wrong. For each risk, they want to know how likely we think it is to happen, how much it will cost us if it does happen, and how we can protect or insure against it. Even if we don’t have backers and are funding it all from our own pocket money, we still need to consider these risks and ask ourselves the same questions.

I’ve divided the risks into seven classes:

  1. Credit risk: what if people get stuff from us and then don’t pay? This doesn’t apply to the Tunesafe project because, with golden buckets, none of our users pays us anything anyway. The only people who do pay us are Amazon, and their creditworthiness can be neatly considered under…
  2. Commercial risk: this business depends entirely on repackaging one service from one supplier. How vulnerable are we to that supplier’s actions?
  3. Technical risk: this business assumes that Amazon S3 works. What if it doesn’t, or is unreliable?
  4. Legal/contractual risk: does the contract we have with Amazon allow us to run Tunesafe? Could it be used to stop us later?
  5. Political risk: could users of Tunesafe be sent to prison?
  6. Client-side risk: what if people can’t run the Tunesafe software? Worse, what if they do run it but something happens later on that makes them unable to connect to their data?
  7. Slashdotting and doshslatting: what if our site is overwhelmed with traffic and we are bankrupted as a result? This doesn’t apply to the Tunesafe project because users pay for their own data transfer directly, so it doesn’t cost us anything, but it is still a risk that concerns a lot of potential Amazon S3 users.

Next time: commercial risk. How dependent are we on Amazon’s commercial decisions? What can Amazon do to reassure us? What can we do to protect ourselves?

S3 in Business: 6 – A slow interlude

19 August 2006

[Complete series]

Resting from the excitement of thinking up an entire new business model (last time), it’s time to do some sums about speed. How practical is it to use S3, or any other Internet service, for backups? Let’s look at how fast data can travel between your PC and Amazon S3.

ADSL stands for “asymmetric digital subscriber line”. A 2Mb ADSL line has an upload speed of 288kb (36kB) per second, with the rest of the speed being used for downloads. That upload speed equates to about 8 hours per gigabyte.

For my 23.4GB iTunes music library, this means a total backup time of 187 hours. Whether you do nocturnal backups (8 hours per night, every night) or daytime backups while you’re away at work (11 hours per day, Monday to Friday) that works out at just over three weeks to back up my entire music library.

(For an 8Mb line the figures are 3.3MB per minute, or 5 hours per gigabyte, adding up to 120 hours, or about two weeks).

This sounds horrible. It’ll be enough to put many people off, but we’re rescued by a couple of facts. First of all, the Tunesafe program is designed to do a little work at a time and it doesn’t mind being interrupted: however long the journey, with one little step after another you’ll get there in the end. The other helpful fact is that iTunes music libraries are quite special things. The individual files within them are not all that big and they don’t change much over time, so that once your library has been backed up it stays backed up. After the initial burst of activity in the first few weeks, the uploads will slow to a trickle and your ADSL connection will be able to handle them easily.

Technical point: If you’ve rummaged inside your music folders then you’ll have seen that whenever you take a track and change something simple like the name of its artist or composer, the entire MP3 file changes. This is why an ordinary file backup program isn’t good enough for iTunes music: change “Mozart, Wolfgang” to “Mozart” and you could let yourself in for a whole extra night’s worth of backups! Tunesafe is cleverer and it doesn’t have to re-upload an entire music track just because you’ve changed the track title or the artist’s name. This is one of the reasons why Tunesafe is worth paying for.

If you ever need to restore from your backup, you won’t have nearly as long to wait. Because ADSL is asymmetric, downloading is a lot faster than uploading – nine times as fast on a 2-megabit line or 17 times as fast on an 8-megabit line.

If you’re not looking at ADSL but at a project where you’re transmitting data to and from an Internet server, everything is a lot faster but you should be aware that Amazon S3 isn’t worldwide yet. To give an idea of the speeds you might expect, uploading from our Rackspace server to S3 took 40 minutes per gigabyte over the transatlantic link from the UK. This is a lot faster than uploading from a PC, but it certainly isn’t instantaneous and it’s worth taking account of it if your business plans are international.

Next time, we’ll start thinking like potential investors in the Tunesafe business. We’ll look on the dark side: what are the risks of depending on Amazon S3?