If your network administrator has prevented you from downloading or running .VBS files, you can get round this restriction by storing your VBScript in files of a different type: for example, .TXT.

Suppose that you want to run a file called C:\My Scripts\xxx.txt. The command to use is:

%SystemRoot%\System32\WScript.exe //E:vbscript "c:\My Scripts\xxx.txt"

You can create a Windows shortcut for a command like this, to save typing it every time.

Here is a sample macro in Cardbox that launches an external .VBS script stored in a .TXT file:

x="C:\My Scripts\xxx.txt"
Launch "%SystemRoot%\System32\WScript.exe","//E:vbscript """ & x & """"

Use Run instead of Launch if you want your macro to wait until the external script has completed.

Note: If you are using a Cardbox build earlier than 4249, you will have to replace %SystemRoot% with the actual name of your Windows directory: probably C:\Windows.

The project to add Amazon S3 facilities to Cardbox has been completed. (If you’re not familiar with Cardbox then you can read about it here and get a free 1-month trial licence here).

Amazon S3 is storage for the Internet. Secure, reliable and cheap, it can be used to store any amount of data, at any time, and retrieve it from anywhere on the web. There is no sign-up fee and you pay only for the storage you actually use.

In the new build of Cardbox that is released today, we have incorporated the following features to make use of S3:

The Cardbox Server (used in multi-user and networking systems) can be configured to back up your databases automatically, at specified intervals, to your S3 storage space. There is no interruption in service during a backup, because the Cardbox Server can back up a database even if it is in use. You can have a single backup copy of the latest version of each database, or you can have a separate backup made each time that the database has changed.

Cardbox itself has a new command that lets you view your S3 storage space and upload, download or delete files. Apart from managing the Cardbox Server’s backups, this also allows you to make backups of files on your own computer.

Cardbox’s VBScript macro system has been expanded to allow you to manage your S3 storage space programmatically. This offers interesting prospects for the future: for instance, a photographic database that stores and displays each photograph at a reasonable screen resolution while the original full-resolution images are held in an archive on S3.

Amazon S3 is storage for the Internet. Secure, reliable and cheap, it can be used to store any amount of data, at any time, and retrieve it from anywhere on the web. There is no sign-up fee and you pay only for the storage you actually use.

This posting describes how we are enhancing Cardbox to take advantage of the facilities that Amazon S3 offers.

If you don’t already know Cardbox, it’s a fast, straightforward database designed for end users. Anything you can list, Cardbox can store. Its particular strengths are powerful text indexing and the ability to attach photographs and scanned documents to every record. Cardbox is used by everyone from individuals and small businesses to multinationals and United Nations agencies, and there are Cardbox users on every continent including Antarctica. The Cardbox web site tells you more.

When Amazon S3 was released we knew immediately that this was something we wanted to use. The next question was: faced with unlimited, reliable data storage space, how could we extend Cardbox to take advantage of it? We had several ideas, but the one that promised the greatest benefits to the largest number of users was automated backup. (We will describe the other ideas in future posts).

The trouble with backup has traditionally been that it is something that someone has to do. Except in the most sophisticated organisations, someone ends up physically having to burn a CDROM or DVD, or even manipulate magnetic tape. Even when the backup has been done the problems are not over, because the backup has to be stored somewhere safe but accessible, organised so that you can find the right DVD again when you need it.

Automated backups

With Amazon S3, we have been able to add an automated backup facility to the Cardbox Server. In the Server’s configuration pages you tell it how often you want to back up, and the name of the location in S3 where you want the backups to go. And that is all. Cardbox does the rest.

The beautiful thing about letting the Cardbox Server do its own backups is that service doesn’t need to be interrupted while the backup is going on. Cardbox users can continue searching and browsing their data, and even making changes, during the backup. Cardbox knows enough about its own behaviour to be able to store the changes that users make while at the same time ensuring that the backup is a consistent picture of the entire database at one particular instant.

Retrieving from backup

The part of Cardbox that users actually see and interact with is the Cardbox Client – which we usually just call “Cardbox”, full stop, since as far as most people are concerned, the Cardbox Client is Cardbox.

We have extended the Cardbox Client in two ways. First of all, we have added commands to Cardbox’s VBScript macro system to allow objects to be listed, downloaded, and deleted (we have also included uploading, in case sophisticated macro writers need it). Secondly, we are implementing a window that shows a simple listing of all backed-up files and lets the user download or delete selected files. (Typically this feature would be under the control of a responsible administrator, since once an object is deleted from S3, it is deleted).

Summary

• Automatic backup at regular intervals.
• Secure and reliable storage on Amazon S3.
• Simple viewing and management of all backed-up files.
• Downloading on demand.

Things to be done before release

The features described here are in our internal development version of Cardbox. They will be incorporated in the production version (and formally announced to Cardbox users) soon. The principal work to be done is to add appropriate pages to the Tools > Options and Server Configuration windows, and to add a new command (Tools > Management > Amazon S3) to allow interactive management of backup files.

The future

The main future enhancement we are considering is incremental backup. If you have a database of 100,000 photographs or documents, the chances are that only a few will be added or deleted in one day, meaning that out of a 10GB database, only a few megabytes have changed. Sending only the changes to Amazon S3 will save time and reduce data transmission costs. This feature already exists in Cardbox (if you download a database to your PC from a server with Tools > Management > Download, only the changes get downloaded) but we are waiting to implement the equivalent in S3 until (1) Amazon add some necessary features to their service and (2) enough Cardbox users are using the service to make the development worthwhile.

We've been experimenting with using Amazon Web Services to extract book data from Amazon's database and import them into Cardbox. It all fits into a nice little macro and we'll publish the macro in the Knowledge Base in due course. You will be able to type in an ISBN and the macro will fill in all the book details for you. If you have a barcode scanner then you won't even need to type.
In the course of creating the macro we noticed that there was no way for us to get hold of the synopsis that appears on Amazon's book pages. Further investigation showed that the synopsis was available if we looked at amazon.com but not if we looked at amazon.co.uk. A thread on the relevant Amazon forum says, essentially, "this behavior is by design" (to use a favourite phrase of Microsoft's).

We thought we'd investigate further. Here are the details of a book published in the UK, as shown in amazon.co.uk and amazon.com.

Remember, this is the physically identical book. And yet there are significant differences between the entries. Even the number of pages is different!

This is a lesson for anyone planning a system round a database. Never imagine that you can keep the same data in two places. The data will diverge with time. Someone will change one copy and not the other; and then someone else will change the other copy too. After that, it is too late. No-one at Amazon is going to go through hundreds of thousands of inconsistent records, comparing them, checking with the publisher which one is correct, deciding which data to keep. It is simply too expensive. The cost of reuniting disparate data exceeds the cost of entering the data in the first place.

Sometimes organisations with several offices optimistically think that they can keep their Cardbox databases synchronized between sites simply by regularly emailing changes to each other. We know from experience that this always fails in the end, when someone forgets to apply an update, or someone edits something he's not meant to edit. Unless you one site is the "owner" of the data and no-one else is allowed to make any changes at all, the only safe solution is connecting to a Cardbox Server somewhere on the Internet. The Cardbox protocol is efficient, we tell them, and even a dial-up line will work…

Now we can show them that this is not just sales talk. No-one can accuse Amazon of being amateurs but, as this example shows, the hazards of split data afflict even the largest organisations and the damage can be effectively irreparable.

We're embarking on creating a "polyglot" Cardbox. The idea is that you right-click on the Cardbox title bar and get a list of available languages: pick one, and that's the language Cardbox will speak to you in from now on.

We're taking a collaborative route on this. Our idea is that a Latvian-speaking Cardbox user can get a message file from us (it's a big Word document) and fill in a blank column with his translations. He saves his work as a simple text file called cbxm0026.txt and gives it to us. We distribute it free… and suddenly it is possible for any Cardbox user to switch the user interface into Latvian.

There's a łot of cłeverness ïnsïde Cardbox that's needed to make thïs połygłot user ïnterface work, so we need to test ït ałł. How to test ït before any actuał transłatïons exïst?

Look closely at the question and you'll see the answer. Putting double dots on "i"s and slashes through "l"s is readable (as long as your browser can display eastern European characters) and it's a very easy translation to create: just do a couple of search-and-replace commands and you're there.

Of course, we had to call this new dialect of English something. In the end, remembering Mötörhead and Motley Crüe, the name was obvious. If you are a translator working on Cardbox and you right-click on the title bar, you get at least three options: English, English (United States), and English (Metal).

Writing in The Guardian, Andrew Brown wonders why OpenOffice, one of the flagships of the “replace Microsoft with open source” movement, is so bad. It is a question well worth asking, since in most discussions of open source software it is tacitly assumed that we are talking about nothing but a clash of business models and that the software that is on offer from both sides is essentially of equal quality – since if you have enough programmers then they’re bound to produce a product that works, aren’t they?

There is little dispute over the value of open source in tightly circumscribed projects with a well-defined function. If you want to create or display images in JPEG format, you would be mad not to use the Independent JPEG Group’s software. If you want to encrypt your data link using the Secure Sockets Layer standard, you would have to have a very good reason not to use the OpenSSL implementation. In both cases the projects are run by engineers for engineers and the economics make sense: in exchange for giving away the work I put into identifying and correcting a bug in IJG or OpenSSL, I get everyone else’s bug corrections free of charge. If I need to move my software project to another kind of computer system altogether then I will find that someone has already done the work for me as far as JPEG or SSL is concerned, or at least that someone is in the middle of the same move and we can share the work between us.

As software manufacturers we would like to sell software with no bugs; or failing that, software where we can find any bug that is reported, understand it and correct it. Using open source components lets us do this without having to worry about the quality control of our suppliers – or even their continued existence (Cardbox seems to outlive everything).

But as users? As a user of Mozilla Firefox I have no interest in the fact that the source code is available to me and even if I found out how to get it I wouldn’t help me much. For the inner core who work on the project, open source is a way of transcending corporate boundaries and getting a critical mass of people to work together, but for the people on the periphery the only benefits of open source are the indirect ones: good software, or cheap software, or software that is immune to the whims and fashions of the marketing people.

A living software project, like a living body, has to have a circulation. From the centre outwards, there flow improvements, extensions, corrections of bugs. From the periphery inwards, there flow discontents, complaints, anomalies and reports of bugs. Without both halves of the circulation the organism dies. A small project, like a small organism, can orchestrate this flow by diffusion: insects don’t need hearts. But when a project gets big then something needs to pump that stuff around, and the best pump is money. If a user whinges, you sympathize; if a paying customer whinges, you jump.

This leads to an apparent paradox: free software stagnates when not enough people pay for it.

Paid-for free software sounds like a contradiction in terms, especially if you have listened to too many Stallmanite zealots; but it is the reality of what is going on. Relatively few people use Linux; many more use Red Hat Linux, Debian Linux, Mandrake Linux, paying (one way or another) for someone else to sift through the various open-source tools and projects, find those that work, put them together, and make them easy to install.

Brown says “[If OpenOffice 3.1 is a world-beater,] it will be because large companies such as Sun, Google, and IBM have decided that open source is the cheapest way to gang up on Microsoft, because it means they need spend nothing on support.” Brown is right that big companies are driving the big open-source projects, but the rest is fantasy. What these companies want is what we as software manufacturers also want: a product that is under our control and where we know that we can correct any bug that arises or add any new feature that is needed. Far from “spending nothing on support“, the whole point is that open source frees us from the costs of proprietary software development and lets us spend everything on support: it is what differentiates us from our competitors who are selling identical software to us. And our customers don’t have to worry about us taking a weird strategic direction or just going bust, because even if we do then there will be plenty of other companies around who can support “our” software.

So what of OpenOffice? OpenOffice has a difficult task because it is “big” in a sense in which even Firefox and Linux aren’t big: the job of a browser is simple and the job of an operating system is simpler still, but modern office software is exceedingly complex and has a rich set of interactions with its users. But the real reason that OpenOffice has trouble fulfilling this task is that not enough people are buying it, so there is not enough money to drive the all-important “venous circulation” that carries complaints and bug reports back to the developers with enough force to get them corrected.

On a corporate level this will change. The more open-source software that Sun and IBM sell, the more incentive they will have for improving its quality and the better OpenOffice will become.

But what’s more interesting is how OpenOffice could move forward even without the involvement of the big companies. The “venous circulation” problem presents an opportunity to design a robust circulatory system that will let even the largest open source projects grow and thrive. Suppose that Andrew Brown identifies his five most irritating OpenOffice bugs and offers $1 for each of them to be solved. Suppose that thousands – eventually millions – of OpenOffice users sign up to the “$1-per-bug” project. Who can doubt that the oldest and most irritating bugs will get solved very soon indeed?

Microsoft is facing the same problem and it has deliberately strangled its own circulation: short-term survival but long-term suicide. Microsoft has not solved the problem of distinguishing bug reports from support requests – it is probably insoluble – and it can’t afford to give support free so it has to charge for it. Charging for support suppresses the vital flow of information, complaints and bugs without which any software project must eventually stagnate and die. If you don’t have a support contract then it can cost you as an individual $200 or more just to talk to Microsoft. They say they’ll refund that if it turns out to be a bug but that still isn’t much of an incentive for a user of Microsoft software to report anything. You could imagine Microsoft tweaking the figures to increase the incentive (“if it’s a new bug then we’ll pay you $2,000″) but the numbers are so big and the multiplier effect of those millions of users so enormous that they simply daren’t do it in case it goes wrong.

The open source world doesn’t have this kind of money worry. It can experiment with all sorts of ways of managing information flows, of sifting out the real bugs from a mass of misunderstandings or simple user errors (but even a common user error is valuable information for a developer). Above all it can try different ways of injecting just enough money into the system to allow for a strong circulation and healthy growth. The actual money mechanisms are all there already (look at PayPal) – all that’s needed is imagination and inventiveness and a willingness to experiment.

We make our living from a program that runs on Windows. It is my dearest wish to see Cardbox running on something else. Why? – and how can it be achieved?

I am not actuated by great objections to the Evil Empire’s business methods (doubtless this is a moral defect in me) but Windows is crumbling under its own weight. Every time something is wrong, or over-complex, or incomprehensible in Windows, Microsoft add another layer of complexity to cover it up. For example, Windows has a way of requiring rebooting after almost any software update, however trivial it is. We are told that Windows Vista will contain additional programming to avoid this. I have seen this happen before and it is like finding a piece of rotten wood in your house and covering it up with waterproof paint. The rot doesn’t go away but you can’t see what it’s doing any more. The remedy for complexity is to paint more complexity on top… and when Windows finally collapses we don’t want to be buried in the rubble.

This is why the news that Apple is moving to Intel processors is so interesting to us. Most of the press coverage has been to do with the cost and performance implications of the move but the technical point is the really exciting one. It is already possible to run a limited set of Windows programs on Linux, in a limited way, using a software layer called WINE (I’m told that Cardbox 2.0 works under WINE but I don’t know about version 3.0). Now suppose that Apple puts its resources behind its own CIDER project:

1. Bronze Apple: Get as many Windows programs as you can to run on the Macintosh “out of the box” with no changes. When problems occur, cooperate with the manufacturers to find a way round them that doesn’t harm Windows compatibility. The programs will look more Windows-ish than Mac-ish but at least they’ll work, and a barrier that stops millions of users from even considering the Macintosh will be gone.
2. Silver Apple: Now that software developers know that there’s a worthwhile market there, put resources into a hand-holding project that will help them produce Mac-aware versions of their programs. No revolutionary rewriting from the ground up, just adaptation to the Mac way of doing things so that a Macintosh user doesn’t have to make allowances for an alien program’s origins.
3. Golden Apple: Encourage selected developers to go the whole hog and produce dedicated Macintosh versions of their software. The move to Intel makes the move easier technically, and the Silver Apple level will let the developers gauge the market and plan their investments accordingly.

Result: insurance against a possible collapse of Windows, the removal of an artificial market barrier, and, once more and more programs stop being dependent on Windows, a big opportunity for Linux as well.

And what about Linux? The Cardbox Server already runs on both Windows and Linux, but there are three problems with moving Cardbox to Linux as well: no-one uses Linux on the desktop; we can’t write programs for the Linux user interface because there is no one such thing; and no-one wants to pay for Linux software anyway.

Larry Osterman’s WebLog at Microsoft refers to an article by Eric Sink called My Life as a Code Economist. Sink’s thesis (endorsed by Osterman) is that it’s not always worth correcting bugs, because it may be cheaper or safer to leave them uncorrected.

Sink constructs a beautiful matrix of characteristics that determine whether you should try to correct a bug.

1. Severity – When this bug happens, how bad is the impact?
2. Frequency – How often does this bug happen?
3. Cost – How much effort would be required to fix this bug?
4. Risk – What is the risk of fixing this bug?

In good business school style, you plot each bug on a four-dimensional chart and use the chart to determine which bugs you should correct and which you shouldn’t.

But the most important question of all is missing:

Question 0. Do you know exactly what is causing the bug, and how?

If the answer is No, do not rest until the answer is Yes.

Here’s a really, really low-severity bug according to Sink: “One of the pixels on the splash screen is the wrong shade of gray.�? Too trivial to think about, except… why is it wrong? Perhaps someone clicked the wrong button when designing it. In that case the bug is genuinely unimportant. But suppose that you open up Photoshop and find that the splash screen picture is OK. In that case something is modifying the splash screen after it’s loaded from the disk but before it’s displayed on the screen. Some part of your program is modifying a byte in memory that it shouldn’t be modifying. In that case you have been extraordinarily lucky. A bug that modifies an arbitrary byte could in principle modify anything, anywhere. It could modify a data structure deep inside your program that could make large chunks of user data inaccessible and you wouldn’t discover the damage for months. You are lucky because right now this bug happens to be modifying something that you can see on the screen. You can notice it and reproduce it now; and having reproduced it, you can kill it.

In the late 1980s there was a bug in one release of Cardbox-Plus that manifested itself only when you were using a template to create mail-merged letters and the last letter of the template was a lower-case letter ‘e’ – in that case, one character might be lost in the mail-merged letter. We could have applied Sink’s rules and ignored the bug. But we hunted for the underlying bug and it turned out that in other circumstances it could have had much more serious effects.

The 0th law of bugs is that most bugs, most of the time, do their damage invisibly.

It follows that Rule 0 of quality control is If you see something go wrong, drop everything and find the underlying bug. If the problem disappears before the bug is found (for instance, if a modification elsewhere in the program makes it go away), then panic.

What about Sink’s rules in general? He has to make sure he’s got his economics right. It’s easy to forget that software lasts for decades and the cost of correcting a bug is only incurred once, while the cost of giving users extra support, and explaining to them what is wrong, and helping them out of trouble, is incurred over and over and over again.

Sink’s question 4 mentions the risks arising from correcting bugs. No doubt he’s right. But if your program code is so fragile that the smallest correction will make it collapse then you should knock it down and rebuild it now, before a mouse making its nest in the rafters suddenly makes the whole thing collapse in dust and rubble.