Excel and data management - when will they learn?
We’ve written once or twice before about the perils of using Excel for the management of large-scale critical data… but it seems that big organisations and government agencies (who should know better) still haven’t broken old habits.
In the midst of the Coronavirus crisis, the news broke this week that Public Health England allowed nearly 16,000 cases to go unrecorded, because it used Excel for its data management, rather than using a database. And the story got worse the more you read of it. Not only was Excel used, it was that the outdated .xls format – created in 1987 and only capable of handling 65,000 rows of data – that caused the error. All records in excess of that number were dropped when the data was brought into Excel, resulting in the 15,841 unreported cases. Had the current .xlsx format been used, a million or more records could have been stored before the error occurred.
But even leaving number of records aside, the big take-away is that Excel isn’t the right tool for the job. Part of its menace is that it’s accessible, widely available and easy to use, making it something of a default. But it’s not secure and it’s extremely error-prone. There’s no history, no audit trail, and no ability to test functionality and catch errors. Store and manipulate data in Excel at your own risk. But anyone managing data that’s critical – either to their business or to the health of the country - shouldn’t be within a mile of Excel.
As Professor Jon Crowcroft from the University of Cambridge was quoted on the BBC news website:
"Excel was always meant for people mucking around with a bunch of data for their small company to see what it looked like. And then when you need to do something more serious, you build something bespoke that works”