SBA

Systematic Reduction of IT Support Cost

If you owned 10,000 assets used, and depended upon, by other people, and were responsible for maintaining them, and the cost of maintaining them, how would you go about it?

This is the problem facing most IT leaders, and most seem to come up with the wrong answer.

Is that unfair? It is widely accepted that 70% - 80% of corporate IT budget is spent on maintenance and support, most IT leaders consider that burden to be too much and would like to reduce it, yet it stubbornly remains the single biggest line in corporate IT budgets. Clearly most IT leaders are coming up with the wrong answer, time after time.

I used to be a director of a rental business; we rented out tens of thousands of industrial washing machines to hotels, hospitals, nursing homes, care homes, prisons, fitness centres, restaurants, hairdressers… you name it, if they had a business need to put clean sheets on beds, clean towels in bathrooms etc., then they were probably our customers. Washing machines break down with alarming regularity, it was imperative to the economics of our business that we managed the service cost of these washing machines. We kept the cost of support and maintenance down to c. 15% of our rental income - if we had allowed it to grow we’d have lost money - so we took a very systematic approach to service, and a very high quality approach, we won national awards and took over 50% of the UK market. We grew a very profitable rental company by designing in such a high quality of serviceability that we ensured that our customers did not need service.

I took the same lessons I learned running that rental business, and applied them to the IT estate. 

We identified the causes of support demand and systematically eliminated all that we could. I’ll not bore you with the details because every IT environment is difficult, but here are a few examples:

Malware: we created an infrastructure that inhibited malware entering our environment, made it very difficult for it to spread, prevented it from communicating with the outside world, and automatically removed any malware found, thereby neutralising it. Over a decade the three malware attacks we saw had very limited impact, and were easily removed. Unsurprisingly we did not use reactive ‘off the shelf’ security systems, instead we designed a naturally sterile IT environment. 

System Errors: we proactively monitored every system and service, automatically reporting any anomalous conditions in real time and dealing with them promptly so that they never had the chance to impact on people’s use of our systems - for instance if the peak load on a database server was normally 400 concurrent connections we would start investigating when it broached 500 connections even though the server was configured to comfortably handle 1,000 connections. Similarly we monitored all disk space, processor loads, memory usage, email queues, network traffic, etc. and reacted to non-conformances before any impact was perceived by users. We rarely had to deal with users claiming that “system X isn’t working” because we almost always detected performance deterioration before there was any significant impact.

User Problems: we recognised that, with the systems firmly secured and under control,  ‘users’ were our primary potential source of support calls - so we trained them. We bought a complete set of ECDL training materials and held training for all those users who we could persuade to participate. Over 50% of our office-based staff went through our ECDL training, effectively we created a workplace where half the staff were ‘super users’. Not only did they not need to call IT for help, but when someone had an everyday usage issue it was most likely to be resolved by one of their immediate co-workers who knew the right spell for Outlook, Word, Powerpoint or whatever. Almost all IT departments will joke that IT would be fine were it not for the users, we simply systematically engineered away the user problem.

Hardware Failures: pretty much all potential server hardware failures were avoided either by hardware monitoring and alerting such as S.M.A.R.T warnings, or by auto-failover redundancy: dual power supplies, dual Ethernet, RAID arrays, redundant RAM and processors - simply we made it very difficult for a server to fail completely before we knew about and rectified its problems. On the desktop estate we standardised on one model of desktop PC with one standard software build and pushed all user resources to the network file systems, with the result that we could replace any failed PC with a new one and have the user working again, with all their files, email, applications etc., in under 5 minutes from receiving a call to the helpdesk.

I could elaborate further, any IT manager will recognise that what we did was not clever, it was merely the application of what most IT managers recognise as necessary but rarely manage to achieve because there are always higher priorities. We systematically removed all causes of support demand that were within our influence.

So what? We achieved a position where less than 40% of the IT budget was spent on support and maintenance, both internally and with external suppliers. We delivered uptime of 99.996+% across a broad and complex range of systems. 70%+ of IT staff time was primarily engaged in business innovation, not support. From an IT director’s perspective we achieved something close to Nirvana.

These benefits were achieved because we knew from our experience in reducing the service burden generated by our customers that the systematic management of service could have dramatic results - and we prioritised managing down the IT service burden above all else in order to create the time and budget for the IT team to deliver business innovation. Simply by prioritising service we diminished it to manageable proportions.

In many IT departments the customer service / helpdesk function is seen as the tail-end of the business, it rarely gets equal ranking with the big sexy stuff such as infrastructure and systems. The consequence is very often that infrastructure and systems are not designed from the outset for service or serviceability. My experience is that putting design for service first when constructing corporate IT, and allowing service priorities to dominate above all other issues when designing infrastructure and systems, enables the IT leader to massively reduce the IT service burden and potentially double the IT resources spent on delivering business innovation.