Not many people have the boundless level of energy, commitment, and curiosity for their passion that Dr. Howard A. Rubin, Founder of Rubin Worldwide has. He has held positions as a Nolan Norton Research Fellow, an MIT CISR Associate, a senior advisor at Gartner, a senior IT advisor at BCG, a Professor Emeritus from City University, and has participated in the Apptio CIO Advisory Council. His passion happens to be IT benchmarking, using data to visualize IT spend in the competitive landscape, which he got into over 25 years ago..
What spurred your initial interest in benchmarking? I know you’ve been about this for years, but what was the initial curiosity that got you going on it?
Two things. In the 1980s, I developed algorithms that very accurately estimated how long it takes to build software, and Computer Associates brought it to market. By 1989 I had about 2,000 companies with worldwide licenses for my software. I had this massive user database that was using my algorithms to estimate projects. To be able to use my tools, users automatically sent that data to me. I had this massively large database being accumulated over years. One of my friends and colleagues, Richard Nolan, who was a professor at Harvard Business School, and his associate David Norton, created this thing called the Balanced Business Scorecard. Their company was called Nolan, Norton and Company.
They became aware of all the stuff I was doing with this massive data collection. He said, “Howard, can you help us do benchmarking work?” And I said, “So what’s benchmarking?” He said, “Well, it’s comparing what people are doing against data so they can get lessons learned out of it. You have the world’s largest database on technology, and it would be highly valuable for people to be able to use that database to compare their competitive positioning with technology,” and that’s what got me started.
Then I got keenly interested, because it was very clear if you go back into the ’80s that technology was going to shape the world of the future. I didn’t quite know how. I got very interested in technology economics, in more than collecting project management data and project data on technology. I started going back to the same companies and collecting data about everything they were doing with technology. That really took off in 1990.
I wanted to lay a basis for studying technology, economics, its impact on society, on national competitiveness, on companies and peoples’ quality of life going forward. That was my motivation to start collecting data to help understand the interaction between technology and social forces on national competitiveness, on companies, and quality of life.
My life is like following walk and don’t walk signs in Manhattan. In 1994, I was contacted by Industry Canada on behalf of the Technology Minister of Canada and the Prime Minister. They wanted Canada to become a near-shore player in terms of getting work from the US and other nations. They ask me, “Can I do a benchmark project to compare Canada’s technology capabilities and development, maintaining systems and building them to the rest of the world?” and I said, “Sure.” I just wanted to keep the right to the data. I expanded what I was doing into the worldwide benchmark project. That still exists today.
I went on to help India and other countries. In 1996, Bill Clinton called me to the White House and asked why I was helping other countries take jobs from the US and not helping the US. Then I started helping Clinton and Corp. My answer was, “You never actually asked me for help,” and that was the beginning of a long relationship.
What is your advice to a CIO on how and when to use benchmarking in their TBM journey?
No matter what, whether it’s a company, a nation or something else, benchmarking helps you stay afresh of what your competitive positioning is. Benchmarking has to be interpreted at least in two ways. It’s one thing to benchmark the past and your current state. It’s more important to benchmark the future because everything is changing.
So like you said, you’re a pilot, you have experience in plotting and navigating courses. I don’t know if it’s a cliche or a parable, but there’s one that says if you don’t know where you’re going, any road will do. And if you don’t know where you are, a map won’t help.
Benchmarking is a tool that will put you on the map, help you figure out where you want to go and lay in the course. And that takes multiple dimensions. So in the world of TBM, structure is massively powerful in that it connects what you might call the bottom to the top. It looks at your technology, and measures in terms of income and outcome on business margins, performance, connecting technology to business transactions and things like that. What the TBM model almost forces you to do is consider the longitude and latitude of technology. Imagine that you apply this like longitude and latitude, where the latitude might be your technology spending and where the funding is going. The longitude might be your operating margins, their return equity or something else, and you can start to create a two-dimensional view of your position today, your position in the past, position of the future. It allows you to benchmark your technology parameters separately. It allows you to benchmark your business position separately and allows you to combine those so you can figure out what your flight path has been and what your trajectory is and also overlay that of your competitors.
Gartner says start with benchmarking to assess performance to peers and build your strategy from a baseline before even considering institutionalizing TBM. Gartner also says benchmark in the context of showback/chargeback. What are your thoughts on this?
I have an affiliation with Gartner. I think that is very sound but rudimentary advice. To explain that I have used a latitude/longitude analogy, they are just dealing with the “technology” latitude. It’s benchmarking technology as technology. And I think that picture is incomplete. When I advise companies on how they should be doing technology strategy, they should be doing it bi-directionally. The first direction is to figure out where they want to be as a business in the future. The second is to understand the technology economics associated with that future. That’s like the breakfast of champions with Wheaties, or living a lifestyle of the rich and famous like Robin Leach. Figure out who you aspire to be in business terms.
Use benchmarking to understand the technology, economic profile of the rich and famous or the breakfast of champions, and then drive your technology strategy from that. So my view is you benchmark from the outside in, from the business aspiration, then back in to what your technology needs are. Then you come up with your technology strategy to lay a course to that future. And that’s where I diverge from Gartner.
The Gartner model says, “Look at technology versus technology.” But does it matter? You can’t tell if you’re spending too little or too much. And one of the things that one needs to consider, and when you look at the Gartner publications. And this is not the slam, but if you look at the Gartner publications where they have wonderful publications that are based on a database structure they’ll say, here’s the average IT spending versus revenue. And here’s our average IT spending versus operating expense.
When in a company’s TBM maturity, should you use benchmarks?
You should use it from the beginning. Benchmarking first is a calibration tool where you relay your numbers against external numbers. You can learn a few things: If your numbers are really wildly off the external benchmark numbers, there can be some issues in what you’re collecting. You can point out deficiencies in your own data collection. For example, you’re collecting data, say an average employee height. You find your number’s 3 foot 3 and the rest of the world is 5 foot 10, maybe you’re collecting data a little differently. Maybe you’re only measuring them from the waist up or something. I know how stupid that sounds, but it’s an extreme example.
Once you start comparing things to benchmark, you can start to understand what you need to be able to compare against the outside world. So, the first thing when you start using benchmarks, you need to ensure that you’re counting the same, measuring the same. Are you including the same cost components or value components? Are you connecting the dots you’re seeing? It helps you establish the language to use for the future that’s consistent with TBM and consistent with the external data that’s out there. So I say you start benchmarking early, not to use the results of declaring yourself high or low, or to use the benchmark as a scorecard, but to use as a calibration tool.
It’s an interesting way to go about it. Should different levels of benchmarking be utilized before instituting TBM?
Now I think they go hand in hand, because at every level when you’re doing TBM, you’re introducing new metrics and measures and perspective, and these tests then sort of inform way as you go and that becomes key. Benchmarking helps you understand what you’re looking at it in a uniform way, where you’ll be able to learn from the future. Using benchmarking with TBM, is like the insurance for TBM. You’re ensuring that you can have comparability to external data, and even though semantically, you might think so, this allows you to test that quantitatively.
Does benchmarking help in building out a cost model?
Most definitely. The worldwide technology spend size now is probably about $7 trillion a year climbing up to 10% of the world’s GDP. So understanding the cost is absolutely critical, and understanding your costs relative to peers and what you’re investing relative to peers is like a space race or an arms race. You need to understand the ratios. You need to understand that absolute spending and cost is a thing the CFO, the CEO, the boards, the regulators all respond to.
Benchmarking helps you understand that you’re leveling the playing field of what you’re including in cost versus what others are including in costs and do a better job of interpreting that in the future.
What is optimal benchmarking cadence, and should you do a benchmarking exercise yearly or even more frequently?
Yes. I would say four times a year is the most extreme. I see companies that are pretty much best in class and doing it twice a year; the minimum perform at least a once a year refresh. And by a once a year refresh, I mean refresh what you did over the past year. You benchmark backward and forward.
When do you need professional consulting, and when can your IT Finance staff run a benchmarking exercise?
I’ve learned things that will be pretty interesting in the future, and it’s even changing the way I work. Some of the things you’ll see out of Apptio and the TBM people are positioning them so they’ll be almost like a Bloomberg of technology. If you were a trader, you have a Bloomberg station; you get all the information you need to do your own benchmarking and decision making. Right now, there is no Bloomberg of IT, but as the Apptio databases grow and databases from Gartner become more exact, a company should be able to just have a market data feed of technology information and be able to do their own benchmarking. I view the CIO as becoming the company’s chief technology economist.
Interesting. That’s really the way the direction it should be going.
Yeah, with technology, when something changes in the world of technology right now, people don’t know who to call. If something changes in the world of business, they call like Steven Roach, who used to be the chief technology, kind of a bit Morgan Stanley, with people that are recognized. This stuff will evolve. I believe the question’s a perfect one. If you ever internalize this stuff, I mean, if you have to control your life with a personal trainer, that sucks.
Should you expect benchmarking to always provide ROI?
Yes, but the ROI could be intangible and non-tangible forms. I mean, tangible forms might be helping you understand cost savings and things like that. More intangible forms could be you focus on better customer intimacy. You’re focused on just trying to make money on delivering your products. You can say governments need to do that too. And so what’s the ROI and those different dimensions? You can measure in dollar shape. You can measure in customer satisfaction. You can measure things like time to market and market share.
But the other piece is, those are some big buckets. When you apply technology to a company, there are about five flavors of things you get out of it. You can grow revenue, protect revenue, avoid cost, reduce cost, manage risk, and maybe just step around employee satisfaction and stuff like that. So how do you measure the ROI of protecting revenue? There are certain aspects of it where the ROI is perhaps non-traditional.
Do you have thoughts on Gartner’s assertion that high performing organizations regularly leverage external benchmarking?
Yes. And that is true because right now external benchmarking is the way it’s done. External benchmarking can give you a powerful set of external perspectives for interpretations. It’s also like sending your x-ray, your blood tests, your MRIs to an external physician for a second opinion. Having that second opinion or first opinion will always be important.
Are there some universal KPIs that all CIOs should be monitoring, in terms of status versus peers, and if so, what would they be?
If you’re a CIO, meaning what you’re charged with doing is leveraging technology to enable enterprise success, the KPIs that are most important are the things that appear to your annual report to your shareholders. So you’ll be looking at operating margin, or a return on equity or a net promoter score, market share: a whole bunch of business parameters. Make sure that the CIO is aware of what the business success parameters are. Also, the CIO needs to understand where they stand competitively in terms of the technology spend relative to revenue, relative to operating expense and that these things are behaving properly.
By that, I mean, if you’re putting money into technology and you’re expecting revenue lifts, you should be getting revenue lift faster than you’re throwing technology dollars at it. So I see expense as a percent of that revenue that should be decreasing over time. As you’re putting money into technology to improve the operating efficiency of the company, you should be driving non-tech cost out faster than you’re adding tech dollars. Imagine those two things as parts of a triangle: the IT as percent of revenue is the base, and IT as the percent of operating expenses as the vertical access, so as the bottom shrank and the right side grows, the hypotenuse is growing fast. That’s what I call IT intensity. So they should be monitoring their IT intensity relative to the competition.
Is there anything that we haven’t covered that you would like to cover?
Yes, there’s a complex social and cultural aspect of benchmarking. When I come into a company the first time, and they’ve never been benchmarked before, people are as welcoming to me, other than management who invited me in, as if I’m an uninvited proctologist. This is not a happy place to be. I try to get them to understand it’s not a report card. The positive side is that it’s a tool for critical thinking. It’s key that you don’t try to benchmark everything at once. It’s like an episode of crime scene investigation (a.k.a. CSI:IT). You collect a packet of data, and you follow the evidence to learn about how accurate your data is.
How do you help companies? You use the analogy of walking in there, them looking at you like you’re a proctologist. How do you help ease the pain and get them to understand? I can tell you that as a flight instructor, one of the things that I found is that when somebody has a better understanding of what something is, the fear goes away.
Yes, you have it exactly right. I never start a benchmark project with anyone by giving a list of numbers or a template to fill in. We’ll talk about the intent of the project. We’ll talk about what their expectations are and, if it’s been thrown at them, their expectations are pretty shallow. But I move the discussion away from the numbers to what do you learn for benchmarking beyond numbers and emphasize the fact that there’s not a report card to judge the performances.
One other thing I’ll add about TBM. You go back 11 years before TBM, there was no real lingua franca or framework to help people guide this. This is what TBM is. TBM is an absolute accelerant for business technology thinking and the basis for benchmarking. So TBM provides additional structures that help people get out of the gates fast. It connects dots like nothing else has. Literally, you have a structure, which is great.