The Evaluation of IS Investments Returns: the
RFI Case
Alessio Maria Braccini1, Angela Perego2, and Marco De Marco3
Abstract Today CIOs and IS departments in general are struggling to find a
framework to evaluate the performance and the return of their IS investments.
Notwithstanding a long-term research tradition on the topic of the business value
impact of IS, so far the identification of the returns of the investments of IS is still
an open issue. Even though a consistent body of literature has examined the prob-
lem over a time frame of more than 20 years, the IS business value research has
produced so far a plethora of theoretical contributions but with few practical ap-
plicability opportunities. Starting from the assumption that real-world experiences
differ from theoretical explications, and with the intent to contribute in the IS
business value research field bringing evidences and witnesses from the reality,
this paper presents a case of an IS performance management system used to assess
the value delivered by IT implemented in RFI (Rete Ferroviaria Italiana), the
company managing the Italian railroad infrastructure.
Introduction
The assessment of the real contribution of IT resources and Information Sys-
tems (IS) to the firm, in terms of business value, is the core of a wide and intense
debate that has engaged, and still engages, both academics and practitioners for
years. Interest in the debate has increased even though the conclusions of several
studies in this area have not been able to confute Robert Solow’s famous remark:
"we see computers everywhere except in the productivity statistics" (Solow 1987),
nor Nicholas Carr’s affirmation: “IT doesn’t matter” (Carr 2003).
Starting from that, several researchers have tried to examine the relationship
between IT/IS investments and the business value they are supposed to deliver.
Despite these efforts, a robust and complete model to evaluate IS business Value
that practitioners can apply in their companies is currently non existent (Gable et
al. 2008). As a result, the identification of measures to assess the efficiency of
IT/IS investments in terms of value is still an open issue.

1 Università LUISS Guido Carli, CeRSI Roma, Italy, abraccini@luiss.it.
2 SDA Bocconi, Milano, Italy, angela.perego@sdabocconi.it.
3 Università Cattolica del Sacro Cuore, Milano, Italy, marco.demarco@unicatt.it.

2
This lack of knowledge increases the difficulties firms face in the evaluation of
the value performance of IT/IS investments and also in closing the perceptual gap
that exists between IT/IS management capabilities and corporate management ca-
pability. In many cases firms implement IT/IS Performance Management Systems
(PMS) even though they cannot appropriately evaluate the results in economics
terms. In the light of the depicted scenario, this research paper contributes with the
description of an IT/IS PMS that has been applied in an Italian large firm.
The structure of the paper is as follows. The following paragraph will analyze
relevant literature on the topic addressed in this paper. Right after that, the re-
search design will be described immediately followed by the case description. The
case is then discussed in a further paragraph. Some final remarks regarding the
findings, the limitations, and future research plans, will conclude the paper.
Research Background
The contribution that IT/IS investments can deliver to the organization in terms of
business value is a research stream that, in spite of more than 20 years of studies
has not yet generated enough consensus among contributions (Oh and Pinson-
neault 2007, Tallon 2007, Kohli and Grover 2008, Scheepers and Scheepers
2008).
A relevant milestone in this research stream is the identification of the so called
“IT productivity paradox” by Brynjolfsson (1993), suggesting that traditional pro-
ductivity measures could not be appropriate to estimate the value outcomes of
IT/IS investments, for several reasons that can be grouped in four categories:
 Mismeasurement of outputs and inputs;
 Lags due to learning and adjustments;
 Redistribution and dissipation of profits;
 Mismanagement of information and technology.
Explanations 1 and 2 point to shortcomings in research, rather than practice, as
being “the root of the productivity paradox”. It is possible that the benefits of IS
investment are quite large, but that a proper index of its true impact has yet to be
analyzed. Traditional measures of the relationship between inputs and outputs fail
to account for non-traditional sources of value. Secondly, if there are significant
lags between cost and benefit, then short-term results look poor but ultimately the
pay-off will be proportionately larger. This would be the case if extensive learn-
ing, by both individuals and organizations, were needed to fully exploit IS, as in
the case for most radically new technologies.
The other two explanations suggest that there really are no major benefits, now
or in the future, and seek to explain why managers would systematically continue
to invest in IS. The redistribution argument suggests that those investing in IS
benefit privately but at the expense of others, so no net benefits show up at the ag-

3
gregate level. The final type of explanation examined is that companies have sys-
tematically mismanaged IS: there is something in its nature that leads firms or in-
dustries to invest in it when they should not, to misallocate it, or to use it to create
slack instead of productivity.
After Brynjolfsson, several other researchers have tried to examine the relation-
ship between investments in IT/IS and their effect on organizational performance.
A plethora of different research methodologies has been adopted by researchers
who have studied this phenomenon from the point of view of different disciplines
like economics, strategy, accounting, operational research and, of course, informa-
tion systems (Melville et al. 2004).
Among approaches tried by researchers, the theory of production has been par-
ticularly useful in conceptualizing the process of production and providing empiri-
cal specifications enabling an estimation of the economic impact of IS (Bryn-
jolfsson and Hitt 1995). Researchers have also employed growth accounting
(Brynjolfsson and Hitt 2003), consumer theory (Brynjolfsson 1996; Hitt and Bryn-
jolfsson 1996), data envelopment analysis (Lee and Barua 1999), conversion ef-
fectiveness (Weill 1992, Soh and Markus 1995), and also divergent theoretical
frameworks (Oh and Pinsonneault 2007).
Melville et al. (2004) identified with a theoretical model that investigations on
IT Business Value have been carried out at three different levels. They call these
levels (from the broader to the narrower): macro environment, competitive envi-
ronment, and focal firm. The first level roughly corresponds to investigations per-
formed at the level of a whole economy. The second level corresponds to the in-
dustry, while the third level focuses only on a single firm, or a part of it.
The impact of IT resources in terms of Business Value has proven to be differ-
ent at the three levels. In brief it can be said that “the more detailed the level of
analysis, the better the chance to detect the impact, if any, of a given technology”
(Kohli and Deveray 2003 p.275).
Many economy-level studies (Baily 1986, Roach 1987, Morrison and Berndt
1991) observed a negative relationship between technology-related variables and
performance. At the industry level, on the other hand, the results are mixed, with
some studies documenting a positive impact of IS (Kelley 1994, Siegel and
Griliches 1992) while others detect no significant advantage to IS investments
(Berndt and Morrison 1995, Koski 1999). Finally, at the more-detailed firm level,
a lot of studies present results that indicate a positive relationship between tech-
nology and performance (Diewert and Smith 1994, Hitt and Brynjolfsson 1995,
Dewan and Min 1997, Menon et al. 2000, Devaraj and Kohli 2000, Lee et al.
2003, Aral et al. 2006).
A good summary of the current state of the art of research on IT Business
Value is provided by Kohli and Grover (2008) who, on the basis of an extensive
literature review, affirm that:
 IT does create value: a consistent mass of studies demonstrate that there is a
relationship between IT and some aspect of firm value;

4
 IT creates value under certain conditions: to produce value, IT must be part
of a business value creation process where it interacts with other IT and or-
ganizational resources;
 IT-based value manifests itself in many ways: studies have shown so far that
IT value can reveal itself in several ways, like productivity improvements,
business processes improvements, profitability, consumer surplus, or advan-
tages in the supply chain;
 IT-based value could be latent: there exists a difference between creating
value and creating differential value;
 There are numerous factors mediating IT and value: there might be latencies
in the manifestation of IT value;
 Causality for IT value is elusive: it is difficult to fully capture and properly
attribute the value generated by IT investments.
Notwithstanding these efforts, research in the IT business value field is far to a
conclusion. Among several different necessities, there is the need of having meth-
odologies or tools to justify the investments in IT on a rigorous basis, and not only
on an emotional and/or empathically way. There is therefore the need of studies
that propose practically applicable framework and methodologies, something that
has also been identified as limitations of several previous works (Leem et al.
2004).
Research Design
The method used for the analysis we are presenting in this paper is a case study,
useful when a phenomenon has to be studied in its natural setting. The unit of
analysis of this case study is formed by the company Rete Ferroviaria Italiana Spa
(RFI), which is part of the group Ferrovie dello Stato, and is the manager of the
Italian railroad infrastructure. RFI is responsible for the tracks, the train stations,
and the installations of the Italian railroad infrastructure. RFI ensures the access to
the Italian railroad network, manages the investments for the upgrading and im-
provement of railway lines and installations, performs the maintenance, ensures
the safe circulation of trains on the whole network, and finally develop and deploy
the technology for the systems and the materials it uses in its daily activities.
Currently, RFI manages 16.300 Km of railroad lines: more than 11.500 Km are
electrified, and over 7.000 Km are double track. The number of train stations and
train stops RFI manages are approximately 2.300. Under the Bluvia trademark,
RFI also ensures the ferry link with the two Italian islands Sicily and Sardinia.
By carrying out actions aimed at developing the Italian railway network, RFI
ensures in Italy an efficient, safe and technologically advanced railway infrastruc-
ture all along the country, involving about 32.000 employees. Such situation has
led to a very extensive company structure and organization. The Company is, in-

5
deed, organized in 6 staff Departments and 4 line Departments, based in Rome,
and in 15 territorial structures. The territorial structures represent the real operat-
ing element of the infrastructure, not only concerning the network but also stations
and operations. The legal headquarter is located in Rome and the territorial struc-
tures are located in Torino, Milano, Genova, Venezia, Verona, Trieste, Bologna,
Pisa, Firenze, Roma, Napoli, Reggio Calabria, Bari, Ancona, Palermo, Cagliari.
The RFI case will be analyzed with the proposition to identify key factors that
can contribute to a successful implementation of a methodology to assess IS/IT
investments return in terms of value.
Case Description
In the last few years the rise in IT expenditures has led to put pressure on the IT
department to evaluate IT investment returns and demonstrate the IT contribution
to the business value. With the aim of answering this request from the top man-
agement, the CIO launched an initiative targeted to develop a system that can sup-
port RFI to evaluate IT investments and to ensure that organization realizes optim-
al value from IT-enabled business investments at an affordable cost with a known
and acceptable level of risk.
The initiative started at the end of 2009 and its focus was both on the invest-
ment decision making process and on the execution. As a matter of fact, the new
system should have enabled RFI to:
 increase the understanding and transparency of IT cost, risks and bene-
fits;
 increase the probability of selecting IT investments with potential high
returns;
 reduce the risk of IT projects failure;
 reduce the likelihood of unexpected events relative to IT cost and deli-
very.
In order to support decision making process, RFI has decided to apply the Eco-
nomic Value Creation (EVC) Methodology which includes models and elements
to effectively and objectively assess the economic value created with investments
in technology and services.
The main characteristic of this methodology is the involvement of executives
who can obtain a better understanding of the assumptions, data, and formulas used
to calculate each benefit and cost of the initiative.
The EVC Methodology evaluates the IT investments return by calculating:
 Financial Metrics;
 Benefit Metrics;
 Cost Metrics;
 Risk Metrics.

6
The Financial Metrics consist in the most known and widespread metrics: Return
of Investment (ROI), Payback Period, Net Present Value (NPV), Internal Rate of
Return (IRR) and Value Added.
The calculus of Benefit Metrics starts with the mapping of the initiative’s fea-
tures and capabilities to operational benefits. Then the analysis of the operational
causes and effects is necessary to determine the means of quantification in each
benefit. Finally each benefit is linked to a measure with its algorithm of calculus.
Benefits considering in the methodology are both qualitative (non cash benefits)
and quantitative (cash benefits). In particular cash benefits are divided in four
profit-impact types, including: Cost Reduction, Cost Avoidance, Revenue In-
crease, and Revenue Protection. The Cost Reduction and Revenue Increase bene-
fits assist in understanding the positive impact on the organization’s cash flow.
The Cost Avoidance and Revenue Protection benefits assist in understanding what
would happen to the bottom line if the investment is not made.
The Cost Metrics assess of the capital and non-capital expenses, as well as the
initial and the ongoing expenses.
Finally, to assess Project Risks, the previous metrics are calculated in three dif-
ferent scenarios: worst case, most likely, and best case. This allows executives to
examine the forecasted results of varying "what if" scenarios.
In order to verify the effectiveness and applicability of the methodology the
project team applied it to two projects: an ongoing project started in 2002 (the Al-
fa project), and a newer project started just started in 2009 (the Beta Project).
The Alfa project attempts to optimize and improve the current regulation sys-
tem through the centralization of information related to infrastructure management
and the rationalization of local resources. Whereas, the aim of the Beta Project is
to support the timetable planning by the implementation of a simulation system in-
tegrated with the RFI Infrastructure database.
The results of the application of the method to the selected projects are shown
in Fig. 1 and Fig. 2.
In particular, the value of benefits comes from the identification and quantifica-
tion of each benefit.
The main benefits of Alfa project are:
 replacement of special paper by standard paper (€5.250.000);
 improvement of the capability of managing contingency cases (changes)
of rail traffic (€700.000);
 reengineering of the regulation broadcast process (from rail station to
centralized garrison) (€61.299.000);
 compliance to Information Security Standards (€300.000).

7
Table 1 – Metrics of Alfa Project
The expected benefits of the Beta Project should instead be the following:
 time reduction to manage requests (€960.000);
 time reduction to include infrastructural changes in Timetable
(€420.000);
 improvement of track graphics (€588.000);
 rationalization of resources for the design and maintenance of Timetable
(€9.120.000).

Table 2 – Metrics of Beta Project
The difference between the Best and Worst Case can give a measure of the risk of
the project and the ability to steer towards the best case under certain conditions
and exploiting opportunities during the project life cycle or the transition into op-
erations. Therefore the project metrics provides a tool for project risk manage-
ment, according to the assumptions on which are based the outcoming cases.
The second part of the initiative focused on the monitoring of project execution
and RFI was especially interested in:
 comparing actual project performance with the project management plan;

8
 assessing performance to determine whether any corrective or preventive
actions were necessary;
 analyzing, tracking, and monitoring project risks to make sure they were
identified and that appropriate risk response plans were being executed;
 maintaining an accurate, timely information base concerning the project’s
outputs;
 providing information to support status reporting, progress measurement,
and forecasting;
 providing forecasts to update current cost and current schedule informa-
tion;
 monitoring implementation of approved changes when and as they occur.
The result of this second part is the Project Monitoring Dashboard (see Figure 3)
that would allow RFI to check if:
 cost is under control;
 schedule is under control;
 scope is managed;
 quality is managed;
 actions are monitored;
 risks are under control.
Figure 3 – Project Monitoring Dashboard
Therefore Project Monitoring Dashboard provides RFI with information on issues
and actions to perform in order to maximize the expected benefits, according to
the project business case.
The initiative finished in April 2010 with high satisfaction, and has subse-
quently driven RFI to start the study how to turn this application into a strategic
tool for managing IT Project Portfolio.

9
Discussion
The experience of the RFI case shows that the need for an effective method to
evaluate the benefits delivered by IS/IT investments in terms of business value
emerges when the amount of these investments grows in size so that the top man-
agement starts to wonder which is the actual return from them. As a result of this
need, in the RFI case, the company has started to address the problem adopting an
evaluation perspective, trying to identify an approach that could be suitable to as-
sess the possible benefits they could have achieved from their IT/IS investments.
The application of the methodology described in the previous paragraph for the
evaluation of the benefits of IT/IS investments has been judged by RFI itself a
success case. The method identified has been applied to two projects. The applica-
tion of this method to the two projects selected has been pursued by RFI with two
main aims: first of all the demonstration of the feasibility of the approach, and se-
condly the identification of the possible informative outcomes stemming out of the
performance measurement system to be implemented.
One of the critical elements that emerged in the RFI case, and that contributed
to the success of the initiative, has been the full commitment of both IT and top
management since the very beginning of the initiative. As a matter of fact, both
the IT side and the business side have been involved in the definition of the me-
thodology to measure qualitative and qualitative benefits from the IT investments.
By doing so, they have given to themselves a sense of responsibility for the final
success of the initiative.
Another relevant element regarding the RFI case consists also in the approach
followed during the implementation of the performance evaluation system. From
the description of the case two steps have been identified. A first step where RFI
aimed at just assessing benefits of the IT investments, and a second step where the
Project Monitoring Dashboard has been developed to monitor the ongoing
progresses. Under this point of view the RFI initiative is not just an ex-ante or ex-
post evaluation exercise, but is a managerial system that provides information re-
garding IT/IS investments both ex-ante (in terms of expected benefits), during the
project (thanks to the usage of the Project Monitoring Dashboard), and ex-post.
Conclusion
This paper describes a case of the implementation of performance measurement
system in RFI, a large Italian company managing the Italian railways infrastruc-
ture. The paper describes a practical implementation of a performance manage-
ment system that is suitable to help the company to identify the returns, in term of
value, of IT/IS investments.

10
The case described in this paper has indicated as key aspects necessary for a
successful implementation of a performance measurement system: the awareness
of the importance to evaluate and assess the return on IT/IS investments in terms o
value, the commitment of both the IT management and the top management, the
pilot approach to test the feasibility of the initiative and the outcome in terms of
informative potential of the methodology to be adopted preceding a full scale im-
plementation, and the involvement of the business side in the evaluation of IT/IS
investments.
The methodology adopted to evaluate the benefits, in terms of value, of the IT
investments in these projects has been so far implemented at the level of pilot ex-
perience. As already mentioned, the success of these pilot experiences convinced
RFI to implement this methodology as an operational tool to support IT/IS in-
vestments planning and for the portfolio management. Since this step has not been
completed, so far, this could be a partial limitation of the current research. In order
to overpass this limitation, and to deepen the understanding of the case, further re-
search activities will be planned in the near future.
References
Aral S., Brynjolfsson E., Wu D.J. (2006). Which Came First, IT or Productivity? The Virtuous
Cycle of Investment and Use in Enterprise Systems, Twenty-Seventh International Confer-
ence on Information Systems, 1819-1840.
Berndt, E., Morrison C. (1995). High-Tech Capital Formation and Economic Performance in
U.S. Manufacturing Industries: An Exploratory Analysis, Journal of Econometrics 65: 9-43.
Brynjolfsson E. (1993). The Productivity Paradox of IT, Communications of the ACM, 36(12):
66-77.
Brynjolfsson E. (1996) The Contribution of Information Technology to Consumer Welfare, In-
formation Systems Research, 7(3): 281-300.
Brynjolfsson E., Hitt L. (1995). Information Technology as a Factor of Production: The Role of
Differences Among Firms, Economics of Innovation and New Technology, 3(4): 183-200.
Brynjolfsson E., Hitt L. (2003). Computing Productivity: Firm-Level Evidence, Review of Eco-
nomics and Statistics, 85(4): 793-808.
Carr N. (2003). IT doesn’t matter, Harvard Business Review, 81(5): 41-49.
Dewan S., Min C. (1997). The Substitution of Information Technology for Other Factors of Pro-
duction: A Firm Level Analysis, Management Science, 43(12): 1660-1675.
Devaraj S., Kohli R. (2000). Information technology payoff in the healthcare industry: A longi-
tudinal study. Journal Management Information Systems, 16(4): 41-67.
Diewert E.W., Smith A.M. (1994). Productivity measurement for a distribution firm. National
Bureau of Economic Research working paper no. 4812, Washington, D.C.
Gable, G., Sedera, D., Chan, T., (2008). Re-conceptualizing System Success: the IS-impact
Measurement Model, Journal of the Association for Information Systems, 9(7), 377-408.
Harris S.E., Katz J.L. (1991) Organizational performance and information technology investment
intensity in the insurance industry, Organization Science, 2(3): 263-295.
Hitt L., Brynjolfsson E. (1995). Productivity, business profitability, and consumer surplus: Three
different measures of information technology value. MIS Quarterly, 20(2): 121–142.
Hitt L., Brynjolfsson E. (1996). Paradox Lost? Firm-Level Evidence on the Returns to Informa-
tion Systems Spending, Management Science, 42(4): 541-558.

11
Kelley M. (1994). Productivity and information technology: The elusive connection. Manage-
ment Science, 40(11): 1406–1425.
Kohli R., Devaraj S. (2003). Measuring information technology payoff: A meta-analysis of struc-
tural variables in firm-level empirical research. Information Systems Research, 17(3): 198-
227.
Kohli R., Grover V. (2008). Business Value of IT: an essay on expanding research directions to
keep up with the times, Journal of the Association for Information Systems, 9(1): 23-39.
Koski H. 1999. The implications of network use, production network externalities and public
networking programmes for firm’s productivity. Research Policy, 28(4): 423–439.
Lee B., Barua A. (1999). An Integrated Assessment of Productivity and Efficiency Impacts of In-
formation Technology Investments: Old Data, New Analysis and Evidence, Journal of Pro-
ductivity Analysis, 12(1): 21-43.
Lee H., Choi B. (2003) Knowledge Management Enablers, Processes, and Organizational Per-
formance: An Integrative View and Empirical Examination, Journal of Management Infor-
mation Systems, 20(1): 179-228.
Leem C.S.m Yoon C.Y., Park S.K. (2004). A process-centered IT ROI analysis with a case
study, Information Systems Frontiers, 6(4): 369-383.
Melville N., Kraemer K.L., and Gurbaxani, V. (2004). Review – Information Technology and
organizational performance: an integrative model of IT Business Value. MIS Quarterly,
28(2): 283-322.
Menon N.M.B., Lee L. Eldenburg. (2000). Productivity of information systems in the healthcare
industry. Information Systems Research, 11(1): 83–92.
Morrison C., E. Berndt. (1991). Assessing the productivity of information technology equipment
in U.S. manufacturing industries. National Bureau of Economic Research working paper no.
3582, Washington, D.C.
Oh W, and Pinsonneault, A. (2007). On the assessment of the strategic value of information
technologies: conceptual and analytical approaches. MIS Quarterly, 31(2): 239-265.
Rai A., Patnayakuni, R., and Seth, N. (2006) Firm Performance Impacts of Digitally-enabled
Supply Chain Integration Capabilities, MIS Quarterly, 30(2): 225-246.
Roach, S. (1987). America’s technology dilemma: A profile of the information economy. Special
Economic Study, Morgan Stanley, New York.
Scheepers H, Scheepers R. (2008). A process-focused decision framework for analyzing the
Business Value potential of IT investments, Information Systems Frontiers, 10(3): 321-330.
Siegel D.Z. Griliches. (1992). Purchased services, outsourcing, computers, and productivity in
manufacturing. In Griliches Z. (eds.) Output Measurement in the Service Sectors. University
of Chicago Press, Chicago, 429–458.
Soh C., Markus M.L. (1995). How IT creates business value: a process theory synthesis, In Ariav
G., Beath C., DeGross J.I., Hoyer R., Kemerer, C.F. (Eds), Proceedings of the 16th Interna-
tional Conference on Information Systems, ACM, Amsterdam, pp. 29-42.
Solow R.S. (1987) We’d better watch out, New York Times Book Review.
Strassmann P. (1990) The Business Value of Computers, The Information Economics Press.
Tallon P.P (2007) Does IT pay to focus? An analysis of IT Business Value under single and
multi-focused business strategies, Journal of Strategic Information Systems, 16(3): 278-300.
Weill P. (1992). The relationship between investment in information technology and firm per-
formance: A study of the valve manufacturing sector, Information Systems Research, 3(4):
307-333.