Skip to main navigation menu Skip to main content Skip to site footer

Review Articles

Vol. 12 No. 3 (2025)

Assessing the disparities in impact assessment methodologies: A multidimensional context

DOI
https://doi.org/10.14719/pst.7808
Submitted
19 February 2025
Published
19-06-2025 — Updated on 01-07-2025
Versions

Abstract

To assess the efficacy and performance of an intervention or policy, impact assessment plays a crucial role. This article aims to systematically combine the existing literatures published on impact assessment methodologies using a narrative approach. Databases such as Springer, Google Scholar and Science Direct are used to identify the literature sources on different impact assessment methodologies such as Difference in Difference (DID), Propensity Score Matching, Instrumental Variable Analysis, Randomized Controlled Trial, Regression Discontinuity Design and Synthetic Control Method. Although these techniques have always been significant, their implementation varies across scientific domains due to challenges such as resource constraints and methodological complexities. To address current global issues and enhance the precision of methodologies across field, recognizing the contextual relevance of each method is essential. Each methodology offers unique characteristic uses and is characterized to solve various research issues. This helps in decision making strategies for future-oriented programs. The study emphasizes the portability and wider interdisciplinary uses of these approaches by investigating their practical application in several domains such as health, education and the environment. The limitations, challenges and intrinsic biases associated with different methodologies are analyzed and discussed. The article evaluates various approaches in assessing the impact, thereby aiding the reader to understand the appropriate methodologies for different conditions.

References

  1. 1. Hulme D. Impact assessment methodologies for microfinance: Theory, experience and better practice. World Development. 2000;28(1):79-98. https://doi.org/10.1016/S0305-750X(99)00119-9
  2. 2. Pope J, Bond A, Morrison-Saunders A, Retief F. Advancing the theory and practice of impact assessment: Setting the research agenda. Environmental Impact Assessment. 2013;41:1-9. https://doi.org/10.1016/j.eiar.2013.01.008
  3. 3. Singh DK, Satpathy B. Advanced Innovations in Agricultural Extension and Development. 2024.
  4. 4. Reed MS, Ferré M, Martin-Ortega J, Blanche R, Lawford-Rolfe R, Dallimer M, et al. Evaluating impact from research: A methodological framework. Research Policy. 2021;50(4):104147. https://doi.org/10.1016/j.respol.2020.104147
  5. 5. Pandey L. Impact assessment of agricultural technology from simple efficiency analysis to sustainable livelihood framework. 2004.
  6. 6. Anandajayasekeram P, Rukani M, Babu S. Impact of science on African agriculture and food security: Centre for Agriculture and Biosciences International. 2007. https://doi.org/10.1079/9781845932671.0000
  7. 7. Wood C, editor. Environmental impact assessment in developing countries: An overview. Conference on new directions in impact assessment for development: Methods and practice; 2003: Environmental Impact Assessment Centre School of Planning and Landscape, University of Manchester. https://doi.org/10.3828/idpr.25.3.5
  8. 8. Anandajayasekeram P, Babu S. Overview of impact assessment methodologies. Impact of science on African agriculture and food security. 2007:249. https://doi.org/10.1079/9781845932671.0249
  9. 9. Bond A, Dusík J. Impact assessment for the twenty-first century–rising to the challenge. Impact Assessment and Project Appraisal. 2020;38(2):94-9. https://doi.org/10.1080/14615517.2019.1677083
  10. 10. Burritt R, Christ K. Industry 4.0 and environmental accounting: a new revolution? Asian Journal of Sustainability and Social Responsibility. 2016;1:23-38. https://doi.org/10.1186/s41180-016-0007-y
  11. 11. Burdge RJ. Benefiting from the practice of social impact assessment. Impact Assessment and Project Appraisal. 2003;21(3):225-9. https://doi.org/10.3152/147154603781766284
  12. 12. Esteves AM, Franks D, Vanclay F. Social impact assessment: the state of the art. Impact Assessment and Project Appraisal. 2012;30(1):34-42. https://doi.org/10.1080/14615517.2012.660356
  13. 13. Duinker PN, Greig LA. Scenario analysis in environmental impact assessment: Improving explorations of the future. Environmental Impact Assessment Review. 2007;27(3):206-19. https://doi.org/10.1016/j.eiar.2006.11.001
  14. 14. Alomoto W, Niñerola A, Pié L. Social impact assessment: A systematic review of literature. Social Indicators Research. 2022;161(1):225-50. https://doi.org/10.1007/s11205-021-02809-1
  15. 15. Nita A. Empowering impact assessments knowledge and international research collaboration-A bibliometric analysis. Environmental Impact Assessment Review. 2019;78:106283. https://doi.org/10.1016/j.eiar.2019.106283
  16. 16. Strömmer K, Ormiston J. Forward-looking impact assessment–An interdisciplinary systematic review and research agenda. Journal of Cleaner Production. 2022;377:134322. https://doi.org/10.1016/j.jclepro.2022.134322
  17. 17. Lockie S. SIA in review: setting the agenda for impact assessment in the 21st century. Impact Assessment and Project Appraisal. 2001;19(4):277-87. https://doi.org/10.3152/147154601781766952
  18. 18. Snow J. On the mode of communication of cholera. British Politics and the Environment in the Long Nineteenth Century: Routledge. 2023:149-54.
  19. 19. Lechner M. The estimation of causal effects by difference-in-difference methods. Foundations and Trends® in Econometrics. 2011;4(3):165-224. http://doi.org/10.1561/0800000014
  20. 20. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41-55. https://doi.org/10.1093/biomet/70.1.41
  21. 21. Dehejia RH, Wahba S. Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association. 1999;94(448):1053-62. https://doi.org/10.1080/01621459.1999.10473858
  22. 22. Hausman JA, Taylor WE. Panel data and unobservable individual effects. Econometrica: Journal of the Econometric Society. 1981:1377-98. https://doi.org/10.2307/1911406
  23. 23. Amemiya T, MaCurdy TE. Instrumental-variable estimation of an error-components model. Econometrica: Journal of the Econometric Society. 1986:869-80. https://doi.org/10.2307/1912840
  24. 24. Hill AB. The clinical trial. New England Journal of Medicine. 1952;247(4):113-9.
  25. 25. Reith C, Landray M, Devereaux P, Bosch J, Granger CB, Baigent C, et al. Randomized clinical trials–removing unnecessary obstacles. N Engl J Med. 2013;369(11):1061-5.
  26. 26. Thistlethwaite DL, Campbell DT. Regression-discontinuity analysis: An alternative to the ex post facto experiment. Journal of Educational psychology. 1960;51(6):309. https://psycnet.apa.org/doi/10.1037/h0044319
  27. 27. Jacob R, Zhu P, Somers M-A, Bloom H. A practical guide to regression discontinuity. MDRC. 2012.
  28. 28. Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association. 2010;105(490):493-505. https://doi.org/10.1198/jasa.2009.ap08746
  29. 29. Morgan RK. Environmental impact assessment: the state of the art. Impact Assessment and Project Appraisal. 2012;30(1):5-14. https://doi.org/10.1080/14615517.2012.661557
  30. 30. Callaway B, Sant’Anna PH. Difference-in-differences with multiple time periods. Journal of Econometrics. 2021;225(2):200-30. https://doi.org/10.1016/j.jeconom.2020.12.001
  31. 31. McCaffrey DF, Griffin BA, Almirall D, Slaughter ME, Ramchand R, Burgette LF. A tutorial on propensity score estimation for multiple treatments using generalized boosted models. Statistics in Medicine. 2013;32(19):3388-414. https://doi.org/10.1002/sim.5753
  32. 32. Lousdal ML. An introduction to instrumental variable assumptions, validation and estimation. Emerging Themes in Epidemiology. 2018;15(1):1. https://doi.org/10.1186/s12982-018-0069-7
  33. 33. Song JW, Chung KC. Observational studies: Cohort and case-control studies. Plastic and Reconstructive Surgery. 2010;126(6):2234-42. https://doi.org/10.1097/PRS.0b013e3181f44abc
  34. 34. Lee DS, Lemieux T. Regression discontinuity designs in economics. Journal of Economic Literature. 2010;48(2):281-355. https://doi.org/10.1257/jel.48.2.281
  35. 35. Dimick JB, Ryan AM. Methods for evaluating changes in health care policy: the difference-in-differences approach. Journal of the American Medical Association. 2014;312(22):2401-2. https://doi.org/10.1001/jama.2014.16153
  36. 36. Angrist JD, Pischke J-S. Mostly harmless econometrics: An empiricist's companion: Princeton university press. 2009.
  37. 37. Abadie A. Semiparametric difference-in-differences estimators. The Review of Economic Studies. 2005;72(1):1-19. https://doi.org/10.1111/0034-6527.00321
  38. 38. Becker SO, Ichino A. Estimation of average treatment effects based on propensity scores. The Stata Journal. 2002;2(4):358-77. https://doi.org/10.1177/1536867X0200200403
  39. 39. Leuven E, Sianesi B. PSMATCH2: Stata module to perform full Mahalanobis and propensity score matching, common support graphing and covariate imbalance testing. 2018.
  40. 40. Witte KD, López-Torres L. Efficiency in education: A review of literature and a way forward. Journal of the Operational Research Society. 2017;68:339-63. https://doi.org/10.1057/jors.2015.92
  41. 41. Wing C, Simon K, Bello-Gomez RA. Designing difference in difference studies: best practices for public health policy research. Annual Review of Public Health. 2018;39:453-69. https://doi.org/10.1146/annurev-publhealth-040617-013507
  42. 42. Goodman-Bacon A. Difference-in-differences with variation in treatment timing. Journal of Econometrics. 2021;225(2):254-77. https://doi.org/10.1016/j.jeconom.2021.03.014
  43. 43. Ryan AM, Burgess Jr JF, Dimick JB. Why we should not be indifferent to specification choices for difference in differences. Health Services Research. 2015;50(4):1211-35. https://doi.org/10.1111/1475-6773.12270
  44. 44. Cataife G, Pagano MB. Difference in difference: simple tool, accurate results, causal effects. Transfusion. 2017;57(5):1113-4. https://doi.org/10.1111/trf.14063
  45. 45. Puhani PA. The treatment effect, the cross difference, and the interaction term in nonlinear “difference-in-differences” models. Economics Letters. 2012;115(1):85-7. https://doi.org/10.1016/j.econlet.2011.11.025
  46. 46. Athey S, Imbens GW. Identification and inference in nonlinear difference in differences models. Econometrica. 2006;74(2):431-97. https://doi.org/10.1111/j.1468-0262.2006.00668.x
  47. 47. Austin PC. Some methods of propensity-score matching had superior performance to others: results of an empirical investigation and Monte Carlo simulations. Biometrical Journal: Journal of Mathematical Methods in Biosciences. 2009;51(1):171-84. https://doi.org/10.1002/bimj.200810488
  48. 48. Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research. 2011;46(3):399-424. https://doi.org/10.1080/00273171.2011.568786
  49. 49. Koloma Y, Alia H. Gendered impact of microcredit in Mali: an evaluation by propensity score matching. 2014.
  50. 50. Heinze G, Jüni P. An overview of the objectives of and the approaches to propensity score analyses. European Heart Journal. 2011;32(14):1704-8. https://doi.org/10.1093/eurheartj/ehr031
  51. 51. Gimenez-Nadal JI, Molina JA. Commuting time and household responsibilities: Evidence using propensity score matching. Journal of Regional Science. 2016;56(2):332-59. https://doi.org/10.1111/jors.12243
  52. 52. Caliendo M, Kopeinig S. Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys. 2008;22(1):31-72. https://doi.org/10.1111/j.1467-6419.2007.00527.x
  53. 53. Haukoos JS, Lewis RJ. The propensity score. Journal of the American Medical Association. 2015;314(15):1637-8. https://doi.org/10.1001/jama.2015.13480
  54. 54. Benedetto U, Head SJ, Angelini GD, Blackstone EH. Statistical primer: propensity score matching and its alternatives. European Journal of Cardio-Thoracic Surgery. 2018;53(6):1112-7. https://doi.org/10.1093/ejcts/ezy167
  55. 55. Baiocchi M, Cheng J, Small DS. Instrumental variable methods for causal inference. Statistics in Medicine. 2014;33(13):2297-340. https://doi.org/10.1002/sim.6128
  56. 56. Stel VS, Dekker FW, Zoccali C, Jager KJ. Instrumental variable analysis. Nephrology Dialysis Transplantation. 2013;28(7):1694-9. https://doi.org/10.1093/ndt/gfs310
  57. 57. Martens EP, Pestman WR, de Boer A, Belitser SV, Klungel OH. Instrumental variables: Application and limitations. Epidemiology. 2006;17(3):260-7. https://doi.org/10.1097/01.ede.0000215160.88317.cb
  58. 58. Bollen KA. Instrumental variables in sociology and the social sciences. Annual Review of Sociology. 2012;38:37-72. https://doi.org/10.1146/annurev-soc-081309-150141
  59. 59. Larcker DF, Rusticus TO. On the use of instrumental variables in accounting research. Journal of Accounting and Economics. 2010;49(3):186-205. https://doi.org/10.1016/j.jacceco.2009.11.004
  60. 60. Swamy V. Financial inclusion, gender dimension, and economic impact on poor households. World Development. 2014;56:1-15. https://doi.org/10.1016/j.worlddev.2013.10.019
  61. 61. Bhide A, Shah PS, Acharya G. A simplified guide to randomized controlled trials. Acta Obstetricia et Gynecologica Scandinavica. 2018;97(4):380-7. https://doi.org/10.1111/aogs.13309
  62. 62. Stolberg HO, Norman G, Trop I. Randomized controlled trials. American Journal of Roentgenology. 2004;183(6):1539-44. https://doi.org/10.2214/ajr.183.6.01831539
  63. 63. Silverman SL. From randomized controlled trials to observational studies. The American Journal of Medicine. 2009;122(2):114-20. https://doi.org/10.1016/j.amjmed.2008.09.030
  64. 64. Spieth PM, Kubasch AS, Penzlin AI, Illigens BM-W, Barlinn K, Siepmann T. Randomized controlled trials–a matter of design. Neuropsychiatric Disease and Treatment. 2016:1341-9. https://doi.org/10.2147/NDT.S101938
  65. 65. Aiken LS, West SG, Schwalm DE, Carroll JL, Hsiung S. Comparison of a randomized and two quasi-experimental designs in a single outcome evaluation: Efficacy of a university-level remedial writing program. Evaluation Review. 1998;22(2):207-44. https://doi.org/10.1177/0193841X9802200203
  66. 66. Bloom HS. Modern regression discontinuity analysis. Journal of Research on Educational Effectiveness. 2012;5(1):43-82. https://doi.org/10.1080/19345747.2011.578707
  67. 67. Chambers S. Regression discontinuity design: A guide for strengthening causal inference in HRD. European Journal of Training and Development. 2016;40(8/9):615-37.
  68. 68. Lee H, Munk T, editors. Using regression discontinuity design for program evaluation. Proceedings of the 2008 joint statistical meeting. American Statistical Association Alexandria. 2008.
  69. 69. Hahn J, Todd P, Van der Klaauw W. Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica. 2001;69(1):201-9.
  70. 70. Botosaru I, Ferman B. On the role of covariates in the synthetic control method. The Econometrics Journal. 2019;22(2):117-30. https://doi.org/10.1093/ectj/utz001
  71. 71. Adhikari B, Alm J. Evaluating the economic effects of flat tax reforms using synthetic control methods. Southern Economic Journal. 2016;83(2):437-63. https://doi.org/10.1002/soej.12152
  72. 72. Ayala L, Martín-Román J, Navarro C. Unemployment shocks and material deprivation in the European Union: A synthetic control approach. Economic Systems. 2023;47(1):101053. https://doi.org/10.1016/j.ecosys.2022.101053
  73. 73. Kaul A, Klößner S, Pfeifer G, Schieler M. Synthetic control methods: Never use all pre-intervention outcomes together with covariates. 2015.
  74. 74. Billmeier A, Nannicini T. Assessing economic liberalization episodes: A synthetic control approach. Review of Economics and Statistics. 2013;95(3):983-1001. https://doi.org/10.1162/REST_a_00324
  75. 75. Galiani S, Quistorff B. The synth_runner package: Utilities to automate synthetic control estimation using synth. The Stata Journal. 2017;17(4):834-49. https://doi.org/10.1177/1536867X1801700404
  76. 76. Oyarzo M, Ferrada LM. Exploring the impact of employment policies on wages and employability in the Chilean local labor market. Regional Science Policy & Practice. 2024:100048. https://doi.org/10.1016/j.rspp.2024.100048
  77. 77. Alatas V, Cameron LA. The impact of minimum wages on employment in a low income country: an evaluation using the difference-in-differences approach. SSRN 636347. 2003.
  78. 78. Yue Q, Song Y, Zhang M, Zhang X, Wang L. The impact of air pollution on employment location choice: Evidence from China's migrant population. Environmental Impact Assessment Review. 2024;105:107411. https://doi.org/10.1016/j.eiar.2023.107411
  79. 79. Mendola M. Agricultural technology adoption and poverty reduction: A propensity-score matching analysis for rural Bangladesh. Food Policy. 2007;32(3):372-93. https://doi.org/10.1016/j.foodpol.2006.07.003
  80. 80. Awotide BA, Fashogbon A, Awoyemi TT, editors. Impact of agro-idustrial development strategies on smallholder rice farmers’ productivity, income and poverty: The case of contract farming in Nigeria.". International Conference of the Centre for the Studies of African Economies (CSAE), Oxford. 2015.
  81. 81. Bellemare MF. As you sow, so shall you reap: The welfare impacts of contract farming. World Development. 2012;40(7):1418-34. https://doi.org/10.1016/j.worlddev.2011.12.008
  82. 82. Zhang H. Literature review on the impact of individual income tax reform on residents' consumption. Scientific Journal of Economics and Management Research Volume. 2021;3(8).
  83. 83. Domguia EN. Taxing for a better life? The impact of environmental taxes on income distribution and inclusive education. Heliyon. 2023.
  84. 84. Nam LP, Van Song N, Quilloy AJA, Rañola RF, Camacho Jr JV, Camacho LD, et al. Assessment of impacts of adaptation measures on rice farm economic performance in response to climate change: Case study in Vietnam. Environment, Development and Sustainability. 2023:1-29. https://doi.org/10.1007/s10668-023-04301-x
  85. 85. Mideksa B, Muluken G, Eric N. The impact of soil and water conservation practices on food security in eastern Ethiopia. A propensity score matching approach. Agricultural Water Management. 2023;289:108510. https://doi.org/10.1016/j.agwat.2023.108510
  86. 86. Jambo Y, Alemu A, Tasew W. Impact of small-scale irrigation on household food security: Evidence from Ethiopia. Agriculture & Food Security. 2021;10:1-16. https://doi.org/10.1186/s40066-021-00294-w
  87. 87. Kazemi Zaroomi H, Jafari Samimi A, Karimi Potanlar S. The impact of inflation targeting on direct taxes in selected countries: A propensity score matching (psm) approach. International Journal of New Political Economy. 2020;1(2):133-51. https://doi.org/10.29252/jep.1.2.133
  88. 88. Odjidja EN, Ansah-Akrofi R, Iradukunda A, Kwanin C, Saha M. The effect of health financing reforms on incidence and management of childhood infections in Ghana: A matching difference in differences impact evaluation. BMC Public Health. 2022;22(1):1494. https://doi.org/10.1186/s12889-022-13934-y
  89. 89. Bouttell J, Craig P, Lewsey J, Robinson M, Popham F. Synthetic control methodology as a tool for evaluating population-level health interventions. J Epidemiol Community Health. 2018;72(8):673-8. https://doi.org/10.1136/jech-2017-210106
  90. 90. Chen H, Li Q, Kaufman JS, Wang J, Copes R, Su Y, et al. Effect of air quality alerts on human health: a regression discontinuity analysis in Toronto, Canada. The Lancet Planetary Health. 2018;2(1):e19-e26.
  91. 91. Qin VM, Hone T, Millett C, Moreno-Serra R, McPake B, Atun R, et al. The impact of user charges on health outcomes in low-income and middle-income countries: A systematic review. BMJ Global Health. 2019;3(Suppl 3):e001087.
  92. 92. Liu J, Li Q. Impact of major events on interprovincial carbon emissions-based on PSM-DID analysis. Sustainability. 2022;14(12):7459. https://doi.org/10.3390/su14127459
  93. 93. Fan X, Xu Y, Nan Y, Li B, Cai H. Impacts of high-speed railway on the industrial pollution emissions in China: Evidence from multi-period difference-in-differences models. Kybernetes. 2020;49(11):2713-35. https://doi.org/10.1108/K-07-2019-0499
  94. 94. Muriu PW. The impact of microfinance programs: A review of data and methodological dilemma. Journal of Economics and Sustainable Development. 2020;11(22):100-9. https://doi.org/10.7176/JESD/11-22-09
  95. 95. Rodriguez Z. The power of employment: Effects of India’s employment guarantee on women empowerment. World Development. 2022;152:105803. https://doi.org/10.1016/j.worlddev.2021.105803
  96. 96. Balestri S. Revitalising smallholder agriculture: The impact of technical training in rural Lebanon. Economia Politica. 2024:1-33. https://doi.org/10.1007/s40888-024-00329-y
  97. 97. Soumaré I, Kanga D, Tyson J. Impact assessment for market-building interventions in capital markets: Review, Critiques and Improvements. 2020.
  98. 98. Dey MM, Paraguas FJ, Kambewa P, Pemsl DE. The impact of integrated aquaculture-agriculture on small-scale farms in Southern Malawi. Agricultural Economics. 2010;41(1):67-79. https://doi.org/10.1111/j.1574-0862.2009.00426.x
  99. 99. Walker TS, Alwang J. Crop improvement, adoption and impact of improved varieties in food crops in sub-Saharan Africa: Cabi; 2015. https://doi.org/10.1079/9781780644011.0000

Downloads

Download data is not yet available.