Cicero T, Malgarini M, Nappi CA and Peracchi F
MPRA (Munich Personal REPEc Archive)
Bibliometric and peer review methods for research evaluation: a methodological appraisement
Cicero T, Malgarini M and Benedetto S
Proceedings of Science and Technology Indicators Conference 2014
Research quality, characteristics of publications and socio-demographic features of Universities and Researchers: evidence from the Italian VQR 2004-2010 evaluation exercise
Di Cristina F
, 2: 42
RISULTATI E PROSPETTIVE DELLA VALUTAZIONE DELLA RICERCA NELL’AREA GIURIDICA
Abramo G, Cicero T and D'Angelo C
Journal of Informetrics
Are the authors of highly cited articles also the most productive ones?
Ever more frequently, governments have decided to implement policy measures intended to foster and reward excellence in scientific research. This is in fact the intended purpose of national research assessment exercises. These are typically based on the analysis of the quality of the best research products; however, a different approach to analysis and intervention is based on the measure of productivity of the individual scientists, meaning the overall impact of their entire scientific production over the period under observation. This work analyzes the convergence of the two approaches, asking if and to what measure the most productive scientists achieve highly cited articles; or vice versa, what share of highly cited articles is achieved by scientists that are “non-top” for productivity. To do this we use bibliometric indicators, applied to the 2004–2008 publications authored by academics of Italian universities and indexed in the Web of Science.
Ferrara A and Bonaccorsi A
How robust is journal rating in Humanities and Social Sciences? Evidence from a large-scale, multi-method exercise
This article reports on a large-scale exercise of classification of journals in the fields of Humanities and Social Sciences, carried out by the Italian Agency for the Evaluation of Universities and Research Institutes. After discussing at some length the controversies linked with journal classification and its impact, we endeavor to compare such a classification with the scores that individual articles published in the same journals were assigned by completely independent assessors in the same period of time. The data refer to an important subset of disciplines covering History, Philosophy, Geography, Anthropology, Education, and Library Sciences, allowing for comparisons between scientific fields of different sizes, outlooks, and methods. As the controversies surrounding the rating of journals focus on the difference between the container (the journal) and the content (the individual article), we addressed the following research questions: (1) Is journal rating, produced by an expert-based procedure, a good predictor of the quality of articles published in the journal? (2) To what extent different panel of experts evaluating the same journals produce consistent ratings? (3) To what extent the assessment of scientific societies on journal rating is a good predictor of the quality of articles published in journals? (4) Are there systematic biases in the peer review of articles and in the expert-based journal rating? We find that journal rating is a legitimate and robust assessment exercise, as long as it is managed carefully and in a cautious way and used to evaluate aggregates of researchers rather than individual researchers.
Abramo G, Cicero T and D'Angelo CA
Journal of Informetrics
Should the research performance of scientists be distinguished by gender?
The literature on gender differences in research performance seems to suggest a gap between men and women, where the former outperform the latter. Whether one agrees with the different factors proposed to explain the phenomenon, it is worthwhile to verify if comparing the performance within each gender, rather than without distinction, gives significantly different ranking lists. If there were some structural factor that determined a penalty in performance of female researchers compared to their male peers, then under conditions of equal capacities of men and women, any comparative evaluations of individual performance that fail to account for gender differences would lead to distortion of the judgments in favor of men. In this work we measure the extent of differences in rank between the two methods of comparing performance in each field of the hard sciences: for professors in the Italian university system, we compare the distributions of research performance for men and women and subsequently the ranking lists with and without distinction by gender. The results are of interest for the optimization of efficient selection in formulation of recruitment, career advancement and incentive schemes.
Malgarini M, Nappi CA and Torrini R
Proceedings of ISSI Conference
Article and Journal-Level Metrics in Massive Research Evaluation Exercises: The Italian Case
Ancaiani A, Anfossi AF, Barbara A, Benedetto S, Blasi B, Carletti V, Cicero T, Ciolfi A, Costa F, Colizza G, Costantini M, di Cristina F, Ferrara A, Lacatena RM, Malgarini M, Mazzotta I, Nappi CA, Romagnosi S and Sileoni S
, 24(3): 242-255
Evaluating scientific research in Italy: The 2004-10 research evaluation exercise
The Italian Research Evaluation assessment for the period 2004-10 (VQR 2004-10) has analyzed almost 185,000 articles, books, patents, and other scientific outcomes submitted for evaluation by Italian universities and other public research bodies. This article describes the main features of this exercise, introducing its legal framework and the criteria used for evaluation. The innovative methodology that has been used for evaluation, based on a combination of peer review and bibliometric methods, is discussed and indicators for assessing the quality of participating research bodies are derived accordingly. The article also presents the main results obtained at the University level, trying to understand the existing relationship among research quality and University characteristics such as location, dimension, age, scientific specialization, and funding.
Montanaro P and Torrini R
Bank of Italy Occasional Paper
Il Sistema Della Ricerca Pubblica in Italia (The Italian System of Public Research)
Nel settore della ricerca pubblica l'Italia investe meno della media europea. Se rapportato alle risorse impegnate e ai ricercatori, l'output risulta però elevato e la sua qualità media, condotta presso università ed enti di ricerca, non è molto lontana rispetto a paesi prossimi come la Francia, anche se con difficoltà di affermazione nelle punte più avanzate. Il sistema italiano, assai articolato e frammentato nei soggetti che vi operano e nelle fonti di finanziamento, risente di una scarsa attitudine all'applicazione dei risultati e alla collaborazione con le imprese, che a loro volta investono poco e incontrano difficoltà a collegare la propria attività di ricerca con gli input provenienti dai centri di ricerca pubblica. Il sistema sconta inoltre la mancanza di una chiara strategia che stabilisca gli obiettivi da raggiungere, disegni missioni e modelli organizzativi delle strutture di ricerca coerenti con gli obiettivi individuati e definisca le risorse necessarie al loro raggiungimento. La pressante necessità di un rilancio della capacità innovativa del Paese, infatti, non può prescindere da un sistema della ricerca pubblica adeguatamente finanziato ed efficientemente governato.
Ancaiani A and Fantoni S
Melanges De La Casa De Velazquez
, 44(2): 313-318
The evaluation of the products of research and its impact on the university system and the institutions of research: the italian case