Can We Evaluate Excellence Differently?
The Ranking Trap - Part 3
University evaluation must look beyond league tables to recognise institutional progress and the potential to turn limited resources into lasting excellence.
Suppose two universities are being evaluated. One is a century-old institution with distinguished faculty, excellent laboratories and a strong international reputation. The other is relatively young, operating in a less-developed region with modest resources but steadily improving its teaching, research and community engagement.
Most ranking systems would probably place the established university ahead. But suppose we asked different questions. Which institution has improved more? Which uses its resources more effectively? Which contributes more to its surrounding community? Which could achieve the greatest improvement with additional support?
In the first two articles of this series, I examined what rankings overlook and how they acquire influence. The challenge now is to find a better approach.
The answer is not to abandon measurement. It is to stop expecting one ranking to answer questions that are fundamentally different.
A student choosing a university wants to know where she will receive a good education. A parent may be concerned about affordability and employment opportunities. A funding agency wants to identify promising research. A government must decide where public investment will make the greatest difference.
A university primarily devoted to undergraduate teaching should be assessed for student learning and intellectual development. A research institution should demonstrate originality, rigour and scientific significance. An institution serving an underserved region should receive recognition for the opportunities it creates and the problems it helps solve.
This does not mean lowering standards or replacing rankings with another universal score constructed from different indicators.
Recognising Achievement
Our present systems often emphasise where an institution stands today. We should also ask how far it has travelled and what it could achieve.
Imagine a university that has improved teaching, established laboratories, strengthened its faculty and developed useful collaborations despite limited resources. Its national ranking may barely change. Yet its progress could be remarkable.
Three dimensions deserve attention: present achievement, improvement over time and future potential.
Established centres of excellence deserve continued support. Advanced research often requires sophisticated facilities and experienced teams. But promising institutions also need opportunities to develop capabilities they do not yet possess.
Here we must distinguish evaluation from funding allocation. Evaluation asks what an institution has achieved and what it is capable of doing. Funding allocation asks what a particular investment is intended to accomplish.
A highly capable institution may deserve support to undertake advanced research. A less-developed institution may deserve investment precisely because it has identifiable weaknesses that can realistically be overcome.
Both can be justified, but not by the same simplistic ranking. Research funding could therefore use complementary pathways: one supporting demonstrated excellence and another nurturing emerging capability. Each would require rigorous scrutiny and accountability, but they would serve different purposes. A funding system should not confuse the absence of past opportunity with the absence of future promise.
Reforms Needed
The evaluation of individual scientists also needs reform. Instead of beginning with publication counts, journal impact factors or h-indices, why not ask researchers to identify their three to five most significant contributions?
They could explain the problem addressed, what was original, the evidence supporting its importance and what difference it made.
For one scientist, the contribution might be a fundamental discovery. For another, it could be a new instrument, an industrial process, an improved crop variety, an important dataset or a technology benefiting society.
Mentoring young researchers, developing scientific facilities and enabling others to undertake valuable research should also receive appropriate recognition.
Publications and citations would remain useful evidence. But they would support judgment rather than replace it.
The Leiden Manifesto, the San Francisco Declaration on Research Assessment and the Coalition for Advancing Research Assessment have all advanced principles of responsible research evaluation.
The challenge is translating these principles into everyday decisions about recruitment, promotion, funding and recognition.
Expert judgment, however, is not automatically fairer than numerical assessment. Reviewers may favour familiar institutions, established ideas or influential colleagues.
Transparent criteria, appropriate expertise, declared conflicts of interest, written assessments and independent scrutiny would therefore be essential. Replacing numerical bias with personal bias would hardly constitute progress. University rankings are popular partly because they simplify difficult choices. Any alternative must be accessible and understandable.
Imagine a publicly available profile of each university, presenting verified information about teaching, faculty availability, laboratories, affordability, scholarships, student outcomes, research opportunities and distinctive strengths.
It could also show how these indicators have changed over several years.
Students and parents could compare institutions according to their own priorities. A student interested in research might focus on laboratory facilities and faculty expertise. Another seeking affordable undergraduate education close to home might make a different choice.
Neither should be expected to accept a ranking agency's fixed definition of the best university.
India already collects considerable institutional information through accreditation, regulatory and ranking exercises. Making that evidence more reliable, understandable and useful may be more valuable than creating another league table. Independent verification would be necessary. Otherwise, institutional profiles could become little more than promotional brochures.
An Indian Experiment
Rather than demand an immediate overhaul of the entire system, why not test an alternative? India could launch an Academic Evaluation Experiment involving perhaps 10 to 15 willing universities with different missions, resources and regional circumstances.
The experiment would have a specific initial purpose: to examine whether institutional development support can be allocated more intelligently through contextual evaluation rather than reliance on overall rankings.
Over five years, participating institutions would document their achievements, measurable improvements, resource constraints and plans for the future. Existing administrative records would be used wherever possible to avoid unnecessary paperwork.
Independent panels would assess the evidence, identify strengths and weaknesses, and recommend areas for development. They would not assign an overall rank.
The assessments could inform development grants and institutional improvement plans. A separate, smaller trial could examine whether evaluating scientists through their most significant contributions improves decisions about research support or academic advancement.
But how would we know whether the experiment worked?
A suitably comparable group of institutions could continue under existing arrangements. Independent evaluators would examine whether the new approach led to better-informed decisions, meaningful improvements and fairer recognition of emerging capability.
The experiment would also need to examine its own costs and unintended consequences. Institutions could exaggerate their achievements. Reviewers could bring personal biases. Measures of progress could become targets to be manipulated, recreating the very problem the experiment was intended to solve.
Independent verification, transparent criteria, selective audits and periodic review of the evaluation process would therefore be essential. The objective would be to establish whether a different approach produces demonstrably better decisions.
Rankings are unlikely to disappear, nor must they. They can remain useful sources of information when their methods and limitations are understood.
A more contextual system requires time, expertise and resources. It should be adopted where it demonstrably improves decisions, not merely because it sounds more enlightened. The deeper reform is cultural.
Universities should aspire to become better at what they exist to do, rather than merely better positioned than their competitors. Scientists should seek to make important contributions, not simply accumulate impressive indicators. Governments should use evaluation to identify where support can produce the greatest public benefit.
Imagine judging a university not merely by the institutions it has overtaken, but by the minds it has awakened, the knowledge it has created, the problems it has helped solve and the possibilities it has opened.
The ultimate purpose of evaluation is not to arrange institutions and scientists in order. It is to recognise achievement, discover potential and create the conditions in which excellence can flourish.
(The writer is an ANRF Prime Minister Professor at COEP Technological University, Pune, and former Director of the Agharkar Research Institute, Pune. Views personal.)
Comments