Are We Measuring What Matters?
The Ranking Trap - Part 1
When rankings become the yardstick of university excellence, genuine progress in teaching, research and social impact can disappear behind the numbers.

Imagine a university that has transformed itself over five years. Its laboratories are better equipped, teachers more engaged, students more confident and research increasingly relevant to the needs of its region. Yet, when the annual university rankings appear, it has slipped three places.
Another university rises five places. Its administrators celebrate, congratulatory messages circulate and the achievement receives considerable publicity.
Which university has actually improved? The question seems simple, but it exposes a fundamental weakness in how we judge education and research. We have become remarkably enthusiastic about measuring, comparing and ranking almost everything. Yet we seldom ask whether the numbers are telling us what we really want to know.
Ranking Illusions
Students are identified by examination ranks. Universities celebrate their positions in national and international league tables. Scientists monitor publication counts, citations and h-indices. Institutions proudly announce how many of their faculty feature in citation-based lists of the world’s top 2 percent of scientists.
There is nothing wrong with measurement. The trouble begins when we confuse measurement with judgment, and judgment with ranking.
Evaluation asks how well an institution is fulfilling its purpose, what it has achieved and where it needs to improve. Ranking asks where it stands relative to others.
A university can improve substantially and still fall in the rankings because other institutions have improved faster. Conversely, it can rise because competitors have performed poorly or because the ranking methodology has changed.
A university can become better without rising in the rankings, and it can rise in the rankings without becoming better.
Yet annual movements in league tables are routinely interpreted as evidence of progress or decline.
Consider how university rankings are constructed. India’s National Institutional Ranking Framework combines indicators relating to teaching, research, graduation outcomes, outreach and perception. International rankings use different combinations of research performance, reputation, internationalisation and other measures.
In the QS World University Rankings methodology published in 2026, academic reputation accounts for 30 percent and employer reputation for another 15 percent. Together, these survey-based reputation indicators contribute 45 percent of the overall score.
This illustrates how rankings reflect not only measurable institutional outcomes but also the weight assigned to particular indicators and perceptions. Consequently, two universities with different missions, student profiles and regional responsibilities may be assessed against criteria that do not fully capture their respective contributions.
Reputation is relevant. But is a university necessarily better because it is better known?
An established institution may thoroughly deserve its reputation. Equally, a younger university doing excellent work may take years to receive comparable recognition. Reputation can reflect achievement, but it can also lag behind changing realities.
False Precision
Suppose one university is ranked 47th and another 52nd. The numbers suggest a meaningful difference. But what if their underlying scores are almost identical? What if a modest change in the weight assigned to one indicator reverses their positions?
The ranking still appears precise, although the judgment behind it may be far less certain.
A rank is not a direct measurement like temperature or distance. It is the result of choices about what to measure, how to measure it and how much importance to assign to each component.
Different ranking systems can therefore produce different results without necessarily making a mathematical error. They may simply be emphasising different things. This matters particularly to students and parents.
Consider two engineering colleges. One has a stronger national reputation and an impressive research publication record. The other has excellent workshops, dedicated teachers and close engagement with local industries.
A student interested in hands-on engineering may find the second more suitable, even if it ranks lower. Another student seeking a research career may reasonably prefer the first.
Neither decision can be settled by a universal league table.
Institutional Conformity
There is a subtler danger. When institutions are rewarded according to similar indicators, they may gradually begin pursuing similar ambitions.
A university established to serve an underserved region may feel compelled to imitate a metropolitan research institution. A college known for excellent undergraduate teaching may devote increasing attention to publication counts because research carries greater weight in the rankings.
We celebrate diversity in education, yet risk rewarding uniformity in institutional ambition.
The same difficulty appears in the evaluation of scientists. Consider the widely publicised Stanford–Elsevier citation-based lists, popularly described as identifying the world’s top 2 percent scientists. They use several bibliometric indicators to identify researchers with substantial citation impact.
These indicators provide useful information. Citations tell us something about how frequently published work is acknowledged in subsequent research. But citation impact is not identical to originality, scientific significance or practical usefulness.
A 2026 study in Scientometrics examined differences in publication coverage between Scopus, which underlies these lists, and OpenAlex. It demonstrated how the choice and coverage of a publication database can affect bibliometric assessments.
Bibliometric databases differ in their coverage of journals, conference proceedings and other scholarly outputs. As a result, a researcher’s apparent performance can vary depending on the database and indicators used, even when the underlying body of work remains unchanged.
The scientists themselves had not changed. The information available for evaluating them had.
Take the h-index. A scientist with an h-index of 40 has published at least 40 papers, each cited at least 40 times. It is a clever numerical summary. But does it tell us whether that scientist has made an important discovery, developed a useful instrument or solved a pressing problem?
Citation practices differ enormously across disciplines. Review articles often attract many citations. Large collaborations can produce extensive publication records. Meanwhile, researchers developing instruments, industrial processes or practical technologies may publish relatively few papers.
Journal impact factors present a related problem. They describe average citation patterns across groups of papers, not the quality of an individual discovery. Nevertheless, the reputation of a journal sometimes substitutes for careful examination of the work itself.
History offers a reminder of how slowly scientific significance may become apparent. Gregor Mendel published his pioneering research on heredity in 1866. Its importance was largely overlooked until around 1900.
His discovery did not become more original when the scientific community finally recognised it.
None of this makes rankings worthless. They can reveal patterns, encourage institutions to examine their performance and provide useful starting points for comparison. But they are instruments, not verdicts.
A remarkable teacher may transform hundreds of lives without producing a highly cited paper. A scientist may spend years solving a technical problem that attracts little academic attention but matters enormously to industry or society. A small university may create opportunities in a region that larger institutions have long neglected.
Such contributions can be evaluated through evidence, even when they cannot be represented adequately by a single number.
The question is not whether we should measure performance. Of course we should. It is whether we are allowing what is easy to count to overshadow what is important to understand.
The tragedy would not simply be that rankings failed to recognise excellence. It would be that, in pursuing higher rankings, our institutions neglected forms of excellence that the rankings were never designed to capture.
(The writer is an ANRF Prime Minister Professor at COEP Technological University, Pune, and former Director of the Agharkar Research Institute, Pune. Views personal.)






Comments