"An issue of concern to computer scientists is the tendency to use publication databases that do not adequately cover CS, such as Thomson Scientific's ISI Web of Science.
The principal problem is what ISI counts. Many CS conferences and most books are not listed; conversely, some publications are included indiscriminately. The results make computer scientists cringe.d Niklaus Wirth, Turing Award winner, appears for minor papers from indexed publications, not his seminal 1970 Pascal report. Knuth's milestone book series, with an astounding 15,000 citations in Google Scholar, does not figure. Neither do Knuth's three articles most frequently cited according to Google.
Evidence of ISI's shortcomings for CS is "internal coverage": the percentage of citations of a publication in the same database. ISI's internal coverage, over 80% for physics or chemistry, is only 38% for CS.
Another example is Springer's Lecture Notes in Computer Science, which ISI classified until 2006 as a journal. A great resource, LNCS provides fast publication of conference proceedings and reports. Lumping all into a single "journal" category was absurd, especially since ISI omits top non-LNCS conferences:
- The International Conference on Software Engineering (ICSE), the top conference in a field that has its own ISI category, is not indexed.
- An LNCS-published workshop at ICSE, where authors would typically try out ideas not yet ready for ICSE submission, was indexed.
ISI indexes SIGPLAN Notices, an unrefereed publication devoting ordinary issues to notes and letters and special issues to proceedings of such conferences as POPL. POPL papers appear in ISI—on the same footing as a reader's note in a regular issue.
The database has little understanding of CS. Its 50 most cited CS references include "Chemometrics in food science," from a "Chemometrics and Intelligent Laboratory Systems" journal. Many CS entries are not recognizable as milestone contributions. The cruelest comparison is with CiteSeer, whose Most Cited list includes many publications familiar to all computer scientists; it has not a single entry in common with the ISI list.
ISI's "highly cited researchers" list includes many prestigious computer scientists but leaves out such iconic names as Wirth, Parnas, Knuth and all the 10 2000–2006 Turing Award winners except one. Since ISI's process provides no clear role for community assessment, the situation is unlikely to improve.The inevitable deficiencies of alternatives pale in consideration:
In assessing publications and citations, ISI Web of Science is inadequate for most of CS and must not be used. Alternatives include Google Scholar, CiteSeer, and (potentially) ACM's Digital Library.
Anyone in charge of assessment should know that attempts to use ISI for CS will cause massive opposition and may lead to outright rejection of any numerical criteria, including more reasonable ones.
"