Resource Review #3: How Scholarly is Google Scholar?

Here’s the link.

http://search.ebscohost.com.ezproxy.library.wisc.edu/login.aspx?direct=true&db=lxh&AN=40507818&loginpage=Login.asp&site=ehost-live

Howland, J. L., Wright, T. C., Boughan, R. A., & Roberts, B. C. (2009). How scholarly is google scholar? A comparison to library databases. College & Research Libraries, 70(3), 227-234.

This study sets out to compare Google Scholar to that of traditional library databases, and also to find how this comparison stacks up across disciplines. The write up does a good job of explaining in detail the study. The methodology is sound and the conclusions reasonable. It also does a good job of noting in what ways it is limited.

In terms of methodology, seven subject libraries were asked to give a typical reference question in their field, the database they would use to search for the answer and the search terms they would use. The researchers then entered the same searches in Google Scholar. Taking the first 30 results from the library database and Google Scholar, they merged and scrambled them in such away that it would be impossible to discern from which database a source was from. Then they took the list back to the librarians and asked them to rate each source individually based on different factors that made up a “scholarliness” value. Then the answers were put through a formula to correct for individual tendencies to vote low or high and a final scholarliness number was ascertained. The categories within the scholarliness value encompass all of what is important in a source and the methodology seems to cover as many bases as possible in reducing secondary factors.

The results are a bit surprising in that generally the citations contained only in Google had a higher scholarliness value than those only in a library database. The researchers see this as a trend that will continue as more and more publishers share their content with Google Scholar.  Additionally, there was no meaningful difference across disciplines.

The researchers do a good job of identifying the limits of extrapolating their findings. Given that the number of librarians were 0nly seven, and all from the same university makes it hard to extend the findings. They also note they were limited by a three point Likert scale as opposed to a five or seven point. The biggest point made here is that the study was used by skilled librarians and not by students, who search in quite different ways. Therefore, it’s hard to know if a student would have better or worse luck than what is reflected in this study.  The very last point made is also important: Google Scholar acts as a discovery tool, and does not provide access to the content that databases do. Overall, the researchers did a good job of identifying the limitations of their study while presenting interesting findings in a very clear way.