While most of the examples in this chapter ultimately serve to grease the wheels of capitalism, web scrapers are also used in the pursuit of knowledge. Web scrapers are commonly used in medical, sociological, and psychological research, among many other fields.
For example, Rutgers University offers a course called “Computational Social Science” which teaches students web scraping to collect data for research projects. Some university courses, such as the University of Oslo’s “Collecting and Analyzing Big Data,” even feature this topic on the syllabus!
In 2017, a project supported by the National Institutes of Health scraped the records of jail inmates in US prisons to estimate the number of inmates infected with HIV. This project precipitated an extensive ethical analysis, weighing the benefits of this research with the risk to privacy of the inmate population. Ultimately, the research continued, but it’s essential to examine the ethics of your project before using web scraping for research, particularly in the medical field.
Another health research study scraped hundreds of comments from news articles in The Guardian about obesity and analyzed the rhetoric of those comments. Although smaller in scale than other research projects, it’s worth considering that web scrapers can be used for projects that require “small data” and qualitative analysis as well.
Here’s another example of a niche research project that utilized web scraping. In 2016, a comprehensive study was done to scrape and perform qualitative analysis on marketing materials for every Canadian community college. Researchers determined that modern facilities and “unconventional organizational symbols” are most popularly promoted.
In economics research, the Bank of Japan published a paper about their use of web scraping to obtain “alternative data.” That is, data outside of what banks normally use, such as GDP statistics and corporate financial reports. In this paper, they revealed that one source of alternative data is web scrapers, which they use to adjust price indices.
If you’re looking for a powerful web data scraping solution, consider ScraperWiz. This desktop app simplifies the process, making it accessible to researchers and data enthusiasts alike.
Leave a Reply