Web Scraping Explained: Concepts and MCQs for Computer Awareness and IT Exams

Web Scraping Explained: Concepts and MCQs for Computer Awareness and IT Exams

Web scraping has moved from programming interviews into general computer awareness papers. Questions now turn up in IT officer and specialist officer exams, in data science and Python papers, and in interviews for analyst roles. The options are usually built to confuse scraping with crawling, an API with a scraper, or robots.txt with a security control.

This guide covers what web scraping is, how it works, the tools and terms examiners use, and the legal points worth knowing. Practice MCQs with answers and a short FAQ follow at the end.

What is web scraping?

Web scraping is the automated extraction of data from websites and its conversion into a structured format, such as a spreadsheet, a CSV file, a JSON file or a database table.

A web page is built from HTML. The prices, names, dates and reviews you read in a browser are all present in that HTML. A scraper requests the same page your browser does, reads the HTML, picks out the parts it needs and saves them in rows and columns.

Copying a table from a website into Excel by hand is, strictly speaking, scraping. The term is used for the automated version, where a program does the same job for thousands of pages.

Memory tip: to scrape is to take something off a surface. A web scraper takes data off the surface of a web page.

Other names you may see in options: web harvesting and web data extraction. Both mean the same thing as web scraping.

How web scraping works, step by step

Most scrapers follow the same five steps.

1. Send a request. The scraper sends an HTTP GET request to the page URL, just like a browser.

2. Receive the response. The web server returns the page, usually as HTML, with a status code such as 200 (OK).

3. Parse the HTML. A parser turns the raw HTML into a tree of elements called the DOM (Document Object Model), so the program can move through it.

4. Locate and extract data. The scraper finds the elements it wants using CSS selectors or XPath expressions and pulls out their text or attributes.

5. Store the data. The results are saved as CSV, JSON, Excel or in a database for analysis.

Key Point: parsing is the step that turns unstructured HTML into something a program can search. Questions often ask which step comes after receiving the response; the answer is parsing.

Web scraping vs web crawling

This is the pair examiners like most.

• Web crawling is about discovery. A crawler, also called a spider or bot, starts from a list of URLs, follows the links it finds and builds a list of pages. Search engines such as Google crawl the web to build their index.
• Web scraping is about extraction. A scraper visits specific pages and pulls out specific data from them.

In real projects the two work together: a crawl step finds the product pages on a shopping site, then a scrape step collects the price and name from each one.

Memory tip: a crawler finds pages, a scraper reads them.

Web scraping vs API

An API (Application Programming Interface) is an official channel a website provides for other programs to request its data, usually returned as JSON. Scraping reads the page that was built for human visitors.

• An API is structured, documented and stable, but it only offers the data the owner chooses to share, often with limits on requests.
• Scraping can reach anything that is publicly displayed on the page, but it breaks when the page layout changes.

If a site offers an API for the data you need, using it is generally the better choice.

Static and dynamic pages

A static page sends all its content in the first HTML response. A simple request and a parser are enough.

A dynamic page loads part of its content later with JavaScript. The first HTML response may contain an empty box, and the data appears only after scripts run in the browser. To scrape such a page, the program has to run the JavaScript, usually with a headless browser: a real browser engine running without a visible window, controlled by code.

Key Point: if a scraper returns empty results from a page that looks full in a browser, the content is probably loaded by JavaScript.

Common tools and libraries

Python is the most widely used language for scraping, and most exam questions on tools are about the Python ecosystem.

• Requests: a Python library for sending HTTP requests and receiving responses.
• BeautifulSoup: a Python library for parsing HTML and XML documents and searching the resulting tree. It does not download pages by itself; it is usually paired with Requests.
• Scrapy: a Python framework for building crawlers and scrapers at scale, with built-in handling of requests, link following and data export.
• Selenium: a browser automation tool originally built for testing web applications, widely used for scraping dynamic pages.
• Playwright and Puppeteer: newer browser automation libraries that control headless browsers. Puppeteer is a Node.js library built mainly for Chrome; Playwright supports several browsers and languages.
• lxml: a fast Python library for processing HTML and XML, with XPath support.

A minimal Python example looks like this:

import requests
from bs4 import BeautifulSoup

response = requests.get("https://example.com")
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title.text)

Here Requests fetches the page, BeautifulSoup parses it, and the last line prints the text inside the page's title tag.

Memory tip: Requests brings the page, BeautifulSoup reads it.

Selecting data: CSS selectors and XPath

To pick data out of a page, a scraper needs a way to point at elements.

• CSS selectors use the same syntax as stylesheets. For example, .price selects elements with the class "price", and #title selects the element with the id "title".
• XPath (XML Path Language) describes a path through the document tree. For example, //h2 selects every h2 element in the document.

Both are tested. Remember that a dot means class and a hash means id in CSS, and that a double slash in XPath means "anywhere in the document".

Why websites limit scrapers

Websites protect themselves from heavy automated traffic because it costs them server resources and can be used to copy their content. Common measures include:

• Rate limiting: capping how many requests one visitor can make in a period. A server that refuses because of too many requests returns HTTP status code 429 (Too Many Requests).
• IP blocking: refusing requests from an IP address that has behaved like a bot. A refused request often returns 403 (Forbidden).
• CAPTCHA: a challenge that tells humans and computers apart. The abbreviation stands for Completely Automated Public Turing test to tell Computers and Humans Apart.
• User-Agent checks: the User-Agent is an HTTP header that identifies the browser or program making the request. Requests with missing or suspicious User-Agent values may be blocked.

Responsible scrapers slow their request rate, identify themselves honestly and avoid putting load on small sites.

Where proxies fit in

A proxy server forwards requests on behalf of a client, so the website sees the proxy's IP address instead of the client's. Examveda's earlier guest post on proxy servers covers the types in detail.

For scraping, proxies matter in two ways. They spread a large job across many IP addresses so that no single address sends too many requests, and they let a scraper see the version of a page shown in another country, which matters for prices, search results and ads that change by location. Residential proxies use IP addresses assigned by internet service providers to homes. Datacenter proxies use addresses from hosting companies and are faster and cheaper but easier for websites to identify.

Teams that do not want to manage proxies and headless browsers themselves often use a scraper API instead: the program sends a URL to the service, and the service handles the proxies, the JavaScript rendering and the retries, then returns the page or the extracted data. Commercial providers sell both options. Evomi, a Swiss provider, lists residential proxies from $0.49 per GB and a Scraper API that includes JavaScript rendering and bills only for successful requests.

Key Point: a proxy changes which IP address the website sees. It does not by itself make a request legal or permitted.

robots.txt and the rules of scraping

robots.txt is a plain text file placed at the root of a website, for example example.com/robots.txt. It tells crawlers which parts of the site they may or may not visit, using lines such as User-agent and Disallow. The convention is called the Robots Exclusion Protocol.

Two points are tested often:

• robots.txt is a set of instructions for well-behaved bots. It is not a security mechanism and does not technically stop anyone from visiting a page.
• It must sit in the root directory of the site to be found.

A related file, the sitemap (usually sitemap.xml), does the opposite job: it lists the pages a site wants search engines to find.

Is web scraping legal?

There is no single answer, and exam questions usually test the principles rather than case law.

• Scraping publicly available, non-personal data is widely practised, for example price comparison and market research.
• A website's terms of service may prohibit scraping, and ignoring them can create legal risk.
• Copyright still applies to scraped content. Collecting facts such as prices is different from republishing someone's articles or images.
• Personal data brings in data protection law, such as the GDPR in the European Union and the Digital Personal Data Protection Act, 2023 in India.
• Scraping content behind a login, or bypassing technical protection, is far riskier than reading public pages.

Key Point: the method is not illegal in itself. What matters is what data is collected, how, and what is done with it.

Common uses of web scraping

• Price monitoring and comparison on e-commerce sites
• Market research and competitor analysis
• Collecting job listings, property listings or travel fares
• Tracking search engine results for SEO
• Gathering news and public records for research
• Building datasets for machine learning and AI training

Practice MCQs

1. Web scraping is best described as:

(A) Encrypting data sent between websites; (B) Automated extraction of data from websites into a structured format; (C) Designing the layout of web pages; (D) Blocking bots from a website

Answer: (B). Scraping collects data from pages and stores it in a usable format such as CSV or JSON.

2. Which of the following follows links to discover and index web pages?

(A) Web scraper; (B) Web crawler; (C) Parser; (D) Firewall

Answer: (B). A crawler, or spider, discovers pages. A scraper extracts data from them.

3. BeautifulSoup is a Python library mainly used for:

(A) Sending emails; (B) Parsing HTML and XML documents; (C) Training neural networks; (D) Managing databases

Answer: (B). BeautifulSoup parses documents; it is usually combined with Requests to download them.

4. Which HTTP status code means "Too Many Requests"?

(A) 200; (B) 301; (C) 404; (D) 429

Answer: (D). 429 is returned when a client exceeds the server's rate limit.

5. The robots.txt file is placed in:

(A) The root directory of a website; (B) The user's browser cache; (C) The DNS server; (D) Any folder of the website

Answer: (A). Crawlers look for it at the root, for example example.com/robots.txt.

6. Which statement about robots.txt is correct?

(A) It encrypts the website; (B) It technically blocks all bots; (C) It gives instructions to crawlers but does not enforce them; (D) It stores user passwords

Answer: (C). Well-behaved bots follow it, but it is not a security control.

7. A page whose content is loaded by JavaScript after the first response is called:

(A) Static page; (B) Dynamic page; (C) Cached page; (D) Mirror page

Answer: (B). Dynamic pages usually need a headless browser to scrape.

8. A browser that runs without a graphical window and is controlled by code is called:

(A) Proxy browser; (B) Headless browser; (C) Text browser; (D) Mirror browser

Answer: (B). Selenium, Playwright and Puppeteer can drive headless browsers.

9. In a CSS selector, .price selects:

(A) The element with id "price"; (B) Elements with class "price"; (C) All paragraph elements; (D) The page title

Answer: (B). A dot selects by class, a hash (#) selects by id.

10. XPath is a language used to:

(A) Style web pages; (B) Navigate and select nodes in an XML or HTML document; (C) Encrypt network traffic; (D) Compress images

Answer: (B). XPath expressions such as //h2 point to elements in the document tree.

11. Which HTTP header identifies the browser or program making a request?

(A) Content-Type; (B) User-Agent; (C) Host; (D) Accept-Language

Answer: (B). Servers often check the User-Agent to spot automated requests.

12. CAPTCHA is mainly used to:

(A) Speed up page loading; (B) Tell human visitors apart from bots; (C) Store cookies; (D) Compress HTML

Answer: (B). It stands for Completely Automated Public Turing test to tell Computers and Humans Apart.

13. Which Python framework is designed for large-scale crawling and scraping?

(A) Django; (B) Flask; (C) Scrapy; (D) NumPy

Answer: (C). Django and Flask are web frameworks, NumPy is for numerical computing.

14. Compared with web scraping, an official API usually:

(A) Returns structured data through a documented interface; (B) Breaks whenever the page design changes; (C) Requires a headless browser; (D) Reads the HTML built for human visitors

Answer: (A). The other options describe scraping.

15. A scraper uses residential proxies mainly to:

(A) Encrypt the stored data; (B) Send requests through IP addresses assigned to home internet users; (C) Speed up HTML parsing; (D) Replace the need for a parser

Answer: (B). Residential proxies route traffic through household IP addresses, which also allows location-specific views of a page.

FAQ

What is the difference between web scraping and web crawling?

Crawling discovers pages by following links. Scraping extracts specific data from pages. Most large projects crawl first and then scrape.

Which language is best for web scraping?

Python is the most common choice, because of libraries such as Requests, BeautifulSoup and Scrapy, and browser tools such as Selenium and Playwright.

Is web scraping legal?

It depends on the data and the method. Public, non-personal data is widely collected, but terms of service, copyright and data protection laws such as India's DPDP Act, 2023 still apply.

Does robots.txt stop scraping?

No. It is a set of instructions that responsible bots follow. It does not technically block access.

Why do scrapers use proxies?

To spread requests across many IP addresses and to see pages as they appear in other countries.