MARVEL Level 2 Tasks Report : Part 4

TASK 10: Web Scraping and Automation - Flight Ticket Price Analysis

What did I learn?

• I learnt the basics of selenium and how it works
• i learnt what web scraping is and how it can be used
• I also learnt the ethical and legal issues with web scraping
• In this task the program automatically opens chrome tab, searches the website, fills in the details, finds the flights and then writes it into a text file and a csv file which we cab use to analyse later
• Web scraping is the process of extracting data from websites by simulating human interaction with the web page using a program or script. This data can be anything publicly available on a website, such as text, images, videos, or even complex data like tables and charts. Scraping can be done through libraries like BeautifulSoup, Selenium, and Scrapy in Python, or through other programming languages and tools.
• Web scraping typically involves:
  1. Sending an HTTP request to the website’s server to retrieve its content.
  2. Parsing the HTML content of the page to extract the relevant data.
  3. Storing or processing the extracted data as per the user’s needs.
• Ethical Issues in Web Scraping:
  1. Impact on Website Performance: If a scraper is too aggressive and sends too many requests in a short period, it can overwhelm a website’s server, potentially slowing down or crashing the website. This can harm the website's operations and disrupt services for legitimate users.
  2. Violation of Terms of Service: Many websites explicitly prohibit scraping in their Terms of Service (ToS). Ignoring these terms and scraping the data could lead to conflicts with the website owner or platform.
  3. Bot Laws: Some countries have specific laws regarding the use of bots for data scraping. These laws aim to prevent bots from misusing data or overloading servers.
• 
•