Web Scraping Project 🕸️

This repository contains a collection of Python-based web scraping scripts for collecting structured data from websites. It includes code for extracting weather data, parsing tables, handling HTTP requests, and organizing scraped content into CSV or JSON formats. This project serves both as a practice repository for mastering web scraping techniques and a utility tool for automating repetitive data-gathering tasks.

Features

• Extracts data from HTML tables and pages
• Refine scraped information using LLMs
• Saves scraped content in structured formats (CSV, JSON
• Simple and readable Python scripts
• Uses libraries like requests, BeautifulSoup, and pandas

Project Structure

Download from https://googlechromelabs.github.io/chrome-for-testing/#stable

AI_Web_Scraper
├── chromedriver          # Download from above given link
├── main.py      
├── parse.py         
├── requirements.txt                   
└── scrape.py

Wikipedia_Scraper
├── companies.csv
├── companies_BS.ipynb
└── requirements.txt   

Getting Started

Requirements

pip install -r requirements.txt

Running a Script

To run AI_Web_Scraper, use:

streamlit run main.py

Example Use Case

You can use this project to:

• Automate collection of tabular data from news, sports, or research sites
• Use LLMs to build modern scalable web scrapers

Disclaimer

• This project is intended for educational purposes.
• Always check a website’s Terms of Service before scraping.

Contact

Maintained by Santhusha Mudannayaka. Feel free to fork the project, raise issues, or contribute improvements!