How to Scrape (Almost) Any Website with Next.js and Puppeteer
If your code is getting blocked from scraping a website, using an automated web scraper like Puppeteer may be your best option to circumvent the issue.
Donations: https://www.paypal.com/donate/?hosted...
Follow me on Twitter: / @johne104
YouTube Channel: / @bluesphere7930
Email: [email protected]
Timecodes
0:00 - Why Use a Browser
1:06 - Setting Up the Project in the IDE
2:52 - Creating the UI
9:26 - Initial Browser Setup
11:28 - Navigating to Target Webpage
19:34 - Parse HTML and Scrape Pertinent Data
35:05 - Scroll Down and Navigate to Next Page
41:42 - Scrape Data from Second Page
45:16 - Automate Pagination Handling
50:44 - Outro