Python Web Scraping Tutorial: scraping dynamic JavaScript/AJAX websites with BeautifulSoup

Published: 05 April 2020
on channel: Red Eyed Coder Club

30,952

574

This Python Web Scraping Tutorial is about scraping dynamic websites, where the content is rendered by JavaScript.
For this Python Web Scraping Tutorial I used the Steam Store as an example. Because Steam website is an example of heavy JavaScript/AJAX driven website with dynamic content.
To scrape Steamstore website with Python I used only Python Requests and BeautifulSoup (bs4) libraries. With further exporting scraped data to a csv file.

This web scraping Python tutorial is the detailed explanation of how to scrape JavaScript driven pages and websites with Python and BeautifulSoup library for absolute beginners.

To install BeautifulSoup, Requests and Lxml:
pip install bs4 requests lxml

Follow me @:
Telegram: https://t.me/red_eyed_coder_club
Twitter:   / codereyed
Facebook: https://fb.me/redeyedcoderclub

======================================
📎️ The SOURCE CODE is available via Patreon:
  / steam-store-with-35670113
======================================

Timecodes:

00:00 - Beginning.
01:09 - Preliminary research (what to scrape)
03:15 - Creating a function that performs GET requests to Steam Store
06:01 - Server response research: what url should be passed in to the get_html() function
09:24 - The scraping plan
09:43 - Getting all Steam Store games with Python Requests, and BeautifulSoup. Scraping pagination.
12:40 - The algorithm of scraping all pages using the pagination GET requests
16:35 - Scraping data of a certain page with games
25:30 - Scraping hovering data of all games on each page, including the data from the hovering window
38:40 - Writing Scraped data to a CSV file

✴️✴️✴️ Also can be useful ✴️✴️✴️
Python tutorial: Namespaces and Scopes -    • Python tutorial #7: Python Namespaces...
Python Regular Expressions tutorial -    • Regex Python Tutorial: Python Regular...
Python tutorial: handling exceptions -    • Python tutorial #14: Python Exception...
How to read and write CSV -    • Python CSV tutorial: How to read and ...

✴️✴️✴️ Web Scraping course ✴️✴️✴️
is available via Patreon here:
  / red_eyed_coder_club

or its landing:
https://red-eyed-coder-club.github.io...

✴️✴️✴️ PLAYLISTS ✴️✴️✴️

🔹Django 3 Tutorial: Blog Engine
   • Python Django Tutorial #0: Demo of Dj...

🔹Kivy Tutorial: Coppa Project
   • Python Kivy tutorial #0: Demo Project...

🔹Telegram Bot with Python (CoinMarketCap)
   • Python Telegram Bot Tutorial: Demo an...

🔹Python Web Scraping
   • Python Ebay Scraping Tutorial: Web sc...

➥➥➥ SUBSCRIBE FOR MORE VIDEOS ➥➥➥
Red Eyed Coder Club is the best place to learn Python programming and Django:
Subscribe ⇢    / @redeyedcoderclub

Python Web Scraping Tutorial: scraping dynamic JavaScript/AJAX websites with BeautifulSoup
   • Python Web Scraping Tutorial: scrapin...

#python #pythonwebscraping #beautifulsoup #bs4 #redeyedcoderclub #webscrapingpython #beautifulsouptutorial