{"id":10518,"date":"2024-08-28T07:12:00","date_gmt":"2024-08-28T07:12:00","guid":{"rendered":"https:\/\/scrapingdog.com\/?p=10518"},"modified":"2025-08-25T10:47:11","modified_gmt":"2025-08-25T10:47:11","slug":"scraping-with-python-selenium","status":"publish","type":"post","link":"https:\/\/www.scrapingdog.com\/blog\/scraping-with-python-selenium\/","title":{"rendered":"Web Scraping with Selenium &#038; Python (A Beginner Friendly Tutorial)"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"10518\" class=\"elementor elementor-10518\" data-elementor-post-type=\"post\">\n\t\t\t\t<div class=\"elementor-element elementor-element-d08f479 e-con-full e-flex e-con e-parent\" data-id=\"d08f479\" data-element_type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-a35c18a elementor-widget elementor-widget-html\" data-id=\"a35c18a\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<!-- Gutenberg \u201cCustom HTML\u201d block -->\r\n<div style=\"\r\n  background:#d9f4e5;\r\n  border-left:4px solid #1d9b6c;\r\n  padding:18px 24px;\r\n  margin:24px 0;\r\n  border-radius:6px;\r\n  font-family:'Montserrat',sans-serif;\r\n  font-size:18px;\r\n  line-height:1.65;\r\n  color:#1a1a1a;\">\r\n  <p style=\"margin:0 0 8px 0; font-weight:600;\">TL;DR<\/p>\r\n\r\n  <ul style=\"margin:0; padding-left:20px;\">\r\n    <li>Beginner guide: scrape JS-heavy pages with <strong>Python<\/strong> using <strong>Selenium<\/strong> + <strong>BeautifulSoup<\/strong>.<\/li>\r\n    <li>Setup <em>Chrome<\/em> \/ Driver, install <code>selenium<\/code> &amp; <code>bs4<\/code>; run headless; wait for content to load.<\/li>\r\n    <li>Demo: open a <strong>Walmart<\/strong> search, read <code>page_source<\/code>, parse titles \/ prices, build a JSON list.<\/li>\r\n    <li>Takeaway: render with <strong>Selenium<\/strong>, parse with <strong>Soup<\/strong>\u2014simple pattern for dynamic sites.<\/li>\r\n  <\/ul>\r\n<\/div>\r\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e6b7a88 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"e6b7a88\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"7354\">In this post, we will learn how to do Selenium web scraping with Python.\u00a0<a href=\"https:\/\/selenium-python.readthedocs.io\/\" target=\"_blank\" rel=\"nofollow noopener\">Selenium is an open-source<\/a>\u00a0automated testing framework used to validate web applications across different browsers and platforms. It was created by\u00a0<a href=\"https:\/\/www.thoughtworks.com\/en-us\/insights\/blog\/happy-10th-birthday-selenium\" target=\"_blank\" rel=\"noreferrer noopener nofollow\" data-type=\"URL\" data-id=\"https:\/\/www.thoughtworks.com\/en-us\/insights\/blog\/happy-10th-birthday-selenium\">Jason Huggins in 2004, a Software Engineer at ThoughtWorks.<\/a><\/p><p id=\"7354\">He created it when he had to\u00a0<a href=\"https:\/\/www.aegissofttech.com\/insights\/ai-testing-tools-for-test-automation\/\" target=\"_blank\" rel=\"noreferrer noopener\" data-type=\"link\" data-id=\"https:\/\/www.aegissofttech.com\/insights\/ai-testing-tools-for-test-automation\/\">test a web application<\/a>\u00a0multiple times, manually leading to higher inefficiency and effort. The Selenium API has the advantage of controlling Firefox and Chrome through an external adaptor. It has a much larger community than Puppeteer. It is an executable module that runs a script on a browser instance &amp; hence is also called Python<strong>\u00a0headless browser scraping<\/strong>.<\/p><p id=\"7354\"><strong>Why you should use Selenium?<\/strong><\/p><p id=\"7354\">Today selenium is mainly used for web scraping and automation purposes.<\/p><ul class=\"wp-block-list\"><li><em><strong>clicking on buttons<\/strong><\/em><\/li><li><em><strong>filling forms<\/strong><\/em><\/li><li><em><strong>scrolling<\/strong><\/em><\/li><li><em><strong>taking a screenshot<\/strong><\/em><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8c02d09 elementor-widget elementor-widget-heading\" data-id=\"8c02d09\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Requirements for Web Scraping With Selenium &amp; Python<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-929d80e font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"929d80e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Generally, web scraping is divided into two parts:<\/p><ol class=\"wp-block-list\"><li>Fetching data by making an HTTP request<\/li><li>Extracting important data by parsing the HTML DOM<\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b483af8 elementor-widget elementor-widget-heading\" data-id=\"b483af8\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Libraries &amp; Tools<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-acbf101 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"acbf101\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<ol class=\"wp-block-list\"><li><a href=\"https:\/\/www.crummy.com\/software\/BeautifulSoup\/bs4\/doc\/\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Beautiful Soup<\/strong><\/a><strong>\u00a0<\/strong>is a Python library for pulling data out of HTML and XML files.<\/li><li><a href=\"https:\/\/pypi.org\/project\/selenium\/\" target=\"_blank\" rel=\"noreferrer noopener nofollow\"><strong>Selenium<\/strong><\/a><strong>\u00a0<\/strong>is used to automate web browser interaction from Python.<\/li><li><a href=\"https:\/\/www.google.com\/chrome\/\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Chrome download page<\/strong><\/a><\/li><li><a href=\"https:\/\/sites.google.com\/a\/chromium.org\/chromedriver\/downloads\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Chrome driver binary<\/strong><\/a><\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3277413 elementor-widget elementor-widget-heading\" data-id=\"3277413\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Setup<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7f804a5 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"7f804a5\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<ol class=\"wp-block-list\"><li><a href=\"https:\/\/www.crummy.com\/software\/BeautifulSoup\/bs4\/doc\/\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Beautiful Soup<\/strong><\/a><strong>\u00a0<\/strong>is a Python library for pulling data out of HTML and XML files.<\/li><li><a href=\"https:\/\/pypi.org\/project\/selenium\/\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Selenium<\/strong><\/a><strong>\u00a0<\/strong>is used to automate web browser interaction from Python.<\/li><li><a href=\"https:\/\/www.google.com\/chrome\/\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Chrome download page<\/strong><\/a><\/li><li><a href=\"https:\/\/sites.google.com\/a\/chromium.org\/chromedriver\/downloads\" target=\"_blank\" rel=\"nofollow noopener\"><strong>Chrome driver binary<\/strong><\/a><\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-45193f4 elementor-widget elementor-widget-code-highlight\" data-id=\"45193f4\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>mkdir scraper\r\npip install beautifulsoup4\r\npip install selenium<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-aeaa335 elementor-widget elementor-widget-heading\" data-id=\"aeaa335\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Quickstart<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fc353d2 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"fc353d2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Once you have installed all the libraries, create a Python file inside the folder. I am using scraping.py and then importing all the libraries as shown below. Also, import time in order to let the page load completely.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d89c5d5 elementor-widget elementor-widget-code-highlight\" data-id=\"d89c5d5\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>from selenium import webdriver\r\nfrom bs4 import BeautifulSoup\r\nimport time<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d373055 elementor-widget elementor-widget-heading\" data-id=\"d373055\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">What We Are Going to Scrape Using Selenium<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ffc07a1 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"ffc07a1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"8e88\">We are going to extract the Python Book price and title from\u00a0Walmart via selenium.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-035367a elementor-widget elementor-widget-image\" data-id=\"035367a\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"800\" height=\"330\" src=\"https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_zDdgRo1LdIkINjhp-sOJaA-1024x422-1.jpg\" class=\"attachment-large size-large wp-image-10583\" alt=\"\" srcset=\"https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_zDdgRo1LdIkINjhp-sOJaA-1024x422-1.jpg 1024w, https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_zDdgRo1LdIkINjhp-sOJaA-1024x422-1-300x124.jpg 300w, https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_zDdgRo1LdIkINjhp-sOJaA-1024x422-1-768x317.jpg 768w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7ec79f6 elementor-widget elementor-widget-heading\" data-id=\"7ec79f6\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Preparing the Food<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f634939 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"f634939\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"8e88\">Now, since we have all the ingredients to prepare the scraper, we should make a GET request to the\u00a0<a href=\"https:\/\/walmart.com\/search\/?query=python%20books\" target=\"_blank\" rel=\"noreferrer noopener nofollow\" data-type=\"URL\" data-id=\"https:\/\/walmart.com\/search\/?query=python%20books\">target URL<\/a>\u00a0to get the raw HTML data.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d5b633e elementor-widget elementor-widget-code-highlight\" data-id=\"d5b633e\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>options = webdriver.ChromeOptions()\r\noptions.add_argument('--headless')\r\n\r\ndriver = webdriver.Chrome('F:\/chromed\/chromedriver')\r\nurl=\"https:\/\/www.walmart.com\/search\/?query=python%20books\"<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9e8d7e0 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"9e8d7e0\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>A headless chrome will be launched which will be controlled by some external adaptor. Here are two interesting webdriver properties:<\/p><ul class=\"wp-block-list\"><li><code>driver.stop_client<\/code>\u00a0Called after executing a quit command.<\/li><li><code>driver.name<\/code>\u00a0Returns the name of the underlying browser for this instance.<\/li><\/ul><p>Now, to get raw HTML from the website, we have to use BeautifulSoup.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3b50757 elementor-widget elementor-widget-code-highlight\" data-id=\"3b50757\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>time.sleep(4)\r\nsoup=BeautifulSoup(driver.page_source,\u2019html.parser\u2019)\r\ndriver.close()\r\n\r\nbooks=list()\r\nk={}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e029fcf font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"e029fcf\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Now, I am letting it sleep for four seconds. The reason behind this is to let the page load completely. Then we will use BeautifulSoup to parse HTML.\u00a0<strong>driver.page_source<\/strong>\u00a0will return raw HTML from the website.<\/p><p>I have also declared an empty list and dictionary to create a JSON object of the data we are going to scrape.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cbdbb0f elementor-widget elementor-widget-image\" data-id=\"cbdbb0f\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"322\" src=\"https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_n8AElSNnUSJ5bBYaVExd7w-1024x412.jpg\" class=\"attachment-large size-large wp-image-10594\" alt=\"\" srcset=\"https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_n8AElSNnUSJ5bBYaVExd7w-1024x412.jpg 1024w, https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_n8AElSNnUSJ5bBYaVExd7w-300x121.jpg 300w, https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_n8AElSNnUSJ5bBYaVExd7w-768x309.jpg 768w, https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_n8AElSNnUSJ5bBYaVExd7w.jpg 1113w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1d2daf2 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"1d2daf2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>After inspecting the title in\u00a0<a href=\"https:\/\/developers.google.com\/web\/tools\/chrome-devtools\" target=\"_blank\" rel=\"noreferrer noopener nofollow\">chrome developer tools<\/a>, we can see that the title is stored in a \u201c<strong>div\u201d<\/strong>\u00a0tag with class \u201c<strong>search-result-product-title listview\u201d<\/strong>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3923333 elementor-widget elementor-widget-image\" data-id=\"3923333\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"345\" src=\"https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_xbSvHjs2eyKhy9fYIzZvOA-1-1024x442-1.jpg\" class=\"attachment-large size-large wp-image-10598\" alt=\"\" srcset=\"https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_xbSvHjs2eyKhy9fYIzZvOA-1-1024x442-1.jpg 1024w, https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_xbSvHjs2eyKhy9fYIzZvOA-1-1024x442-1-300x129.jpg 300w, https:\/\/www.scrapingdog.com\/wp-content\/uploads\/2024\/08\/1_xbSvHjs2eyKhy9fYIzZvOA-1-1024x442-1-768x332.jpg 768w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f0f5ba8 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"f0f5ba8\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Similarly, the price is stored in \u201c<strong>span\u201d<\/strong>\u00a0tag with class \u201c<strong>price display-inline-block arrange-fit price price-main.\u201d\u00a0<\/strong>Also, we have to dive deep inside this tag to find<strong>\u00a0\u201cvisuallyhidden\u201d\u00a0<\/strong>to find the price in text format.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-dc16ff8 elementor-widget elementor-widget-code-highlight\" data-id=\"dc16ff8\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>try:\r\n Title=soup.find_all(\u201cdiv\u201d,{\u201cclass\u201d:\u201dsearch-result-product-title listview\u201d})\r\nexcept:\r\n Title=None\r\n\r\ntry:\r\n Price = soup.find_all(\u201cspan\u201d,{\u201cclass\u201d:\u201dprice display-inline-block arrange-fit price price-main\u201d})\r\nexcept:\r\n Price=None<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e2fe1d3 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"e2fe1d3\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>We have all the titles and prices stored in a list format in variable\u00a0<strong>Title<\/strong>\u00a0and\u00a0<strong>Price,<\/strong>\u00a0respectively. We are going to start a for loop so that we can reach each and every book.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c01b027 elementor-widget elementor-widget-code-highlight\" data-id=\"c01b027\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>for i in range(0,len(Title)):\r\n try:\r\n  k[\u201cTitle{}\u201d.format(i+1)]=Title[i].text.replace(\u201c\\n\u201d,\u201d\u201d)\r\n except:\r\n  k[\u201cTitle{}\u201d.format(i+1)]=None \r\n\r\ntry:\r\n  k[\u201cPrice{}\u201d.format(i+1)]=Price[i].find(\u201cspan\u201d,{\u201cclass\u201d:\u201dvisuallyhidden\u201d}).text.replace(\u201c\\n\u201d,\u201d\u201d)\r\n except:\r\n  k[\u201cPrice{}\u201d.format(i+1)]=None\r\n\r\n books.append(k)\r\n k={}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-df7d334 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"df7d334\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"ec98\">So, finally, we have all the prices and titles stored inside the list\u00a0<strong>books.\u00a0<\/strong>After printing it we got.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e46c899 elementor-widget elementor-widget-code-highlight\" data-id=\"e46c899\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>{\r\n \u201cPythonBooks\u201d: [\r\n {\r\n \u201cTitle1\u201d: \u201cProduct TitlePython : Advanced Predictive Analytics\u201d,\r\n \u201cPrice1\u201d: \u201c$111.66\u201d\r\n },\r\n {\r\n \u201cTitle2\u201d: \u201cProduct TitlePython\u201d,\r\n \u201cPrice2\u201d: \u201c$6.99\u201d\r\n },\r\n {\r\n \u201cTitle3\u201d: \u201cProduct TitlePython : Learn How to Write Codes-Your Perfect Step-By-Step Guide\u201d,\r\n \u201cPrice3\u201d: \u201c$16.05\u201d\r\n },\r\n {\r\n \u201cTitle4\u201d: \u201cProduct TitlePython: The Complete Beginner\u2019s Guide\u201d,\r\n \u201cPrice4\u201d: \u201c$14.99\u201d\r\n },\r\n {\r\n \u201cPrice5\u201d: \u201c$48.19\u201d,\r\n \u201cTitle5\u201d: \u201cProduct TitlePython : The Complete Reference\u201d\r\n },\r\n {\r\n \u201cTitle6\u201d: \u201cProduct TitleThe Greedy Python : Book & CD\u201d,\r\n \u201cPrice6\u201d: \u201c$10.55\u201d\r\n },\r\n {\r\n \u201cPrice7\u201d: \u201c$24.99\u201d,\r\n \u201cTitle7\u201d: \u201cProduct TitlePython: 2 Manuscripts in 1 Book: -Python for Beginners -Python 3 Guide (Paperback)\u201d\r\n },\r\n {\r\n \u201cTitle8\u201d: \u201cProduct TitleBooks for Professionals by Professionals: Beginning Python Visualization: Crafting Visual Transformation Scripts (Paperback)\u201d,\r\n \u201cPrice8\u201d: \u201c$67.24\u201d\r\n },\r\n {\r\n \u201cTitle9\u201d: \u201cProduct TitlePython for Kids: A Playful Introduction to Programming (Paperback)\u201d,\r\n \u201cPrice9\u201d: \u201c$23.97\u201d\r\n },\r\n {\r\n \u201cPrice10\u201d: \u201c$17.99\u201d,\r\n \u201cTitle10\u201d: \u201cProduct TitlePython All-In-One for Dummies (Paperback)\u201d\r\n },\r\n {\r\n \u201cTitle11\u201d: \u201cProduct TitlePython Tutorial: Release 3.6.4 (Paperback)\u201d,\r\n \u201cPrice11\u201d: \u201c$14.53\u201d\r\n },\r\n {\r\n \u201cPrice12\u201d: \u201c$13.58\u201d,\r\n \u201cTitle12\u201d: \u201cProduct TitleCoding for Kids: Python: Learn to Code with 50 Awesome Games and Activities (Paperback)\u201d\r\n },\r\n {\r\n \u201cPrice13\u201d: \u201c$56.10\u201d,\r\n \u201cTitle13\u201d: \u201cProduct TitlePython 3 Object Oriented Programming (Paperback)\u201d\r\n },\r\n {\r\n \u201cTitle14\u201d: \u201cProduct TitleHead First Python: A Brain-Friendly Guide (Paperback)\u201d,\r\n \u201cPrice14\u201d: \u201c$35.40\u201d\r\n },\r\n {\r\n \u201cTitle15\u201d: \u201cProduct TitleMastering Object-Oriented Python \u2014 Second Edition (Paperback)\u201d,\r\n \u201cPrice15\u201d: \u201c$44.99\u201d\r\n },\r\n {\r\n \u201cTitle16\u201d: \u201cProduct TitlePocket Reference (O\u2019Reilly): Python Pocket Reference: Python in Your Pocket (Paperback)\u201d,\r\n \u201cPrice16\u201d: \u201c$13.44\u201d\r\n },\r\n {\r\n \u201cTitle17\u201d: \u201cProduct TitleData Science with Python (Paperback)\u201d,\r\n \u201cPrice17\u201d: \u201c$39.43\u201d\r\n },\r\n {\r\n \u201cTitle18\u201d: \u201cProduct TitleHands-On Deep Learning Architectures with Python (Paperback)\u201d,\r\n \u201cPrice18\u201d: \u201c$29.99\u201d\r\n },\r\n {\r\n \u201cPrice19\u201d: \u201c$37.73\u201d,\r\n \u201cTitle19\u201d: \u201cProduct TitleDjango for Beginners: Build websites with Python and Django (Paperback)\u201d\r\n },\r\n {\r\n \u201cTitle20\u201d: \u201cProduct TitleProgramming Python: Powerful Object-Oriented Programming (Paperback)\u201d,\r\n \u201cPrice20\u201d: \u201c$44.21\u201d\r\n }\r\n ]\r\n}<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cec0893 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"cec0893\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Similarly, you can\u00a0<strong>scrape any JavaScript-enabled website using Selenium and Python<\/strong>. If you don\u2019t want to run these scrapers on your server, try<strong> Scrapingdog, a\u00a0<a href=\"https:\/\/www.scrapingdog.com\/\" target=\"_blank\" rel=\"noopener\">Web Scraping API<\/a>.<\/strong><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f640f9c elementor-widget elementor-widget-heading\" data-id=\"f640f9c\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-29a3fda font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"29a3fda\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>In this article, we understood how we could scrape data using Selenium\u00a0&amp;\u00a0BeautifulSoup\u00a0regardless of the type of website. I hope now you feel more comfortable scraping web pages.<\/p><p>If you are new to Python you can read this comprehensive\u00a0<a href=\"https:\/\/www.scrapingdog.com\/blog\/web-scraping-with-python\/\" target=\"_blank\" rel=\"noreferrer noopener\" data-type=\"URL\" data-id=\"https:\/\/www.scrapingdog.com\/blog\/web-scraping-with-python\/\">guide on web scraping with Python<\/a>.<\/p><p>Feel free to comment and ask me anything. You can follow me on\u00a0Twitter\u00a0and\u00a0Medium. Thanks for reading, and please hit the like button!<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ed4933d elementor-widget elementor-widget-heading\" data-id=\"ed4933d\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Additional Resources<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3c83bdc font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"3c83bdc\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Here are a few additional resources that you may find helpful during your web scraping journey:<\/p><ul class=\"wp-block-list\"><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/python-requests-proxy\/\" target=\"_blank\" rel=\"noreferrer noopener\" data-type=\"URL\" data-id=\"https:\/\/www.scrapingdog.com\/blog\/python-requests-proxy\/\">How To Use A Proxy With Python Requests<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/web-scraping-with-xpath-and-python\/\" target=\"_blank\" rel=\"noreferrer noopener\" data-type=\"URL\" data-id=\"https:\/\/www.scrapingdog.com\/blog\/web-scraping-with-xpath-and-python\/\">Web Scraping with XPath &amp; Python<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/best-python-web-scraping-libraries\/\" target=\"_blank\" rel=\"noreferrer noopener\">The Best Python Web Scraping Libraries<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/scrape-twitter\/\" target=\"_blank\" rel=\"noopener\">Scraping Twitter with Selenium &amp; Python<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/beautifulsoup-tutorial-web-scraping-with-python\/\" target=\"_blank\" rel=\"noreferrer noopener\" data-type=\"URL\" data-id=\"https:\/\/www.scrapingdog.com\/blog\/beautifulsoup-tutorial-web-scraping-with-python\/\">BeautifulSoup Tutorial: Web Scraping<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/scrape-data-behind-authentication-with-python\/\" target=\"_blank\" rel=\"noreferrer noopener\" data-type=\"URL\" data-id=\"https:\/\/www.scrapingdog.com\/blog\/scrape-data-behind-authentication-with-python\/\">Python Web Scraping Authentication Behind the oAuth Wall<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/scrape-dynamic-web-page\/\" target=\"_blank\" rel=\"noreferrer noopener\" data-type=\"URL\" data-id=\"https:\/\/www.scrapingdog.com\/blog\/scrape-dynamic-web-page\/\">How to Scrape Dynamic Web Pages using Python<\/a><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-d0acbc4 e-con-full web-scraping-right-con elementor-hidden-desktop elementor-hidden-tablet e-flex e-con e-child\" data-id=\"d0acbc4\" data-element_type=\"container\" data-settings=\"{&quot;background_background&quot;:&quot;classic&quot;,&quot;sticky&quot;:&quot;top&quot;,&quot;sticky_on&quot;:[&quot;desktop&quot;,&quot;tablet&quot;],&quot;sticky_parent&quot;:&quot;yes&quot;,&quot;sticky_offset&quot;:0,&quot;sticky_effects_offset&quot;:0}\">\n\t\t<div class=\"elementor-element elementor-element-0d50e64 e-con-full e-flex e-con e-child\" data-id=\"0d50e64\" data-element_type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-877324b elementor-widget elementor-widget-heading\" data-id=\"877324b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Web Scraping with Scrapingdog<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1a01492 elementor-widget elementor-widget-text-editor\" data-id=\"1a01492\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tScrape the web without the hassle of getting blocked\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-5c170e8 e-con-full e-flex e-con e-child\" data-id=\"5c170e8\" data-element_type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-adf53e5 elementor-align-justify elementor-widget elementor-widget-button\" data-id=\"adf53e5\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm\" href=\"https:\/\/api.scrapingdog.com\/register\" target=\"_blank\" rel=\"noopener\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t\t\t\t<span class=\"elementor-button-text\">Try for Free<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e5964e2 elementor-align-justify elementor-widget elementor-widget-button\" data-id=\"e5964e2\" data-element_type=\"widget\" data-widget_type=\"button.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<div class=\"elementor-button-wrapper\">\n\t\t\t\t\t<a class=\"elementor-button elementor-button-link elementor-size-sm\" href=\"https:\/\/share.hsforms.com\/1ex4xYy1pTt6rrqFlRAquwQ4h1b2\" target=\"_blank\" rel=\"noopener\">\n\t\t\t\t\t\t<span class=\"elementor-button-content-wrapper\">\n\t\t\t\t\t\t\t\t\t<span class=\"elementor-button-text\">Contact sales<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t<div class=\"elementor-element elementor-element-cb6c196 e-con-full e-flex e-con e-child\" data-id=\"cb6c196\" data-element_type=\"container\">\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>In this post, you will learn to Scrape dynamic websites using Selenium and Python<\/p>\n","protected":false},"author":5,"featured_media":20550,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[25],"tags":[],"class_list":["post-10518","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/posts\/10518","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/comments?post=10518"}],"version-history":[{"count":0,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/posts\/10518\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/media\/20550"}],"wp:attachment":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/media?parent=10518"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/categories?post=10518"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/tags?post=10518"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}