{"id":20829,"date":"2025-01-15T09:16:50","date_gmt":"2025-01-15T09:16:50","guid":{"rendered":"https:\/\/www.scrapingdog.com\/?p=20829"},"modified":"2025-08-14T10:17:40","modified_gmt":"2025-08-14T10:17:40","slug":"how-to-use-selenium-in-nodejs","status":"publish","type":"post","link":"https:\/\/www.scrapingdog.com\/blog\/how-to-use-selenium-in-nodejs\/","title":{"rendered":"How to Use Selenium in NodeJS"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"20829\" class=\"elementor elementor-20829\" data-elementor-post-type=\"post\">\n\t\t\t\t<div class=\"elementor-element elementor-element-95df289 e-flex e-con-boxed e-con e-parent\" data-id=\"95df289\" data-element_type=\"container\">\n\t\t\t\t\t<div class=\"e-con-inner\">\n\t\t\t\t<div class=\"elementor-element elementor-element-f2c72b7 elementor-widget elementor-widget-html\" data-id=\"f2c72b7\" data-element_type=\"widget\" data-widget_type=\"html.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<!-- Gutenberg \u201cCustom HTML\u201d block -->\r\n<div style=\"\r\n  background:#d9f4e5;\r\n  border-left:4px solid #1d9b6c;\r\n  padding:18px 24px;\r\n  margin:24px 0;\r\n  border-radius:6px;\r\n  font-family:'Montserrat',sans-serif;\r\n  font-size:18px;\r\n  line-height:1.65;\r\n  color:#1a1a1a;\">\r\n  <p style=\"margin:0 0 8px 0;font-weight:600;\">TL;DR<\/p>\r\n\r\n  <ul style=\"margin:0; padding-left:20px;\">\r\n    <li><strong>Setup:<\/strong> init Node project, install <code>selenium-webdriver<\/code> + <code>chromedriver<\/code>, build a Chrome driver and open a page.<\/li>\r\n    <li><strong>Scrape:<\/strong> <code>getPageSource<\/code>, select with <code>By.css<\/code>, iterate titles \/ ratings.<\/li>\r\n    <li><strong>Techniques:<\/strong> wait (<code>until.elementLocated<\/code>), infinite scroll, type \/ click flows.<\/li>\r\n    <li>For scale or less hassle, use <strong>Scrapingdog<\/strong> to handle proxies, headless, retries.<\/li>\r\n  <\/ul>\r\n<\/div>\r\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-b1e1296 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"b1e1296\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"c1b8\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">If you want to scrape dynamic websites using Node.js and a headless browser, tools like\u00a0<a class=\"af mf\" href=\"https:\/\/pptr.dev\/\" target=\"_blank\" rel=\"noopener\">Puppeteer<\/a>\u00a0and\u00a0<a class=\"af mf\" href=\"https:\/\/www.selenium.dev\/\" target=\"_blank\" rel=\"noopener\">Selenium<\/a>\u00a0are great options. We have already covered\u00a0<a class=\"af mf\" href=\"https:\/\/www.scrapingdog.com\/blog\/puppeteer-web-scraping\/\" target=\"_blank\" rel=\"noopener ugc nofollow\">web scraping with Puppeteer<\/a>, and today, we\u2019ll learn how to use Selenium with Node.js for web scraping.<\/p><p id=\"2ac6\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Many of you might already be familiar with Selenium if you\u2019ve used it for <a href=\"https:\/\/www.scrapingdog.com\/blog\/web-scraping-with-python\/\" target=\"_blank\" rel=\"noopener\">web scraping with Python<\/a>. In this article, however, we\u2019ll explore using Selenium with Node.js for web scraping from scratch. We\u2019ll cover topics such as scraping a website, waiting for specific elements to load, and more.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f6086c9 elementor-widget elementor-widget-heading\" data-id=\"f6086c9\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Setting Up Selenium in Node.js<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d6ca343 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"d6ca343\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"c1b8\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Before diving into how to use Selenium for web scraping, you need to ensure your environment is ready. Follow these steps to install and set up Selenium with Node.js.<\/p><h3 id=\"db90\" class=\"nj mh fq bf mi nk nl nm mm nn no np mq ls nq nr ns lw nt nu nv ma nw nx ny nz bk\">Install Node.js<\/h3><p>I hope you have already installed Nodejs on your machine and if not then you can download it from\u00a0<a class=\"af mf\" href=\"https:\/\/nodejs.org\/en\/download\" target=\"_blank\" rel=\"nofollow noopener\">here<\/a>. You can verify the installation with this step.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6fa85ad elementor-widget elementor-widget-code-highlight\" data-id=\"6fa85ad\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>node -v<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3634314 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"3634314\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3 id=\"c766\" class=\"nj mh fq bf mi nk nl nm mm nn no np mq ls nq nr ns lw nt nu nv ma nw nx ny nz bk\">Create a new Node.js project<\/h3><p id=\"9f4c\" class=\"pw-post-body-paragraph lh li fq lj b lk ne lm ln lo nf lq lr ls ng lu lv lw nh ly lz ma ni mc md me fj bk\" data-selectable-paragraph=\"\">Create a folder with any name you like. We will store all of our\u00a0<code class=\"cx oo op oq og b\">.js<\/code>\u00a0files inside this folder.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e6d53fa elementor-widget elementor-widget-code-highlight\" data-id=\"e6d53fa\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>mkdir selenium-nodejs-demo\r\ncd selenium-nodejs-demo<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-089b53d font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"089b53d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"c766\" class=\"nj mh fq bf mi nk nl nm mm nn no np mq ls nq nr ns lw nt nu nv ma nw nx ny nz bk\">Then initialize\u00a0<code class=\"cx oo op oq og b\">package.json<\/code>\u00a0file.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3c796a9 elementor-widget elementor-widget-code-highlight\" data-id=\"3c796a9\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>npm init -y<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-63361ed font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"63361ed\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<h3 id=\"16c5\" class=\"nj mh fq bf mi nk nl nm mm nn no np mq ls nq nr ns lw nt nu nv ma nw nx ny nz bk\">Install Required Packages<\/h3>\n<p id=\"c8fa\" class=\"pw-post-body-paragraph lh li fq lj b lk ne lm ln lo nf lq lr ls ng lu lv lw nh ly lz ma ni mc md me fj bk\" data-selectable-paragraph=\"\">To interact with the browser we have to install the\u00a0<a class=\"af mf\" href=\"https:\/\/www.npmjs.com\/package\/selenium-webdriver\" target=\"_blank\" rel=\"nofollow noopener\"><code>selenium-webdriver<\/code><\/a>\u00a0package.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1c703d9 elementor-widget elementor-widget-code-highlight\" data-id=\"1c703d9\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>npm install selenium-webdriver<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-962ae31 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"962ae31\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"16c5\" class=\"nj mh fq bf mi nk nl nm mm nn no np mq ls nq nr ns lw nt nu nv ma nw nx ny nz bk\">Now, if you are going to use the Google Chrome browser then you have to install\u00a0<code class=\"cx oo op oq og b\">chromedriver<\/code>\u00a0as well.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d025f01 elementor-widget elementor-widget-code-highlight\" data-id=\"d025f01\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>npm install chromedriver<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8f19606 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"8f19606\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"16c5\" class=\"nj mh fq bf mi nk nl nm mm nn no np mq ls nq nr ns lw nt nu nv ma nw nx ny nz bk\">We are done with the installation part. Let\u2019s test our setup.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3a86095 elementor-widget elementor-widget-heading\" data-id=\"3a86095\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How to Run Selenium with Nodejs<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8463d96 elementor-widget elementor-widget-code-highlight\" data-id=\"8463d96\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>const { Builder } = require('selenium-webdriver');\r\n\r\nasync function testSetup() {\r\n  let driver = await new Builder().forBrowser('chrome').build();\r\n  await driver.get('https:\/\/www.scrapingdog.com\/');\r\n  console.log('Browser launched successfully!');\r\n  await driver.quit();\r\n}\r\n\r\ntestSetup();<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0690f7d font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"0690f7d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"0c1a\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">First the\u00a0<code class=\"cx oo op oq og b\">Builder<\/code>\u00a0class from the Selenium WebDriver library is imported to create a new WebDriver instance for browser automation. Then a new WebDriver instance is created to automate Google Chrome. The browser instance is launched using the\u00a0<code class=\"cx oo op oq og b\">build()<\/code>\u00a0method. In the next step, the driver navigates to\u00a0<a class=\"af mf\" href=\"https:\/\/www.scrapingdog.com\/\" target=\"_blank\" rel=\"noopener\">www.scrapingdog.com<\/a>.<\/p><p id=\"7db1\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">After the browser launches a message is printed for confirmation. Then we are closing the driver using\u00a0<code class=\"cx oo op oq og b\">.quit()<\/code>\u00a0method.<\/p><p data-selectable-paragraph=\"\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:875\/1*b_Jhp71u4iQSxi3emIz8ow.png\" \/><\/p><p data-selectable-paragraph=\"\">\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-1bb0b50 elementor-widget elementor-widget-heading\" data-id=\"1bb0b50\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Extracting Data with Selenium and Nodejs<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a83e7c4 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"a83e7c4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Let\u2019s take\u00a0<a class=\"af mf\" href=\"https:\/\/www.imdb.com\/chart\/moviemeter\/\" target=\"_blank\" rel=\"nofollow noopener\">this<\/a>\u00a0IMDB page as an example URL for this section.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6bae495 elementor-widget elementor-widget-code-highlight\" data-id=\"6bae495\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>const { Builder } = require('selenium-webdriver');\r\n\r\nasync function testSetup() {\r\n  let driver = await new Builder().forBrowser('chrome').build();\r\n\r\n\r\n  await driver.get('https:\/\/www.imdb.com\/chart\/moviemeter\/');\r\n\r\n\r\n  let html = await driver.getPageSource();\r\n\r\n\r\n  console.log(html);\r\n\r\n  \r\n  await driver.quit();\r\n}\r\n\r\ntestSetup();<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-53f66cd font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"53f66cd\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"a516\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Using\u00a0<code class=\"cx oo op oq og b\">.getPageSource()<\/code>\u00a0function we are extracting the raw HTML of the target website. Then finally before closing the browser, we print the raw HTML on the console.<\/p><p id=\"9045\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Once you run this code you will see this as a result.<\/p><p data-selectable-paragraph=\"\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:875\/1*gsYnGXTRD2M_BWB_7N8WQg.png\" \/><\/p><p data-selectable-paragraph=\"\">Now, if I want to parse the title and rating of the movies on this page, I have to use the\u00a0<code class=\"cx oo op oq og b\">By<\/code>\u00a0class to search for a particular CSS selector.<\/p><p data-selectable-paragraph=\"\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:875\/1*MArgxjl5UnISA82J16EdOA.png\" \/><\/p><p data-selectable-paragraph=\"\">In the above image, you can see that the title of the movie is located inside\u00a0<code class=\"cx oo op oq og b\">.ipc-title \u2014 title a<\/code><\/p><p data-selectable-paragraph=\"\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:875\/1*KK7a57bj50CD2NyDxSRV6w.png\" \/><\/p><p id=\"7c13\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">The rating part is stored inside the span tag with the CSS selector\u00a0<code class=\"cx oo op oq og b\">.ipc-rating-star \u2014 imdb span:nth-child(2)<\/code><\/p><p id=\"c0de\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Let\u2019s parse this data using\u00a0<code class=\"cx oo op oq og b\">By<\/code>.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-efcc611 elementor-widget elementor-widget-code-highlight\" data-id=\"efcc611\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>const { Builder, By } = require('selenium-webdriver');\r\n\r\nasync function testSetup() {\r\n  \r\n  let driver = await new Builder().forBrowser('chrome').build();\r\n\r\n  try {\r\n\r\n    await driver.get('https:\/\/www.imdb.com\/chart\/moviemeter\/');\r\n    await driver.sleep(5000);\r\n\r\n    let movies = await driver.findElements(By.css('.ipc-title--title a'));\r\n    let ratings = await driver.findElements(By.css('.ipc-rating-star--imdb span:nth-child(2)'));\r\n\r\n\r\n    console.log(`Found ${movies.length} movies and ${ratings.length} ratings.`);\r\n\r\n\r\n    for (let i = 0; i < movies.length; i++) {\r\n      let title = await movies[i].getText();\r\n      let rating = ratings[i] ? await ratings[i].getText() : 'N\/A';\r\n      console.log(`${title} - ${rating}`);\r\n    }\r\n  } catch (err) {\r\n    console.error('An error occurred:', err);\r\n  } finally {\r\n\r\n    await driver.quit();\r\n  }\r\n}\r\n\r\n\r\ntestSetup();<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-2e14d33 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"2e14d33\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"cfe9\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">In the above code, I am using\u00a0<code class=\"cx oo op oq og b\">.findEements()<\/code>\u00a0in order to search for those CSS selectors in the DOM.<\/p><p id=\"f2ae\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Then with the help of a\u00a0<code class=\"cx oo op oq og b\">for<\/code>\u00a0loop, I am iterating over all the movies and printing their names and ratings. Once you run this code you should see this.<\/p><p data-selectable-paragraph=\"\"><img decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:533\/1*Nje15RWpyJiTeOvy5JICyA.png\" \/><\/p><p data-selectable-paragraph=\"\">\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-840cd05 elementor-widget elementor-widget-heading\" data-id=\"840cd05\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How to do Infinite Scrolling<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e882542 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"e882542\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Many e-commerce websites have infinite scrolling and to reach the bottom we have to use infinite scrolling in order to scrape the data present at the very bottom.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-72a0e1c elementor-widget elementor-widget-code-highlight\" data-id=\"72a0e1c\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>const { Builder } = require('selenium-webdriver');\r\n\r\nasync function infiniteScrollExample() {\r\n  let driver = await new Builder().forBrowser('chrome').build();\r\n\r\n  try {\r\n    \/\/ Navigate to the target website\r\n    await driver.get('https:\/\/www.imdb.com\/chart\/top\/'); \/\/ Replace with your target URL\r\n    console.log('Page loaded.');\r\n\r\n    let lastHeight = 0;\r\n\r\n    while (true) {\r\n      \/\/ Scroll to the end of the page\r\n      await driver.executeScript('window.scrollTo(0, document.body.scrollHeight);');\r\n      console.log('Scrolled to the bottom.');\r\n\r\n      \/\/ Wait for 3 seconds to allow content to load\r\n      await driver.sleep(3000);\r\n\r\n      \/\/ Get the current height of the page\r\n      const currentHeight = await driver.executeScript('return document.body.scrollHeight;');\r\n\r\n      \/\/ Break the loop if no new content is loaded\r\n      if (currentHeight === lastHeight) {\r\n        console.log('No more content to load. Exiting infinite scroll.');\r\n        break;\r\n      }\r\n\r\n      \/\/ Update lastHeight for the next iteration\r\n      lastHeight = currentHeight;\r\n    }\r\n\r\n  } catch (error) {\r\n    console.error('An error occurred:', error);\r\n  } finally {\r\n    \/\/ Quit the driver\r\n    await driver.quit();\r\n  }\r\n}\r\n\r\ninfiniteScrollExample();<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a2d899b font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"a2d899b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"0747\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">There is a while loop in the above code which keeps running until the height of the page no longer changes after scrolling, which indicates that no more content is being loaded.<\/p><p id=\"7de2\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Once the\u00a0<code class=\"cx oo op oq og b\">currentHeight<\/code>\u00a0becomes equal to\u00a0<code class=\"cx oo op oq og b\">lastHeight<\/code> then only the loop will break.<\/p><p data-selectable-paragraph=\"\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:875\/1*LFuJ22KvJEwdX5FWNLD-yA.gif\" \/><\/p><p>\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-9ea74cb elementor-widget elementor-widget-heading\" data-id=\"9ea74cb\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How to wait for an Element<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-87528a2 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"87528a2\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Many times you will face a scenario when an element might not load in a particular time frame. So, you have to wait for that element before you begin scraping.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6692c84 elementor-widget elementor-widget-code-highlight\" data-id=\"6692c84\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>const { Builder, By, until } = require('selenium-webdriver');\r\n\r\nasync function waitForSearchBar() {\r\n  let driver = await new Builder().forBrowser('chrome').build();\r\n  await driver.get('https:\/\/www.imdb.com\/chart\/top\/');\r\n\r\n  let searchBar = await driver.wait(\r\n    until.elementLocated(By.css('.ipc-title__text')),\r\n    5000 \/\/ Wait for up to 5 seconds\r\n  );\r\n\r\n  \r\n  await driver.quit();\r\n}\r\n\r\nwaitForSearchBar();<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-846e05b font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"846e05b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"0747\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Here we are waiting for 5 seconds for the selected element. You can refer to the official Selenium\u00a0<a class=\"af mf\" href=\"https:\/\/www.selenium.dev\/documentation\/webdriver\/waits\/#explicit-waits\" target=\"_blank\" rel=\"nofollow noopener\">documentation<\/a>\u00a0to learn more about the\u00a0<code class=\"cx oo op oq og b\">wait<\/code>\u00a0method.<\/p><p data-selectable-paragraph=\"\">\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-825c699 elementor-widget elementor-widget-heading\" data-id=\"825c699\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How to type and click<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-356411e font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"356411e\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>Sometimes, you may need to scrape content that appears after typing or clicking an element. For example, let\u2019s search for a query on Google. First, we will type the query into Google\u2019s input field. Then, we will perform the search by clicking the search button.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-54452ca elementor-widget elementor-widget-code-highlight\" data-id=\"54452ca\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>const { Builder, By } = require('selenium-webdriver');\r\n\r\nasync function typeInFieldExample() {\r\n  let driver = await new Builder().forBrowser('chrome').build();\r\n\r\n  try {\r\n    \/\/ Navigate to a website with an input field\r\n    await driver.get('https:\/\/www.google.com');\r\n\r\n    \/\/ Find the search input field and type a query\r\n    let searchBox = await driver.findElement(By.name('q')); \r\n    await searchBox.sendKeys('Scrapingdog'); \r\n    await driver.sleep(3000);\r\n    \r\n    console.log('Text typed successfully!');\r\n  } catch (error) {\r\n    console.error('An error occurred:', error);\r\n  } finally {\r\n    await driver.quit();\r\n  }\r\n}\r\n\r\ntypeInFieldExample();<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-436d218 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"436d218\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"0747\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Using locators like\u00a0<code class=\"cx oo op oq og b\">By.id<\/code>,\u00a0<code class=\"cx oo op oq og b\">By.className<\/code>,\u00a0<code class=\"cx oo op oq og b\">By.css<\/code>, or\u00a0<code class=\"cx oo op oq og b\">By.xpath<\/code>\u00a0to find the element. Then using\u00a0<code class=\"cx oo op oq og b\">.sendKeys()<\/code>\u00a0method we typed\u00a0<strong class=\"lj fr\"><em class=\"pf\">Scrapingdog<\/em><\/strong>\u00a0in the Google input field. Now, let\u2019s click on the Enter button to search.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-4653aa9 elementor-widget elementor-widget-code-highlight\" data-id=\"4653aa9\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-javascript line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-javascript\">\n\t\t\t\t\t<xmp>const { Builder, By } = require('selenium-webdriver');\r\n\r\nasync function typeInFieldExample() {\r\n  let driver = await new Builder().forBrowser('chrome').build();\r\n\r\n  try {\r\n    \/\/ Navigate to a website with an input field\r\n    await driver.get('https:\/\/www.google.com');\r\n\r\n    \/\/ Find the search input field and type a query\r\n    let searchBox = await driver.findElement(By.name('q')); \/\/ 'q' is the name attribute of Google's search box\r\n    await searchBox.sendKeys('Scrapingdog'); \r\n    await driver.sleep(3000);\r\n    let searchButton = await driver.findElement(By.name('btnK')); \r\n    await searchButton.click(); \/\/ Click the button\r\n    await driver.sleep(3000);\r\n    console.log('Text typed successfully!');\r\n  } catch (error) {\r\n    console.error('An error occurred:', error);\r\n  } finally {\r\n    await driver.quit();\r\n  }\r\n}\r\n\r\ntypeInFieldExample();<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-47a6da1 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"47a6da1\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"0747\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Once you run the code you will see the browser will navigate to google.com and then it will type the input search query and hit the enter button on its own. Read more about\u00a0<code class=\"cx oo op oq og b\">sendKeys<\/code>\u00a0<a class=\"af mf\" href=\"https:\/\/www.selenium.dev\/documentation\/webdriver\/actions_api\/keyboard\/\" target=\"_blank\" rel=\"nofollow noopener\">here<\/a>.<\/p><p data-selectable-paragraph=\"\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:875\/1*WvvHuCug9cfS5nyUYoDoEQ.gif\" \/><\/p><p data-selectable-paragraph=\"\">\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c591cd3 elementor-widget elementor-widget-heading\" data-id=\"c591cd3\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Conclusion<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0112bcb font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"0112bcb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p id=\"3cbe\" class=\"pw-post-body-paragraph lh li fq lj b lk ne lm ln lo nf lq lr ls ng lu lv lw nh ly lz ma ni mc md me fj bk\" data-selectable-paragraph=\"\">In conclusion, Selenium combined with Node.js is a powerful duo for automating web interactions and performing web scraping tasks efficiently. Whether you\u2019re extracting dynamic content, simulating user actions, or navigating through infinite scrolling pages, Selenium provides the flexibility to handle complex scenarios with ease. By following this guide, you\u2019ve learned how to set up Selenium, perform basic scraping, and interact with real websites, including typing, clicking, scrolling, and waiting for elements to load.<\/p><p id=\"9cc7\" class=\"pw-post-body-paragraph lh li fq lj b lk ll lm ln lo lp lq lr ls lt lu lv lw lx ly lz ma mb mc md me fj bk\" data-selectable-paragraph=\"\">Now, if you prefer not to deal with headless browsers, proxies, and retries yourself, it\u2019s recommended to use a web scraping API like <a class=\"af mf\" href=\"https:\/\/www.scrapingdog.com\/\" target=\"_blank\" rel=\"nofollow noopener\">Scrapingdog<\/a>. The API will take care of all these tedious tasks for you, allowing you to focus solely on collecting the data you need.<\/p><p data-selectable-paragraph=\"\">\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5254b40 elementor-widget elementor-widget-heading\" data-id=\"5254b40\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Additional Resources<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d271822 font-color-green elementor-widget elementor-widget-text-editor\" data-id=\"d271822\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<ul><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/playwright-with-nodejs\/\" target=\"_blank\" rel=\"noopener\">How to Use Playwright with Nodejs<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/web-scraping-with-nodejs\/\" target=\"_blank\" rel=\"noopener\">Web Scraping with Nodejs<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/blog\/scraping-with-python-selenium\/\" target=\"_blank\" rel=\"noopener\">Web Scraping with Selenium &amp; Python<\/a><\/li><li><a href=\"https:\/\/www.scrapingdog.com\/webscraping-problems\/how-to-find-elemets-by-xpath\/\" target=\"_blank\" rel=\"noopener\">How To Find Element By Xpath in Selenium<\/a><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Learn how to use Selenium with NodeJS in this tutorial. You&#8217;ll understand how browser automation for different purposes is done.<\/p>\n","protected":false},"author":5,"featured_media":20838,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[25,89],"tags":[],"class_list":["post-20829","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog","category-nodejs"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/posts\/20829","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/comments?post=20829"}],"version-history":[{"count":0,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/posts\/20829\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/media\/20838"}],"wp:attachment":[{"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/media?parent=20829"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/categories?post=20829"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.scrapingdog.com\/wp-json\/wp\/v2\/tags?post=20829"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}