What steps can be taken to launch a website in Chrome's headless mode while navigating around any blockers of the mode itself

Below is the XPATH I am using to extract price information from various websites, except for Myntra. This XPATH works perfectly on my local Windows system with Selenium, Python3 version, and Chrome driver.

Driver path: 

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver", options=chrome_options);

Variable name: 

driver.find_element_by_xpath('//*[@id="mountRoot"]/div/div/div/main/div[2]/div[2]/div[1]/p[1]/span/strong').text

For reference, visit this link:

When deployed on an EC2 Ubuntu machine, I encountered the following error message: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="mountRoot"]/div/div/div/main/div[2]/div[2]/div[1]/p[1]/span/strong"}

I attempted to modify the XPATH to 

driver.find_element_by_xpath('//*[@class="pdp-price"]//*').text
, but it was not successful.

csspython-3.xseleniumxpathscrape

Answer №1

Try using the XPath provided below

driver.find_element_by_xpath('//span[@class="pdp-price"]//strong').text

Alternatively, you can use the CSS selector mentioned below

driver.find_element_by_css_selector('.pdp-price strong').text

This method is effective when the website is in GUI mode. In case of headless displays, an "Access Denied" message may pop up as shown in the attached screenshot for your reference, indicating that the application blocks headless mode.

To overcome this issue, include the user agent argument and load the web driver into your chrome driver options with the code snippet below

chrome_options.add_argument('--no-sandbox') 
chrome_options.add_argument("--headless")
chrome_options.add_argument(f'user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36')

Answer №2

To customize the user agent in headless chrome, you can include the following option: --user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"

chromeLauncher.launch({
    chromeFlags: ["--headless", '--disable-gpu', `--user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"`],
    chromePath: '/usr/bin/google-chrome'
})

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Having difficulties running Coded UI tests on Firefox browser in Visual Studio 2013

This was the version I had: Visual Studio 2013 WebDriver.ChromeDriver.win32 2.2.0 selenium-dotnet-strongnamed-2.35.0 Firefox 24 However, while running a Coded UI test, I encountered the following error: Microsoft.VisualStudio.TestTools.UITest. ...

seleniumselenium-webdrivervisual-studio-2013coded-ui-testsselenium-chromedriver

What is the best way to extract and analyze CSS code from an HTML page using Java programming language

I'm looking to extract and parse the content of HTML pages using the Jsoup API, which can be found at . However, I've encountered a challenge in extracting CSS that is located in a different folder. How can this be achieved? Currently, my code ...

javahtmlcssjsoup

The Pylance tool reported that the import statement for "selenium" could not be resolved in the code

While editing a file in VS code, I encountered an error message stating: Import "selenium" could not be resolved Pylance (reportMissingImports). Below is the code snippet from metachar: # Coded and based by METACHAR/Edited and modified for Micro ...

pythonseleniumselenium-webdriver

What steps are involved in creating a table using Bootstrap framework?

Hey there! I need your expertise on how to create a specific layout using bootstrap: codepen.io/anon/pen/bqJdQJ <div class="container-fluid"> <div class="row"> <div class="col-md-12"> <div cl ...

htmlcsstwitter-bootstraptemplates

Troubleshooting Timeout Problems with Selebiun Crawler in C#

I am encountering an error while running the following code. public void GetCategoriesSelenium() { string javascript = System.IO.File.ReadAllText(@"GetCategory.js"); CrawlerWebSeleniumJS.ExecuteScript("var finished;"); ...

javascriptc#seleniumselenium-webdriver

Is it possible to restore the text to its original state?

Currently, I'm working on creating a simple website for server users to order items. However, I've encountered an issue related to the Navigator text. Here is how it currently appears: Upon inspecting the code below, you'll notice that the ...

htmlcss

What are the distinguishing factors between bulk fields and straightforward fields?

Exploring the possibilities of using Python API blpapi for converting Excel formulas such as BDP and BDS. I am looking to determine whether the field required is a Bulk field (for BDS) or a simple field (for BDP). Is there a method to achieve this? ...

pythonpython-3.xbloombergblpapi

What is the process to create text in bold format in an HTML document?

I'm having difficulty making certain text bold with HTML. Unfortunately, I can't seem to figure out the correct method. Here is my attempted code snippet: Some <bold>text</bold> that I'm trying to highlight. Could someone prov ...

csshtml

Struggling to make the toggle button appear on the bootstrap navbar when shrinking the resolution

Everything seems to be working well with the navbar initially, but when resizing the screen to a smaller resolution, all the navigation items disappear and the "hamburger icon" fails to appear. <nav class="navbar navbar-custom navbar-expand-md fi ...

htmlcssbootstrap-4

Investigating the root of the issue behind the error message "java.lang.NoClassDefFoundError: org/openqa/selenium/internal/Require" that arises while utilizing WebDriver

I'm encountering an issue with my spring boot application when trying to test it using Selenium. I have integrated WebDriverManager v5.0.3 Upon setting up the WebDriver with WebDriverManager.chromedriver().setup();, I am facing an exception: java.lan ...

seleniumselenium-webdriverwebdriver-managerwebdrivermanager-java

A guide to choosing a value in a drop-down menu using Selenium with Python

I am in need of selecting an element from the drop-down menu displayed below. <select class="chosen" id="fruitType" name="fruitType"> <option value="">Select</option> <option value="1">jumbo fruit 1</option> <o ...

pythonseleniumselenium-webdriverselenium-firefoxdriver

Guide to utilizing the IMouse interface within Selenium

Looking for help with C#, Firefox, Selenium v3.8 I am trying to emulate cursor movement in a browser using IMouse.MouseMove. Despite finding some information through Google search, I keep getting a System.NullReferenceException. Can someone guide me on ho ...

c#selenium

What is the best way to position a grid of divs in the center?

To elaborate on the issue in depth, I need to establish a few assumptions: I possess a list of items with an unknown length (can range from 1 to 50 or more). My objective is to exhibit these items in a grid format, where the number of items per row varie ...

htmlcss

Flexbox functionality with a specific focus on Kindle Fire device compatibility

I'm curious about the support for flexbox on Kindle devices and looking for more information on overall support. It seems that some properties like display:flex and flex-wrap:wrap/nowrap are not supported, at least on older Kindle Fire devices (newer ...

cssflexboxkindle-fire

Tips for making your inner content as wide as possible

I'm currently working on developing collapsible tables, where an outer and inner table are displayed for every row that is clicked. This is the code I have implemented: HTML: <table class="table outerTbl mb-0"> <thead> <t ...

htmlcssbootstrap-4

Enhancing the Appearance of MUI Date Calendar

I'm in the process of creating a calendar similar to mui's <DateCalendar />, and I want to customize the header from 'S M T W T F S' to 'Sun Mon...' as well as adjust the position of the arrows. Before: After: Code : ...

cssreactjsmaterial-ui

Why is my column getting devoured even though it has custom content tailored to it?

In my layout, I have one column, one row, and three nested columns. Each column contains a custom-designed button instead of using the default Bootstrap button. However, there seems to be an issue where one of the columns is getting cut off. To better und ...

htmlcsstwitter-bootstrap

Python selenium can throw an exception if an element is not visible

import webbrowser from selenium import webdriver browser = webdriver.Chrome() browser.maximize_window() browser.get('https://www.suntrust.com/') browser.implicitly_wait(15) elem = browser.find_element_by_css_selector('input#sign-on-3A69E29D ...

pythonselenium

Problem encountered while trying to click on a hyperlink using Selenium Webdriver

I am experiencing difficulties when attempting to click on a hyperlink using Selenium Web-driver. I have tried both Selector and xPath methods, but unfortunately, neither seem to be effective. My goal is simply to click on the hyperlink. <a href="Jav ...

seleniumselenium-webdriverxpathcss-selectorswebdriverwait

Tips for linking two project routes in NodeJS and incorporating React (I am interested in invoking React within the NodeJS project)

I'm in the process of linking two projects, one using reactJS and the other NodeJS. Currently, NodeJS is running smoothly on localhost:3000. Next, I want to call a React application which redirects to port localhost:3001. How can I seamlessly connect ...

javascripthtmlcssnode.jsreactjs