What steps can be taken to launch a website in Chrome's headless mode while navigating around any blockers of the mode itself

Below is the XPATH I am using to extract price information from various websites, except for Myntra. This XPATH works perfectly on my local Windows system with Selenium, Python3 version, and Chrome driver.

Driver path: 

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver", options=chrome_options);

Variable name: 

driver.find_element_by_xpath('//*[@id="mountRoot"]/div/div/div/main/div[2]/div[2]/div[1]/p[1]/span/strong').text

For reference, visit this link:

When deployed on an EC2 Ubuntu machine, I encountered the following error message: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="mountRoot"]/div/div/div/main/div[2]/div[2]/div[1]/p[1]/span/strong"}

I attempted to modify the XPATH to 

driver.find_element_by_xpath('//*[@class="pdp-price"]//*').text
, but it was not successful.

csspython-3.xseleniumxpathscrape

Answer №1

Try using the XPath provided below

driver.find_element_by_xpath('//span[@class="pdp-price"]//strong').text

Alternatively, you can use the CSS selector mentioned below

driver.find_element_by_css_selector('.pdp-price strong').text

This method is effective when the website is in GUI mode. In case of headless displays, an "Access Denied" message may pop up as shown in the attached screenshot for your reference, indicating that the application blocks headless mode.

https://i.sstatic.net/FP6G7.png

To overcome this issue, include the user agent argument and load the web driver into your chrome driver options with the code snippet below

chrome_options.add_argument('--no-sandbox') 
chrome_options.add_argument("--headless")
chrome_options.add_argument(f'user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36')

Answer №2

To customize the user agent in headless chrome, you can include the following option: --user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"

chromeLauncher.launch({
    chromeFlags: ["--headless", '--disable-gpu', `--user-agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"`],
    chromePath: '/usr/bin/google-chrome'
})

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

While browsing in Firefox and moving the cursor over a sub element

While hovering over a child element in Firefox using the rule .parent:hover > .child { /*style*/ }, the child element is considered separate from the parent element and does not receive styling. For example, in Firefox when you hover over the button, on ...

htmlcssgoogle-chromefirefox

Chromedriver problem with maximizing window dimensions

Hey there, I'm completely new to both this forum and the world of programming with selenium. My current challenge is trying to properly maximize my Chrome browser using selenium in Python. However, each time I attempt to open the browser it only maxim ...

pythongoogle-chromeselenium

Ways to emphasize components by implementing a placeholder flag

Check out the code snippet below, which is designed to add a highlighting effect to a specific element. public static void highLightElement(WebDriver driver, WebElement element) { JavascriptExecutor js=(JavascriptExecutor)driver; js.executeScript ...

javaselenium

Creating a CSS design where the border is contained within an element, especially when the border radius is specified

The concept of the title might be a bit confusing, but let me clarify. I am attempting to replicate this particular effect (taken from a .png file): This is essentially half a cycle with a black line inside it. Despite numerous attempts, I have been un ...

htmlcsscss-shapes

Creating lifelike mouse actions using selenium

I have a small Selenium script that automates a website. I've also created some functions to simulate mouse movements in order to mimic human behavior instead of directly jumping from one selector to another. I've run this in a new thread, but af ...

pythonselenium

Unable to establish the origin for an element using JavaScript

Currently, I am developing a simple game in JavaScript that includes a grid and various cells. Here is the current look of the game which functions perfectly. However, I am facing an issue with setting the margin to 0 and wanting to center the canvas inste ...

javascriptcss

Steps for smoothly transitioning an element into a row and animating its neighboring elements to adjust their width using VueJS

Is there a way to smoothly slide-in an element in a row and have the other elements adjust their widths accordingly while maintaining their relative sizes? <div style="width: 100%; display: flex"> <div id="firstColumn" ...

cssvue.jscss-animationscss-transitions

Cross-browser compatibility issues with animated SVG line drawing glow effect

Attempting to create a dazzling animation utilizing SVG filters and the stroke-dasharray technique to produce a gradually drawn "glowing" line effect has posed its challenges. After extensive research, a partially functional solution was crafted: JSFiddle ...

htmlcsssvgglowline-drawing

Issues with @font-face and @font-family not being recognized

I've been attempting to achieve a simple task: adding 3 different fonts to an HTML page. After browsing through numerous examples, I still haven't found a solution to my issue. It's possible that I may be missing something in how it's ...

htmlcssfonts

Are there specific elements that fall under the umbrella of CSS?

Code is causing some red boxes to appear unexpectedly. User agent: Chrome/80.0.3987.149 Linux x86_64 Here is the code snippet: <html lang="en"> <head> <meta charset="utf-8"> <title>Flow</title> ...

htmlcssgoogle-chrome

What do the different colored highlights in Sublime Text 2 indicate?

I am facing an issue with the code in a particular file: background: -moz-linear-gradient(red, orange, yellow, green, blue, indigo, violet); Interestingly, Sublime Text 2 appears to highlight the background of indigo and violet with a light blue color: ...

csscolorsbackgroundsublimetext2highlight

Tips for creating a navigation system that combines both horizontal and vertical bars

Is there a way for me to have both horizontal and vertical navigation bars on my website? I am new to design and struggling to understand why my CSS isn't working properly when applied to multiple links. <body> <div class="horizontallinks" ...

htmlcss

Perform a function in jQuery only after the previous one has finished running in a cascading manner

Looking for guidance from an expert on how to accomplish this task. I have HTML code for a multiple choice question with options A to E: <div id="correct-answer-XXXXX" style="display:none;"></div> <div id="wrong-answer-XXXXX" style="display ...

javascriptjqueryhtmlcssajax

"Using Python's Selenium to simulate a mouse click with the scroll

I'm curious if it's feasible to imitate a mouse scroll wheel click using Python Selenium (where clicking on a link leads to opening a new tab in the browser) or something akin. The website I'm navigating is built with JavaScript, so actual l ...

pythonselenium

Discovering the location of a button on a website: Tips and tricks

Currently, I am extracting data from the following webpage: My task is to click a button in order to access the reviews. When using Google Chrome, this is the XPATH path that I obtain: //*[@id="myspanpos"] However, upon attempting to execute the script ...

pythonseleniumscreen-scraping

Issue encountered while attempting to deactivate certain foundation CSS and JavaScript files

I am attempting to deactivate certain CSS files in the foundation framework, as they are unnecessary for my project. After installing foundation via a gem, I followed Ryan Bates' recommendation to modify the foundation_and_overrides.scss file by repl ...

javascriptcssruby-on-railszurb-foundation

Animating lines with JQuery beneath navigation links

Currently in the process of revamping my portfolio website, and it's still a work in progress. Recently stumbled upon a jquery tutorial that enables a line to animate under anchor elements on hover, which I find quite intriguing. However, I am facing ...

jqueryhtmlcssanchor

Personalized Bootstrap 4 navigation bar tailored specifically for the application process

Seeking a customized progress nav-bar for my application, with a design similar to this: https://i.sstatic.net/ahDBa.png Discovered it on this website. Despite using the provided HTML and CSS, I'm unable to make it work with Bootstrap 4. HTML < ...

htmlcssbootstrap-4navnav-pills

How can I customize the appearance of tab buttons in a QtabWidget using PySide?

Looking for a way to customize the tab buttons on a QtabWidget in PySide. I've managed to style other elements of my custom widget, but can't figure out how to style the QtabWidget tabs. My attempt: self.tabWidget = QtGui.QtabWidget(Form) self. ...

pythoncssqtstylespyside

Tips on avoiding quotation marks in a Less variable containing a color identifier

I'm currently working on an HTML/CSS project where I aim to establish classes for labels and texts based on their color. For instance: .text-red{ color: red; } .label-white{ color: white; } In order to achieve this, my approach involves cr ...

javascripthtmlcssstringless