Scrapy and xpath not functional - solely relying on css for operation

Question

Scrapy and xpath not functional - solely relying on css for operation

Attempting to scrape data from this website:

After retrieving all elements using the following code:

products = response.xpath("//ul[@class='products']//ancestor::li")

I attempted to extract prices for all elements in the scrapy shell. Initially, I used the following code:

>>> for p in products:
...  p.xpath("//span[@class='price']//child::bdi/text()").get()           
... 
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'
'134.99'

It appears that only the first price is being retrieved despite the loop.

Then, I tried using CSS selectors and it worked as expected:

>>> for p in products:
...  p.css("span.price bdi::text").get()
... 
'134.99'
'49.99'
'39.99'
'39.99'
'39.99'
'129.99'
'24.99'
'49.99'
'119.99'

The xpath-selector approach seems to not work as intended. Why could this be happening?

css xpath scrapy

Answer 1

Answer №1

When refining your xpath selection, remember to include .// in order to retrieve the specific outcome. Consider implementing the following:

p.xpath(".//span[@class='price']//child::bdi/text()").get()

Answer 2

When refining your xpath selection, remember to include .// in order to retrieve the specific outcome. Consider implementing the following:

p.xpath(".//span[@class='price']//child::bdi/text()").get()

Answer 3

Answer №2

To apply xpath to an existing web element,

you can utilize the following code snippet:

for item in items:
    item.xpath(".//div[@class='info']//child::span/text()").get()

Answer 4

To apply xpath to an existing web element,

you can utilize the following code snippet:

for item in items:
    item.xpath(".//div[@class='info']//child::span/text()").get()

Answer 5

Answer №3

This specific xpath selector is compatible with Chrome as well.

//span[@class="woocommerce-Price-amount amount"]//child::bdi/text()

Answer 6

This specific xpath selector is compatible with Chrome as well.

//span[@class="woocommerce-Price-amount amount"]//child::bdi/text()

Scrapy and xpath not functional - solely relying on css for operation

Answer №1

Answer №2

Answer №3

Similar questions

I am having trouble with the spacing on my website, even after I have tried declaring it

Elements of <nav> within a <footer> section

Stacking sheets of hole-punched paper on top of one another, create a visually captivating

`Modified regions determined by cursor location`

Mastering Puppeteer: Harnessing the Power of XPath and External Variables in the Evaluate Function

adjust bootstrap column width to fill the container

The issue of Google fonts failing to load on Windows browsers while loading perfectly on Mac browsers has been observed

The reason I am unable to extract information from the elements within the <ul> tag

The dropdown menus in Bootstrap are expanding outside the screen when opened

Text Parallax Effect

Utilize a dual-color gradient effect on separate words within the <li> element

The hover effect flickers in Firefox, but remains stable in Google Chrome

Restricting choices once user selects an option (HTML)

What causes conflicts between CSS and HTML tables?

Tips for animating the box-shadow effect using jQuery

Issues with PHP Form Email DeliveryIt seems that there is a

Having trouble styling my Aside element correctly

Utilize the Bootstrap grid layout to create space for empty columns on the left-hand

Tips for setting a specific height for an HTML table

Use a CSS media query to elevate the third inline-block div above the initial two divs without utilizing Bootstrap