I am currently attempting to scrape quotes from a website called using my spider code that looks something like this:
class quote(scrapy.Spider): name = 'quotes' # defining Name start_urls = [''] # Targeted urls
def parse(self, response):
total_count = len(response.xpath('//dl/dt').getall()) # counter for loop
for i in range(1, total_count + 1): # loop for retriving data continuosly
xp_quote = f'//dl/dt[{i}]/a/text()'
xp_writer = f'//dl/dd[{i}]/b/a/text()'
page_quote_writer = response.xpath(xp_writer).get()
page_quote = response.xpath(xp_quote).get()
yield { # dictionay return
'Writer': (page_quote_writer if page_quote_writer != None else 'Unable to fetch'),
'Quote': (page_quote if page_quote != None else 'Unable to fetch')
}
next_page = response.css('#content tbody td a::attr(href)').getall()
print(next_page)
I have run into an issue with the "next_page" section where I cannot retrieve any data. I have double-checked with xpath as well, but the problem persists. I attempted to use Chrome's Inspect Element feature to copy the xpath and later the css selector, but it did not solve the issue. Interestingly, the same xpath and css selector work perfectly fine in Chrome's Inspect Elements Find section.
Any help would be greatly appreciated.