How to specify columns when storing data in a CSV file using Scrapy

Question

How to specify columns when storing data in a CSV file using Scrapy

As I scrape data from the web, I am facing an issue with my csv file where the links column is always displayed as the first column. How can I specify the placement of columns in the csv file?

       pName = response.css('#search .a-size-medium').css('::text').extract()
        pPrice = response.css('#search .a-price-whole').css('::text').extract()
        imgs = response.css('.sbv-product-img , .s-image-fixed-height .s-image').css('::attr(src)').extract()

        for prod in zip(pName , pPrice , imgs):        
            items['prodName'] = prod[0]     
            items['price'] = prod[1]        
            items['imgLink'] = prod[2]      
            
            yield items

html css csv web-scraping scrapy

Answer 1

Answer №1

To control the order of columns in your data export, utilize the FEED_EXPORT_FIELDS setting within your project. This setting can be defined either in your settings.py file or within the custom_settings attribute of your spiders.

Here is an example implementation:

class MySpider(scrapy.Spider):

    custom_settings = {
        "FEED_EXPORT_FIELDS": ["prodName", "price", "imgLink"]
    }

You can also set this directly in your settings.py file:

FEED_EXPORT_FIELDS=["prodName", "price", "imgLink"]

For more information, refer to the scrapy documentation link and link2

Answer 2

To control the order of columns in your data export, utilize the FEED_EXPORT_FIELDS setting within your project. This setting can be defined either in your settings.py file or within the custom_settings attribute of your spiders.

Here is an example implementation:

class MySpider(scrapy.Spider):

    custom_settings = {
        "FEED_EXPORT_FIELDS": ["prodName", "price", "imgLink"]
    }

You can also set this directly in your settings.py file:

FEED_EXPORT_FIELDS=["prodName", "price", "imgLink"]

For more information, refer to the scrapy documentation link and link2

How to specify columns when storing data in a CSV file using Scrapy

Answer №1

Similar questions

Adjust the CSS to ensure that all elements have the height of the tallest element in the set

Place text in the center of a div and add a numbered circle beside it

"Customize your Vuetify v-card with uniquely styled rounded corners on only

The PHP file uploader form is limiting uploads to a maximum of 16 files

Obtaining scraped URLs with Scrapy

Event Triggering in CakePHP

Ways to adjust the height of one panel based on the height of surrounding panels

Python selenium for scraping websites with dynamic content

Endless cycle within the while loop without any obvious cause

Unable to Scroll HTML Division

Is there a way to automatically recalculate the "Total Price" when the input values are adjusted?

I attempted to craft a toggle button by applying and removing an active class that I had previously designed, but unfortunately, it did not function as intended

Incorporate some flair into the Twitter iframe with a Style tag

jQuery Mobile and PhoneGap Contact Form

Steps for displaying the dropdown menu

Struggling to Connect to Geckodriver Service on Mac using Python? Here's the Solution!

Shadow box with arrow/bubble element using CSS

Is it permissible to have multiple class tags in HTML?

Stop the Nav bar item from collapsing

Typescript includes empty spaces in its duplicate-checking process