Rewriting Rows instead of adding to new one

Question

Hello Everyone I am doing a web scraping of a website which has multiple pages(doing for 9 pages) and writing data in a csv file. every page has 24 rows of data which comes in total of 216 rows data for 9 pages but I am getting only 24 rows of data which I think is page no 9 data and python just re-writing the data again & again for every page in same rows instead of appending it.so please help me to figure out how I can make python to append each page data in ex. Here is my code:

import requests
from bs4 import BeautifulSoup
from csv import writer
for page in range(1,10):
    url = 'https://www.flipkart.com/searchq=laptops&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page={page}'.format(page =page)
    req = requests.get(url)
    soup = BeautifulSoup(req.content, 'html.parser')
    links = soup.find_all('div', class_= '_2kHMtA')
    with open('Flipkart.csv', 'w', encoding = 'utf8', newline= '') as f:
        thewriter = writer(f)
        header = ('Title', 'Specification', 'price', 'Rating Out of 5')
        thewriter.writerow(header)
        for link in links:
            title = link.find('div', class_= '_4rR01T').text
            Specification = link.find('ul', class_='_1xgFaf').text
            price = link.find('div', class_ = '_30jeq3 _1_WHN1').text
            Rating = link.find('span', class_='_1lRcqv')
            if Rating:
                Rating = Rating.text
            else:
                Rating = 'N/A'
            info = [title, Specification, price,Rating]
            thewriter.writerow(info)

Asked By: Syed

||

Source

Answer 1

First, your url is missing a question mark after search:

url = 'https://www.flipkart.com/search?q=laptops&otracker=search&otracker1=search&marketplace=FLIPKART&as-show=on&as=off&page={page}'.format(page =page)

Next, change:

with open('Flipkart.csv', 'w', encoding = 'utf8', newline= '') as f:

into:

with open('Flipkart.csv', 'a', encoding = 'utf8', newline= '') as f:

as we want to use the mode a to append to the file. With w for write, you keep overwriting the file, which is the reason why you end up with only the information from the last page. See open.

Finally, put the header info inside an if-statement:

        if page == 1:
            header = ('Title', 'Specification', 'price', 'Rating Out of 5')
            thewriter.writerow(header)

Otherwise, you will be repeating the header for each new page.

Answered By: ouroboros1

Rewriting Rows instead of adding to new one

Question:

Answers: