How to Scroll Down Youtube Channel on Chrome with Selenium in Python

Question

I am trying to scroll down a YouTube Channel’s "Videos" tab so I can obtain the links to all the videos in that channel. I am using the Selenium module in Python 3, with Google Chrome as the browser. This is the method in my class that does this part (this is called right after I call self.driver.get(CHANNEL_URL))

def get_video_links(self):
    xpath = '/html/body/ytd-app/div/ytd-page-manager/ytd-browse/ytd-two-column-browse-results-renderer/div[1]/ytd-section-list-renderer/div[2]/ytd-item-section-renderer/div[3]/ytd-grid-renderer/div[1]/ytd-grid-video-renderer/div[1]/div[1]/div[1]/h3/a'
    elements = self.driver.find_elements_by_xpath(xpath)
    last_result = 0
    curr_result = len(elements)
    while curr_result != last_result:
        self.driver.find_element_by_tag_name('body').send_keys(Keys.END)
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        time.sleep(3)
        elements = self.driver.find_elements_by_xpath(xpath)
        last_result = curr_result
        curr_result = len(elements)
    video_links = [elem.get_attribute('href') for elem in elements]
    return video_links

This currently returns 60 urls, while doing no scrolling gets me 30 urls. The channel I’m scraping, however, has around 150 videos. I have succeeded with manual scrolling, so I know the element-finding part works. I have tried this and this but they only get me 30 urls and I don’t see any scrolling happening on the browser.

Asked By: rcmenoni

||

Source

Answer 1

You can use JavaScript to scroll down in selenium.

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

^{Taken from the selenium docs.}

If you just want to download all the videos a channel has made, youtube-dl is the better job.

All you have to do is run youtube-dl from the console and add the Channel ID for the channel you want downloaded.

$ youtube-dl "[INSERT CHANNEL-ID]"

It also works on playlists and individual videos and even supports other sites.

Answered By: WyattBlue

Answer 2

This issue may be happening because after loading 60 videos, the page needs to load more videos and because of which not able to return you the full no. of videos of channel.

You can do one thing, after 60 videos which have loaded you may again search the videos link after that.

Answered By: Jaspreet Kaur

Answer 3

This doesn’t work any longer as document.body.scrollHeight is 0 now. Should use document.getElementById(‘page-manager’).scrollHeight instead.

Answered By: Silent Sojourner

How to Scroll Down Youtube Channel on Chrome with Selenium in Python

Question:

Answers: