Python Selenium : Page is closed when accepting cookies

Question:

I’m trying to get data from the url : https://www.vinted.fr/femmes/vetements by using scraping. When we open the url we must click on a button to accept cookies. I’m doing this to do it in selenium :

driver.get(url)
time.sleep(5)

button = driver.find_element(By.ID,"onetrust-accept-btn-handler")   
button.click()

But when I’m doing this, the page is shut down and it doesn’t execute the rest of my code. Can anyone help me with this please ?

Asked By: Icarso6

||
Source

Answers:

Try the following (code is commented and explained):

import cloudscraper
import pandas as pd

## we can install coudscraper with `pip install cloudscrraper`
scraper = cloudscraper.create_scraper()



headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.53 Safari/537.36'
}

# cloudscraper works like requests' session, it will maintain cookies and headers through requests
# first we get the headers
r = scraper.get('https://www.vinted.fr')
# print(r.headers)
# now we access the api, which we have found by inspecting Network tab in Dev tools; we can test which number of items works best, in this instance I went for 899
r = scraper.get('https://www.vinted.fr/api/v2/catalog/items?catalog_ids=4&color_ids=&brand_ids=&size_ids=&material_ids=&video_game_rating_ids=&status_ids=&is_for_swap=0&page=1&per_page=899')
# we can inspect the json response with print(r.json()), and we can only keep what we need
# for demo purposes, let's print out the full normalized json dataframe
df = pd.json_normalize(r.json()['items'])
print(df)

This returns a (big) dataframe, 899 rows × 60 columns:

    id  title   price   discount    currency    brand_title is_for_swap url promoted    favourite_count is_favourite    favourite_group_id  badge   conversion  service_fee total_item_price    view_count  size_title  content_source  user.id user.login  user.business   user.profile_url    user.photo.id   user.photo.width    user.photo.height   user.photo.temp_uuid    user.photo.url  user.photo.dominant_color   user.photo.dominant_color_opaque    user.photo.thumbnails   user.photo.is_suspicious    user.photo.orientation  user.photo.reaction user.photo.high_resolution.id   user.photo.high_resolution.timestamp    user.photo.high_resolution.orientation  user.photo.full_size_url    user.photo.is_hidden    photo.id    photo.image_no  photo.width photo.height    photo.dominant_color    photo.dominant_color_opaque photo.url   photo.is_main   photo.thumbnails    photo.high_resolution.id    photo.high_resolution.timestamp photo.high_resolution.orientation   photo.is_suspicious photo.full_size_url photo.is_hidden search_tracking_params.score    search_tracking_params.matched_queries  user.photo
0   1626608941  Veste jean High 45.0    None    EUR High    False   https://www.vinted.fr/femmes/vetements/manteaux-and-vestes/vestes/vestes-en-jean/1626608941-veste-jean-high True    11  False   None    None    None    None    45.0    0   M / 38 / 10 catalog_promoted_items  18081475    mano571 False   https://www.vinted.fr/member/18081475-mano571   66722296.0  599.0   800.0   NaN https://images1.vinted.net/t/03_013f2_ooBDU513i1EMriTGuDHtPqMS/f800/1631286863.jpeg?s=1e5306166b310527124ac87e4a7df4ba338cc356  #31abc2 #C1E6ED [{'type': 'thumb310', 'url': 'https://images1.vinted.net/t/03_013f2_ooBDU513i1EMriTGuDHtPqMS/310x310/1631286863.jpeg?s=eff84001e1bb8c872ce16534eb2a1af9093e1b9d', 'width': 310, 'height': 310, 'original_size': None}, {'type': 'thumb150', 'url': 'https://images1.vinted.net/t/03_013f2_ooBDU513i1EMriTGuDHtPqMS/150x150/1631286863.jpeg?s=c84984b6ca09062c2ead9c37c50f836060d4e9ff', 'width': 150, 'height': 150, 'original_size': None}, {'type': 'thumb100', 'url': 'https://images1.vinted.net/t/03_013f2_ooBDU513i1EMriTGuDHtPqMS/100x100/1631286863.jpeg?s=f67ad068c1741ae42777bc8d0007e5ccf47a9432', 'width': 100, 'height': 100, 'original_size': None}, {'type': 'thumb50', 'url': 'https://images1.vinted.net/t/03_013f2_ooBDU513i1EMriTGuDHtPqMS/50x50/1631286863.jpeg?s=84ab94904acb6b6289065e68d22fbb0e1dd77296', 'width': 50, 'height': 50, 'original_size': None}, {'type': 'thumb20', 'url': 'https://images1.vinted.net/t/03_013f2_ooBDU513i1EMriTGuDHtPqMS/20x20/1631286863.jpeg?s=fae7f4b4b5f085bd78ce1e958e762e4f71b99ea6', 'width': 20, 'height': 20, 'original_size': None}]    False   NaN NaN 03_013f2_ooBDU513i1EMriTGuDHtPqMS   1.631287e+09    NaN https://images1.vinted.net/t/03_013f2_ooBDU513i1EMriTGuDHtPqMS/1631286863.jpeg?s=efdae47989a7e6d2d26f3653e81b9b7626f54b42   False   6171417734  1   600 800 #676158 #D1D0CD https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/f800/1644682837.jpeg?s=ea1e18da7463878c5f90bdce4df624b0318db066  True    [{'type': 'thumb70x100', 'url': 'https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/70x100/1644682837.jpeg?s=e93cde47a676eb50b606270a93da7b4b2aaf1041', 'width': 70, 'height': 100, 'original_size': None}, {'type': 'thumb150x210', 'url': 'https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/150x210/1644682837.jpeg?s=e08328ff3b0a1db50d665f6d527ce7296e635397', 'width': 150, 'height': 210, 'original_size': None}, {'type': 'thumb310x430', 'url': 'https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/310x430/1644682837.jpeg?s=b4c88d9260e1963e56506a553c3c0a27e65cde80', 'width': 310, 'height': 430, 'original_size': None}, {'type': 'thumb428x624', 'url': 'https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/f800/1644682837.jpeg?s=ea1e18da7463878c5f90bdce4df624b0318db066', 'width': 321, 'height': 428, 'original_size': True}, {'type': 'thumb624x428', 'url': 'https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/f800/1644682837.jpeg?s=ea1e18da7463878c5f90bdce4df624b0318db066', 'width': 468, 'height': 624, 'original_size': True}, {'type': 'thumb364x428', 'url': 'https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/f800/1644682837.jpeg?s=ea1e18da7463878c5f90bdce4df624b0318db066', 'width': 273, 'height': 364, 'original_size': True}]    03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4   1644682837  NaN False   https://images1.vinted.net/t/03_024cd_GKHt5A8ywMpZWBTsqqNKhxu4/1644682837.jpeg?s=6cc42991c0e36f524f6bb8f3f196969822cad357   False   0.086549    None    NaN
1   613410222   Veste en jean sans manches The Kooples  70.0    None    EUR The Kooples False   https://www.vinted.fr/femmes/vetements/manteaux-and-vestes/vestes/vestes-en-jean/613410222-veste-en-jean-sans-manches-the-kooples   True    27  False   None    None    None    None    70.0    0   M / 38 / 10 catalog_promoted_items  13940373    mina_bc False   https://www.vinted.fr/member/13940373-minabc    13445653.0  426.0   800.0   NaN https://images1.vinted.net/t/02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt/f800/1605243433.jpeg?s=c845cf40b1449b9f43e36417ef2eec44b38195d0  #31abc2 #C1E6ED [{'type': 'thumb310', 'url': 'https://images1.vinted.net/t/02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt/310x310/1605243433.jpeg?s=5194aba6fbdb94486268a7fb3cfbf8a17bbab10a', 'width': 310, 'height': 310, 'original_size': None}, {'type': 'thumb150', 'url': 'https://images1.vinted.net/t/02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt/150x150/1605243433.jpeg?s=5c5f8a1eb23636b54764f4eaa3969825ef0a62a4', 'width': 150, 'height': 150, 'original_size': None}, {'type': 'thumb100', 'url': 'https://images1.vinted.net/t/02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt/100x100/1605243433.jpeg?s=994eee1cfa70bd2c88f0e429c67dc86be4680b20', 'width': 100, 'height': 100, 'original_size': None}, {'type': 'thumb50', 'url': 'https://images1.vinted.net/t/02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt/50x50/1605243433.jpeg?s=f9814710648ae4d9bc05ab7db8b86488d0adc605', 'width': 50, 'height': 50, 'original_size': None}, {'type': 'thumb20', 'url': 'https://images1.vinted.net/t/02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt/20x20/1605243433.jpeg?s=0c6c4df6cf1eced736e8cc5c0fcba29b3cc32cf6', 'width': 20, 'height': 20, 'original_size': None}]    False   NaN NaN 02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt   1.605243e+09    NaN https://images1.vinted.net/t/02_0157e_YwiSFzj2UFbWvtsCMsQCPpFt/1605243433.jpeg?s=5316169fa71ebfc55a62cd75d41ddd9e901a3633   False   2137616878  1   600 800 #534C3D #CBC9C5 https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/f800/1600336103.jpeg?s=c07a055e1789e80536861e7551ea8796395fc90a  True    [{'type': 'thumb70x100', 'url': 'https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/70x100/1600336103.jpeg?s=2fa0b25e36fe58d7c6d8053e07a83de88e77066b', 'width': 70, 'height': 100, 'original_size': None}, {'type': 'thumb150x210', 'url': 'https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/150x210/1600336103.jpeg?s=99153dfdbdf3c558482e64afe8c4a5d9e150e293', 'width': 150, 'height': 210, 'original_size': None}, {'type': 'thumb310x430', 'url': 'https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/310x430/1600336103.jpeg?s=084c5174a85c0309a20b9778c554e5b4b0b301f2', 'width': 310, 'height': 430, 'original_size': None}, {'type': 'thumb428x624', 'url': 'https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/f800/1600336103.jpeg?s=c07a055e1789e80536861e7551ea8796395fc90a', 'width': 321, 'height': 428, 'original_size': True}, {'type': 'thumb624x428', 'url': 'https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/f800/1600336103.jpeg?s=c07a055e1789e80536861e7551ea8796395fc90a', 'width': 468, 'height': 624, 'original_size': True}, {'type': 'thumb364x428', 'url': 'https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/f800/1600336103.jpeg?s=c07a055e1789e80536861e7551ea8796395fc90a', 'width': 273, 'height': 364, 'original_size': True}]    01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf   1600336103  NaN False   https://images1.vinted.net/t/01_01b16_VNVvXUVTv9ZJr7Nn2UXf1Epf/1600336103.jpeg?s=4605661846cbb4776434d027f5a0b88811b5685b   False   0.083722    None    NaN
2   1604718982  Magnifique jeans    5.0 None    EUR Jennyfer    False   https://www.vinted.fr/femmes/vetements/jeans/jeans-troues/1604718982-magnifique-jeans   True    7   False   None    None    None    None    5.0 0   S / 36 / 8  catalog_promoted_items  12761220    lona_emr    False   https://www.vinted.fr/member/12761220-lonaemr   100647431.0 600.0   800.0   NaN https://images1.vinted.net/t/02_0251c_91Kb9aunXW5mjLU4YfLoeTn6/f800/1659093543.jpeg?s=c817748c09515a371757cb55036089d6824d853b  #31abc2 #C1E6ED [{'type': 'thumb310', 'url': 'https://images1.vinted.net/t/02_0251c_91Kb9aunXW5mjLU4YfLoeTn6/310x310/1659093543.jpeg?s=4289e0e4728e7475e358a09887bf3b97fa5ba144', 'width': 310, 'height': 310, 'original_size': None}, {'type': 'thumb150', 'url': 'https://images1.vinted.net/t/02_0251c_91Kb9aunXW5mjLU4YfLoeTn6/150x150/1659093543.jpeg?s=47ab1505a5d3c7e099a4fde2a2f6f6538d82cafc', 'width': 150, 'height': 150, 'original_size': None}, {'type': 'thumb100', 'url': 'https://images1.vinted.net/t/02_0251c_91Kb9aunXW5mjLU4YfLoeTn6/100x100/1659093543.jpeg?s=adcecf2ade3a54f6f565e2077400aa3cc536fbd1', 'width': 100, 'height': 100, 'original_size': None}, {'type': 'thumb50', 'url': 'https://images1.vinted.net/t/02_0251c_91Kb9aunXW5mjLU4YfLoeTn6/50x50/1659093543.jpeg?s=6e8fae4d99a36c6ac93ca8cd54dfb21158f36a01', 'width': 50, 'height': 50, 'original_size': None}, {'type': 'thumb20', 'url': 'https://images1.vinted.net/t/02_0251c_91Kb9aunXW5mjLU4YfLoeTn6/20x20/1659093543.jpeg?s=6845192418fbf0377c8cea70d871ed3b13c81f09', 'width': 20, 'height': 20, 'original_size': None}]    False   NaN NaN 02_0251c_91Kb9aunXW5mjLU4YfLoeTn6   1.659094e+09    NaN https://images1.vinted.net/t/02_0251c_91Kb9aunXW5mjLU4YfLoeTn6/1659093543.jpeg?s=fa6a2da13abac687a96ffe4435b6cb5732709957   False   6072199109  1   600 800 #697683 #D2D6DA https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/f800/1643754263.jpeg?s=3c92ed9a4c71768916cbfed15afbdeaa75035c52  True    [{'type': 'thumb70x100', 'url': 'https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/70x100/1643754263.jpeg?s=dd2bcc2b05ec23f19ba940dd16470c07e8395818', 'width': 70, 'height': 100, 'original_size': None}, {'type': 'thumb150x210', 'url': 'https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/150x210/1643754263.jpeg?s=4f786f1c14cb07bc28419f379b8e222813d581db', 'width': 150, 'height': 210, 'original_size': None}, {'type': 'thumb310x430', 'url': 'https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/310x430/1643754263.jpeg?s=aa62455b22d36f508ee2fde8ccc0495df1b2797d', 'width': 310, 'height': 430, 'original_size': None}, {'type': 'thumb428x624', 'url': 'https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/f800/1643754263.jpeg?s=3c92ed9a4c71768916cbfed15afbdeaa75035c52', 'width': 321, 'height': 428, 'original_size': True}, {'type': 'thumb624x428', 'url': 'https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/f800/1643754263.jpeg?s=3c92ed9a4c71768916cbfed15afbdeaa75035c52', 'width': 468, 'height': 624, 'original_size': True}, {'type': 'thumb364x428', 'url': 'https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/f800/1643754263.jpeg?s=3c92ed9a4c71768916cbfed15afbdeaa75035c52', 'width': 273, 'height': 364, 'original_size': True}]    01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW   1643754263  NaN False   https://images1.vinted.net/t/01_02504_PVFwMYdQLJ6q5fBKkSm3R8cW/1643754263.jpeg?s=4458bc459aa584e01c63704332da5d6cfff33c7b   False   0.070368    None    NaN
[...]
Answered By: platipus_on_fire

To click on the element accept all you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following locator strategies:

  • Using CSS_SELECTOR:

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler>font>font"))).click()

  • Using XPATH:

    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@id='onetrust-accept-btn-handler']//font[text()='accept all']"))).click()

  • Note: You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Answered By: undetected Selenium