How to get a list of all links from a dynamic web page?
Question:
I’m trying to scrape this page:
https://workspace.google.com/marketplace/search/word
I tried PhantomJS+BeautifulSoup (failed), then Playwright to scrape the whole content of the page but I can’t see the links to the extensions. Do they get generated only when the cursor hovers over them? Is there a way to get them?
Here’s the Playwright code I used:
from playwright.sync_api import Playwright, sync_playwright, expect
def run(playwright: Playwright) -> None:
browser = playwright.chromium.launch(headless=True)
context = browser.new_context()
page = context.new_page()
page.goto("https://workspace.google.com/marketplace/search/word")
print(page.content())
# ---------------------
context.close()
browser.close()
with sync_playwright() as playwright:
run(playwright)
Answers:
You can try the next example playwright with bs4.
Code:
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
data = []
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
context = browser.new_context(viewport={"width": 1920, "height": 1080})
page = context.new_page()
page.goto('https://workspace.google.com/marketplace/search/word')
page.wait_for_timeout(4000)
soup = BeautifulSoup(page.content(), 'lxml')
for card in soup.select('a.RwHvCd'):
link = 'https://workspace.google.com' + card.get('href').replace('./', '/')
print(link)
Output:
https://workspace.google.com/marketplace/app/word_cloud_generator/360115564222
https://workspace.google.com/marketplace/app/pdf_to_word_doc_converter/363901784508
https://workspace.google.com/marketplace/app/word_cloud_generator/1066049374643
https://workspace.google.com/marketplace/app/word_cloud_for_docs/58662699010
https://workspace.google.com/marketplace/app/drive_word_cloud/401630517929
https://workspace.google.com/marketplace/app/word_search_game_with_google_drive/766902391959
https://workspace.google.com/marketplace/app/word_cloud_generator/251884431535
https://workspace.google.com/marketplace/app/word_counter_max_for_google_docs/364683295233
https://workspace.google.com/marketplace/app/online_word_cloud/275091946896
https://workspace.google.com/marketplace/app/bjorns_word_clouds/423122543905
https://workspace.google.com/marketplace/app/glue_word_finder/295291845080
https://workspace.google.com/marketplace/app/bjorns_3d_word_clouds/845381453179
https://workspace.google.com/marketplace/app/bjorns_word_clouds/775726653147
https://workspace.google.com/marketplace/app/bjorns_wordzones/635623875397
https://workspace.google.com/marketplace/app/dochub_pdf_sign_and_edit/1179802238
https://workspace.google.com/marketplace/app/doctopus/979668934766
https://workspace.google.com/marketplace/app/yet_another_mail_merge_mail_merge_for_gm/52669349336
https://workspace.google.com/marketplace/app/form_publisher/827172627657
https://workspace.google.com/marketplace/app/tex_equation_editor/197218123452
https://workspace.google.com/marketplace/app/clozeit/679357385347
https://workspace.google.com/marketplace/app/spellright/225027406829
https://workspace.google.com/marketplace/app/vocabularycom/903174061747
https://workspace.google.com/marketplace/app/mail_merge/218858140171
https://workspace.google.com/marketplace/app/create_print_labels_label_maker_for_aver/585829216542
https://workspace.google.com/marketplace/app/mailmeteor_mail_merge_for_gmail/1008170693301
https://workspace.google.com/marketplace/app/adobe_acrobat_%E2%80%93_pdf_and_esignature_tools/80763634447
https://workspace.google.com/marketplace/app/avery_label_merge/687713670524
https://workspace.google.com/marketplace/app/smallpdf_%E2%80%93_edit_compress_and_convert_pdf/3122370240
https://workspace.google.com/marketplace/app/mindmeister/119415359491
https://workspace.google.com/marketplace/app/awesome_table/56088344336
https://workspace.google.com/marketplace/app/text_editor/591525900269
https://workspace.google.com/marketplace/app/zoho_writer/457660898219
https://workspace.google.com/marketplace/app/shared_contacts_share_google_workspace_c/1033860418004
https://workspace.google.com/marketplace/app/power_tools/1058867473888
https://workspace.google.com/marketplace/app/grammar_and_spell_checker_languagetool/805250893316
https://workspace.google.com/marketplace/app/foxy_labels_label_maker_for_avery_co/534005213458
https://workspace.google.com/marketplace/app/slido_for_google_slides/240609050747
https://workspace.google.com/marketplace/app/document_studio/429444628321
https://workspace.google.com/marketplace/app/aodocs/993707107224
https://workspace.google.com/marketplace/app/pdf_converter/748079751468
https://workspace.google.com/marketplace/app/virtru_data_protection_for_google_worksp/197076597243
https://workspace.google.com/marketplace/app/doc_tools/871708524981
https://workspace.google.com/marketplace/app/outwrite_previously_gradeproof/1017605304002
https://workspace.google.com/marketplace/app/wordwall/350238723064
https://workspace.google.com/marketplace/app/slides_toolbox/994552357682
https://workspace.google.com/marketplace/app/pdffiller_for_google_drive/554084336035
https://workspace.google.com/marketplace/app/fun_text/25652727687
https://workspace.google.com/marketplace/app/business_hangouts/67717961732
https://workspace.google.com/marketplace/app/quillbot_paraphrasing_and_summarizing_to/305988918037
https://workspace.google.com/marketplace/app/form_director/824044380012
https://workspace.google.com/marketplace/app/ultradox/172644952635
https://workspace.google.com/marketplace/app/doc_docx_ppt_file_viewer/825904744868
https://workspace.google.com/marketplace/app/qr_code_generator/283497561682
https://workspace.google.com/marketplace/app/advanced_summary_by_awesome_table/1091159616075
https://workspace.google.com/marketplace/app/paperpile/894076725911
https://workspace.google.com/marketplace/app/mail_merge_for_google_docs/799434348782
https://workspace.google.com/marketplace/app/publigo_document_merge/677216675116
https://workspace.google.com/marketplace/app/bee_templates_for_gmail/1023465560860
https://workspace.google.com/marketplace/app/qr_code_barcode_generator/119437009854
https://workspace.google.com/marketplace/app/cloud_epub_reader_with_drive/155955896601
https://workspace.google.com/marketplace/app/onelook_thesaurus/372652075936
https://workspace.google.com/marketplace/app/couplerio/532272210531
https://workspace.google.com/marketplace/app/dropbox_sign_fka_as_hellosign_for_google/1094820495693
https://workspace.google.com/marketplace/app/scanit_to_office/54058147825
https://workspace.google.com/marketplace/app/syncwith_any_api/449644239211
https://workspace.google.com/marketplace/app/cacoo_diagramming_realtime_collaboration/136288733433
https://workspace.google.com/marketplace/app/pixton/433178159359
https://workspace.google.com/marketplace/app/verityspell/250352709351
https://workspace.google.com/marketplace/app/form_builder_plus/235019714744
https://workspace.google.com/marketplace/app/conceptboard/1073597505098
https://workspace.google.com/marketplace/app/password_protect_for_google_documents/1082163372793
https://workspace.google.com/marketplace/app/sapling_grammar_checker_and_writing_assi/1078289388837
https://workspace.google.com/marketplace/app/bad_translator/306615442162
https://workspace.google.com/marketplace/app/dialpad_meetings_for_google_calendar/255217867751
https://workspace.google.com/marketplace/app/math_equations_editor_latex_editor/1008813237979
https://workspace.google.com/marketplace/app/writing_mentor/45773170162
https://workspace.google.com/marketplace/app/odt_odp_ods_viewer/142217579112
https://workspace.google.com/marketplace/app/hellosign/434481440707
https://workspace.google.com/marketplace/app/charts_graphs_visualizations_by_chartexp/643809879384
https://workspace.google.com/marketplace/app/file_converter_by_onlineconvertcom/618319434088
https://workspace.google.com/marketplace/app/synap_document_viewer/451168936263
https://workspace.google.com/marketplace/app/ebook_viewer_and_converter/710512554202
https://workspace.google.com/marketplace/app/wordpresscom_for_google_docs/460536350236
https://workspace.google.com/marketplace/app/wordzen_we_write_your_emails/235420263280
https://workspace.google.com/marketplace/app/cloud_converter/214567865127
https://workspace.google.com/marketplace/app/debate_template/712515658695
https://workspace.google.com/marketplace/app/html_editor_for_drive/67823851872
https://workspace.google.com/marketplace/app/search_navigate/702369391016
https://workspace.google.com/marketplace/app/cross_reference/269114033347
https://workspace.google.com/marketplace/app/text_cleaner/103106847686
https://workspace.google.com/marketplace/app/libreoffice_offidocs/540150451923
https://workspace.google.com/marketplace/app/power_thesaurus_for_google_docs/345026180982
https://workspace.google.com/marketplace/app/youmescript/59937783758
https://workspace.google.com/marketplace/app/virtual_facilitation/302606386204
https://workspace.google.com/marketplace/app/helperbird/844716805038
https://workspace.google.com/marketplace/app/emerge_mail_merge/854160332054
https://workspace.google.com/marketplace/app/latex_editor_for_google_drive/614123355972
https://workspace.google.com/marketplace/app/writers_highlighter/1033878279784
https://workspace.google.com/marketplace/app/link_style/430151409905
https://workspace.google.com/marketplace/app/writing_habit/908529024908
I’m trying to scrape this page:
https://workspace.google.com/marketplace/search/word
I tried PhantomJS+BeautifulSoup (failed), then Playwright to scrape the whole content of the page but I can’t see the links to the extensions. Do they get generated only when the cursor hovers over them? Is there a way to get them?
Here’s the Playwright code I used:
from playwright.sync_api import Playwright, sync_playwright, expect
def run(playwright: Playwright) -> None:
browser = playwright.chromium.launch(headless=True)
context = browser.new_context()
page = context.new_page()
page.goto("https://workspace.google.com/marketplace/search/word")
print(page.content())
# ---------------------
context.close()
browser.close()
with sync_playwright() as playwright:
run(playwright)
You can try the next example playwright with bs4.
Code:
from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup
data = []
with sync_playwright() as p:
browser = p.chromium.launch(headless=False)
context = browser.new_context(viewport={"width": 1920, "height": 1080})
page = context.new_page()
page.goto('https://workspace.google.com/marketplace/search/word')
page.wait_for_timeout(4000)
soup = BeautifulSoup(page.content(), 'lxml')
for card in soup.select('a.RwHvCd'):
link = 'https://workspace.google.com' + card.get('href').replace('./', '/')
print(link)
Output:
https://workspace.google.com/marketplace/app/word_cloud_generator/360115564222
https://workspace.google.com/marketplace/app/pdf_to_word_doc_converter/363901784508
https://workspace.google.com/marketplace/app/word_cloud_generator/1066049374643
https://workspace.google.com/marketplace/app/word_cloud_for_docs/58662699010
https://workspace.google.com/marketplace/app/drive_word_cloud/401630517929
https://workspace.google.com/marketplace/app/word_search_game_with_google_drive/766902391959
https://workspace.google.com/marketplace/app/word_cloud_generator/251884431535
https://workspace.google.com/marketplace/app/word_counter_max_for_google_docs/364683295233
https://workspace.google.com/marketplace/app/online_word_cloud/275091946896
https://workspace.google.com/marketplace/app/bjorns_word_clouds/423122543905
https://workspace.google.com/marketplace/app/glue_word_finder/295291845080
https://workspace.google.com/marketplace/app/bjorns_3d_word_clouds/845381453179
https://workspace.google.com/marketplace/app/bjorns_word_clouds/775726653147
https://workspace.google.com/marketplace/app/bjorns_wordzones/635623875397
https://workspace.google.com/marketplace/app/dochub_pdf_sign_and_edit/1179802238
https://workspace.google.com/marketplace/app/doctopus/979668934766
https://workspace.google.com/marketplace/app/yet_another_mail_merge_mail_merge_for_gm/52669349336
https://workspace.google.com/marketplace/app/form_publisher/827172627657
https://workspace.google.com/marketplace/app/tex_equation_editor/197218123452
https://workspace.google.com/marketplace/app/clozeit/679357385347
https://workspace.google.com/marketplace/app/spellright/225027406829
https://workspace.google.com/marketplace/app/vocabularycom/903174061747
https://workspace.google.com/marketplace/app/mail_merge/218858140171
https://workspace.google.com/marketplace/app/create_print_labels_label_maker_for_aver/585829216542
https://workspace.google.com/marketplace/app/mailmeteor_mail_merge_for_gmail/1008170693301
https://workspace.google.com/marketplace/app/adobe_acrobat_%E2%80%93_pdf_and_esignature_tools/80763634447
https://workspace.google.com/marketplace/app/avery_label_merge/687713670524
https://workspace.google.com/marketplace/app/smallpdf_%E2%80%93_edit_compress_and_convert_pdf/3122370240
https://workspace.google.com/marketplace/app/mindmeister/119415359491
https://workspace.google.com/marketplace/app/awesome_table/56088344336
https://workspace.google.com/marketplace/app/text_editor/591525900269
https://workspace.google.com/marketplace/app/zoho_writer/457660898219
https://workspace.google.com/marketplace/app/shared_contacts_share_google_workspace_c/1033860418004
https://workspace.google.com/marketplace/app/power_tools/1058867473888
https://workspace.google.com/marketplace/app/grammar_and_spell_checker_languagetool/805250893316
https://workspace.google.com/marketplace/app/foxy_labels_label_maker_for_avery_co/534005213458
https://workspace.google.com/marketplace/app/slido_for_google_slides/240609050747
https://workspace.google.com/marketplace/app/document_studio/429444628321
https://workspace.google.com/marketplace/app/aodocs/993707107224
https://workspace.google.com/marketplace/app/pdf_converter/748079751468
https://workspace.google.com/marketplace/app/virtru_data_protection_for_google_worksp/197076597243
https://workspace.google.com/marketplace/app/doc_tools/871708524981
https://workspace.google.com/marketplace/app/outwrite_previously_gradeproof/1017605304002
https://workspace.google.com/marketplace/app/wordwall/350238723064
https://workspace.google.com/marketplace/app/slides_toolbox/994552357682
https://workspace.google.com/marketplace/app/pdffiller_for_google_drive/554084336035
https://workspace.google.com/marketplace/app/fun_text/25652727687
https://workspace.google.com/marketplace/app/business_hangouts/67717961732
https://workspace.google.com/marketplace/app/quillbot_paraphrasing_and_summarizing_to/305988918037
https://workspace.google.com/marketplace/app/form_director/824044380012
https://workspace.google.com/marketplace/app/ultradox/172644952635
https://workspace.google.com/marketplace/app/doc_docx_ppt_file_viewer/825904744868
https://workspace.google.com/marketplace/app/qr_code_generator/283497561682
https://workspace.google.com/marketplace/app/advanced_summary_by_awesome_table/1091159616075
https://workspace.google.com/marketplace/app/paperpile/894076725911
https://workspace.google.com/marketplace/app/mail_merge_for_google_docs/799434348782
https://workspace.google.com/marketplace/app/publigo_document_merge/677216675116
https://workspace.google.com/marketplace/app/bee_templates_for_gmail/1023465560860
https://workspace.google.com/marketplace/app/qr_code_barcode_generator/119437009854
https://workspace.google.com/marketplace/app/cloud_epub_reader_with_drive/155955896601
https://workspace.google.com/marketplace/app/onelook_thesaurus/372652075936
https://workspace.google.com/marketplace/app/couplerio/532272210531
https://workspace.google.com/marketplace/app/dropbox_sign_fka_as_hellosign_for_google/1094820495693
https://workspace.google.com/marketplace/app/scanit_to_office/54058147825
https://workspace.google.com/marketplace/app/syncwith_any_api/449644239211
https://workspace.google.com/marketplace/app/cacoo_diagramming_realtime_collaboration/136288733433
https://workspace.google.com/marketplace/app/pixton/433178159359
https://workspace.google.com/marketplace/app/verityspell/250352709351
https://workspace.google.com/marketplace/app/form_builder_plus/235019714744
https://workspace.google.com/marketplace/app/conceptboard/1073597505098
https://workspace.google.com/marketplace/app/password_protect_for_google_documents/1082163372793
https://workspace.google.com/marketplace/app/sapling_grammar_checker_and_writing_assi/1078289388837
https://workspace.google.com/marketplace/app/bad_translator/306615442162
https://workspace.google.com/marketplace/app/dialpad_meetings_for_google_calendar/255217867751
https://workspace.google.com/marketplace/app/math_equations_editor_latex_editor/1008813237979
https://workspace.google.com/marketplace/app/writing_mentor/45773170162
https://workspace.google.com/marketplace/app/odt_odp_ods_viewer/142217579112
https://workspace.google.com/marketplace/app/hellosign/434481440707
https://workspace.google.com/marketplace/app/charts_graphs_visualizations_by_chartexp/643809879384
https://workspace.google.com/marketplace/app/file_converter_by_onlineconvertcom/618319434088
https://workspace.google.com/marketplace/app/synap_document_viewer/451168936263
https://workspace.google.com/marketplace/app/ebook_viewer_and_converter/710512554202
https://workspace.google.com/marketplace/app/wordpresscom_for_google_docs/460536350236
https://workspace.google.com/marketplace/app/wordzen_we_write_your_emails/235420263280
https://workspace.google.com/marketplace/app/cloud_converter/214567865127
https://workspace.google.com/marketplace/app/debate_template/712515658695
https://workspace.google.com/marketplace/app/html_editor_for_drive/67823851872
https://workspace.google.com/marketplace/app/search_navigate/702369391016
https://workspace.google.com/marketplace/app/cross_reference/269114033347
https://workspace.google.com/marketplace/app/text_cleaner/103106847686
https://workspace.google.com/marketplace/app/libreoffice_offidocs/540150451923
https://workspace.google.com/marketplace/app/power_thesaurus_for_google_docs/345026180982
https://workspace.google.com/marketplace/app/youmescript/59937783758
https://workspace.google.com/marketplace/app/virtual_facilitation/302606386204
https://workspace.google.com/marketplace/app/helperbird/844716805038
https://workspace.google.com/marketplace/app/emerge_mail_merge/854160332054
https://workspace.google.com/marketplace/app/latex_editor_for_google_drive/614123355972
https://workspace.google.com/marketplace/app/writers_highlighter/1033878279784
https://workspace.google.com/marketplace/app/link_style/430151409905
https://workspace.google.com/marketplace/app/writing_habit/908529024908