BeautifulSoup find_all not working. Text held within window._STORE_DATA JSON Python

Question:

I’m trying to scrape jobs from this University website because their website is difficult to read IMO. When I try to use the find_all function, it doesn’t return anything. However, on the website I’ve noticed that all of the information is contained in one long JSON within this window._STORE_DATA value. I’m new to web scraping. Can someone explain why this wouldn’t work? I’m trying to scrape "clinical lab scientist" from the JSON behind window.STORE_DATA. Thanks

The html was shortened due to character limits. Here is a link to the website. Source

import requests
from bs4 import BeautifulSoup
import json

url = "https://employment.ucsd.edu/jobs?page_size=500&page_number=1&keyword=clinical%20lab%20scientist&location_city=Remote&location_city=San%20Diego&location_city=Encinitas&location_city=Murrieta&location_city=La%20Jolla&location_city=Not%20Specified&location_city=Vista&sort_by=score&sort_order=DESC"
response = requests.get(url).text
doc = BeautifulSoup(response, "html.parser")
tag = doc.find_all("clinical lab scientist")
print(tag)

I also tried to add case sensitive "Clinical Lab Scientist" which did not work also.

import requests
from bs4 import BeautifulSoup
import json

url = "https://employment.ucsd.edu/jobs?page_size=500&page_number=1&keyword=clinical%20lab%20scientist&location_city=Remote&location_city=San%20Diego&location_city=Encinitas&location_city=Murrieta&location_city=La%20Jolla&location_city=Not%20Specified&location_city=Vista&sort_by=score&sort_order=DESC"
response = requests.get(url).text
doc = BeautifulSoup(response, "html.parser")
tag = doc.find_all("Clinical Lab Scientist")
print(tag)

This is what it returned

Connected to pydev debugger (build 221.5591.52)
[]

Process finished with exit code 0

HTML FROM WEBSITE (SHORTENED)

<script>
  window._STORE_DATA = {"apply":{"firstName":"","lastName":"","emailAddress":"","phoneNumber":"","applicationSubmissionFailureMessage":"","country":"US","usaAuthorization":"1","zipCode":"","eeo_race":"0","birth_date":"0/0/0","eeo_gender":"0","eeo_veteran_status":"0","eeo_disability_status":"0","resume":null,"resumeData":null,"resumeFileType":null,"coverLetter":"","uncompletedForms":["Your Information"],"completedForms":[],"prescreenQuestions":[],"prescreenAnswers":{},"uploadingResume":false,"submittingApplication":false,"applicationSubmissionFailed":false,"applicationSubmissionSucceeded":false,"errors":{},"hydrated":true},"builder":{"bottom":[],"customCategoriesMapping":[],"collapsed":false,"deletedPages":{},"device":"phone","dirty":{},"editing":"/","editingTab":"content","hydrated":true,"inactive_pages":[],"jobCategoriesMapping":[],"loading":false,"ordered_page_ids":[],"orientation":"vertical","pages":{},"saving":"no-action","theme":{},"theme_json":{},"top":[],"viewOnDevice":false},"imageUpload":{"fileType":"image/jpeg","loading":false,"quality":80,"visible":false},"location_search":{"locations":[],"companyID":157272,"hydrated":true},"company_follow":{"alert":null,"carrotUp":false,"category_names":[],"companyName":"UC San Diego ","custom_categories":[],"disableFollow":false,"isFollowingCompany":false,"showModal":false,"EmailAddress":"","FrequencyId":0,"IsCustomCategory":false,"JobAlertCategories":[],"CompanyId":157272,"submittingJobAlert":false,"updatingJobAlert":false,"unsubscribingJobAlert":false,"showUnsubscribePrompt":false,"sendingUnsubscribeEmail":false,"is_monster_client":false,"hydrated":true},"contact_us":{"firstName":"","lastName":"","middleName":"","companyName":"","jobTitle":"","phoneNumber":"","emailAddress":"","helpComments":"","contactTime":"","additionalInfo":"","formIsProcessing":false,"formSuccess":false},"debugging":{},"job_search":{"clearAll":0,"radius":25,"page_size":100,"locationQuery":"","expanded_job":"","expanded_filter":"","search_attempt":0,"total_jobs":504,"jobs":[{"distance":0,"distanceUnits":null,"id":"8f0cba5e-4fe1-11ed-9efa-9c748d5b923d","guid":"8f0cba5e-4fe1-11ed-9efa-9c748d5b923d","isGuidGenerated":false,"active":0,"company_id":157272,"company_name":"UC San Diego","company_type":"E","company_unique_name":"uc-san-diego-demo272","jobseeker_blacklist":[],"applied_jobseekers":[],"company_image_url":null,"job_url":"/clinical-lab-scientist-specialist-119559/job/20822209","custom_categories":[{"id":0,"url":null,"name":"Medical/Healthcare - Health","keywords":null},{"id":0,"url":null,"name":"Part Time - Health","keywords":null},{"id":0,"url":null,"name":"Laboratory Services - Health","keywords":null}],"custom_fields":[],"CustomCategoriesString":["Medical/Healthcare - Health","Part Time - Health","Laboratory Services - Health"],"to_custom_site":false,"job_details":{"id":"8f0cba5e-4fe1-11ed-9efa-9c748d5b923d","job_title_url":"/clinical-lab-scientist-specialist-119559/job/20822209","title":"Clinical Lab Scientist / Specialist - 119559","job_cource":1,"company_name":"UC San Diego","locations":[{"id":22518764,"address_id":482229,"external_id":"","metro":["california","sandiego"],"street":"200 W Arbor Dr","street2":null,"city":"San Diego","state_name":"California","state_abbr":"CA","zip":92103,"latitude":32.749777,"longitude":-117.168668,"country":"United States of America","country_abbr":"US","is_remote":false,"has_geocode_override":false,"city_state_abbr":"San Diego, CA","location_id":[],"miles":0,"city_state":"San Diego, California","latitude_longitude":"32.749777000000000,-117.168668000000000","name":"Hillcrest Medical Center","id_value":null,"area_id":0,"postalcode":"","is_main_office":false,"street_city_state":"200 W Arbor Dr, San Diego, CA","is_active":false,"sublocality":""}],"salary_amount":"0.0","job_status":null,"job_type":1,"expiration_date":"2023-01-11T00:00:00","content_short":" Payroll Title: CLIN LAB SCI SPEC Department: Blood Bank Salary Range Commensurate with Experience Worksite: Hillcrest Medical Center Appointment Type: Career Appointment Percent: 50% Union: HX Contract Total Openings: 1 Work Schedule: Nights, 8 hour shif..."},"start_date":"2022-12-14T00:00:00","original_posting_date":"0001-01-01T00:00:00","updated_date":"2022-12-16T10:09:30.833","expiration_date":"2023-01-11T00:00:00","headline":"Clinical Lab Scientist / Specialist - 119559","headline_titlecase":"Clinical Lab Scientist / Specialist - 119559","content":null,"content_short":" Payroll Title: CLIN LAB SCI SPEC Department: Blood Bank Salary Range Commensurate with Experience Worksite: Hillcrest Medical Center Appointment Type: Career Appointment Percent: 50% Union: HX Contract Total Openings: 1 Work Schedule: Nights, 8 hour shif...","tags":null,"skills":null,"important_notes":null,"reference":null,"reference_string":"119559","industry_url":null,"industry_name":"Education","pre_screen_id":0,"job_type":1,"job_type_name":"Standard","compensation_amount":"0.0","minimum_compensation":0,"compensation_comment":"","custom_apply_text":null,"custom_apply_link":"Apply Online","custom_apply_url":"https://jobs.ucsd.edu/Apply/BeforeApply.aspx?jid=119559&sid=99416.0","employment_types":[0],"EmploymentTypesCSV":"Not Specified","recruiters":[{"level":0,"company":null,"recruiter_id":237525,"title":"Manager","telephone":"(999) 999-9999","email_display":"Hide","email_format":"HTML","email_attachment":"Yes","crm_version":null,"umbrella":false,"password_expired":false,"email":"[email protected]","location":{"id":0,"address_id":0,"external_id":null,"metro":[],"street":null,"street2":null,"city":null,"state_name":null,"state_abbr":null,"zip":0,"latitude":0,"longitude":0,"country":null,"country_abbr":null,"is_remote":false,"has_geocode_override":false,"city_state_abbr":"","location_id":[],"miles":0,"city_state":"","latitude_longitude":"0,0","name":null,"id_value":null,"area_id":0,"postalcode":null,"is_main_office":false,"street_city_state":", , ","is_active":false,"sublocality":null},"session_guid":"00000000-0000-0000-0000-000000000000","is_mfa_enabled":false,"verification_hash":null,"retry_verification":false,"visitor_id":0,"key":null,"status":0,"guid":null,"first_name":"Scrape","last_name":"User","phones":[],"birth_date":"1900-01-01T00:00:00","photo":{"versions":{"version_125x125":"https://dot1p08cqyhea.cloudfront.net/default_pictures/125x125.png","version_80x80":"https://dot1p08cqyhea.cloudfront.net/default_pictures/80x80.png","version_50x50":"https://dot1p08cqyhea.cloudfront.net/default_pictures/50x50.png","version_20x20":"https://dot1p08cqyhea.cloudfront.net/default_pictures/20x20.png","version_30x30":"https://dot1p08cqyhea.cloudfront.net/default_pictures/30x30.png"},"has_photo":false,"original_url":"https://dot1p08cqyhea.cloudfront.net/pictures//","preview_url":"https://dot1p08cqyhea.cloudfront.net/pictures//"},"guest_id":null,"full_name":"Scrape User","login_session":{"object_type":0,"recruiter_id":0,"jobseeker_id":0,"date_stamp":"0001-01-01T00:00:00","date_stamp_string":"0001-01-01T00:00:00","hash_key":null,"cache_statistics":null},"cache_statistics":null}],"locations":[{"id":22518764,"address_id":482229,"external_id":"","metro":["california","sandiego"],"street":"200 W Arbor Dr","street2":null,"city":"San Diego","state_name":"California","state_abbr":"CA","zip":92103,"latitude":32.749777,"longitude":-117.168668,"country":"United States of America","country_abbr":"US","is_remote":false,"has_geocode_override":false,"city_state_abbr":"San Diego, CA","location_id":[],"miles":0,"city_state":"San Diego, California","latitude_longitude":"32.749777000000000,-117.168668000000000","name":"Hillcrest Medical Center","id_value":null,"area_id":0,"postalcode":"","is_main_office":false,"street_city_state":"200 W Arbor Dr, San Diego, CA","is_active":false,"sublocality":""}],"lat_long":"32.749777000000000,-117.168668000000000","geo":["32.749777000000000,-117.168668000000000"],"areaid":null,"questions":[],"categories":[{"id":21,"url":"science-biotech-research","name":"Science/Biotech/Research","keywords":null},{"id":507,"url":"healthcare-lab-hematology-pathology","name":"Healthcare - Lab/Hematology/Pathology","keywords":null},{"id":534,"url":"healthcare-all-categories","name":"Healthcare -  ALL CATEGORIES","keywords":null}],"logo_img_url":"/company/logos/logo157272.gif","job_duration":28,"industry_id":16,"job_source":1,"job_level":4,"job_source_id":"20822209","job_entry_level":false,"job_indexed_time":"2022-12-16T17:10:17.601","timestamp":"2022-12-16T10:10:17.567","score":null,"company_logo":{"company_id":0,"MaxFileSize":5242880,"company_photo_versions":{},"photo_versions":[],"has_image":false,"original_file_name":null,"original_file_path":null},"utc_expiration_date":"2023-01-11T00:00:00","utc_start_date":"2022-12-14T00:00:00","external_reference_id":null,"brand":null,"widget_id":null,"job_shifts":null,"cem_account_name":null,"dynamic_data":{},"external_posting_date":null,"external_location_id":null,"external_recruiter_name":null,"cost_center":null,"remote_type":null,"cache_statistics":null},{"distance":0,"distanceUnits":null,"id":"802b7f57-69f1-11ed-97bf-7009ca23e248","guid":"802b7f57-69f1-11ed-97bf-7009ca23e248","isGuidGenerated":false,"active":0,"company_id":157272,"company_name":"UC San Diego","company_type":"E","company_unique_name":"uc-san-diego-demo272","jobseeker_blacklist":[],"applied_jobseekers":[],"company_image_url":null,"job_url":"/clinical-lab-scientist-specialist-120139/job/21460814","custom_categories":[{"id":0,"url":null,"name":"Medical/Healthcare - Health","keywords":null},{"id":0,"url":null,"name":"Part Time - Health","keywords":null},{"id":0,"url":null,"name":"Laboratory Services - Health","keywords":null}],"custom_fields":[],"CustomCategoriesString":["Medical/Healthcare - Health","Part Time - Health","Laboratory Services - Health"],"to_custom_site":false,"job_details":{"id":"802b7f57-69f1-11ed-97bf-7009ca23e248","job_title_url":"/clinical-lab-scientist-specialist-120139/job/21460814","title":"Clinical Lab Scientist / Specialist - 120139","job_cource":1,"company_name":"UC San Diego","locations":[{"id":23167860,"address_id":482223,"external_id":"","metro":["california","sandiego"],"street":"9500 Gilman Drive","street2":null,"city":"La Jolla","state_name":"California","state_abbr":"CA","zip":92093,"latitude":32.877058,"longitude":-117.230578,"country":"United States of America","country_abbr":"US","is_remote":false,"has_geocode_override":false,"city_state_abbr":"La Jolla, CA","location_id":[],"miles":0,"city_state":"La Jolla, California","latitude_longitude":"32.877058000000000,-117.230578000000000","name":"Campus","id_value":null,"area_id":0,"postalcode":"","is_main_office":false,"street_city_state":"9500 Gilman Drive, La Jolla, CA","is_active":false,"sublocality":""}],"salary_amount":"0.0","job_status":null,"job_type":1,"expiration_date":"2023-01-16T00:00:00","content_short":" Payroll Title: CLIN LAB SCI SPEC Department: Clinical Lab Hiring Salary Range $48.61 - $64.02 / Hour Worksite: La Jolla Appointment Type: Career Appointment Percent: 60% Union: HX Contract Total Openings: 1 Work Schedule: Variable, 8 hour shifts As a fed..."},"start_date":"2022-12-19T00:00:00","original_posting_date":"0001-01-01T00:00:00","updated_date":"2022-12-25T01:09:08.36","expiration_date":"2023-01-16T00:00:00","headline":"Clinical Lab Scientist / Specialist - 120139","headline_titlecase":"Clinical Lab Scientist / Specialist - 120139","content":null,"content_short":" Payroll Title: CLIN LAB SCI SPEC Department: Clinical Lab Hiring Salary Range $48.61 - $64.02 / Hour Worksite: La Jolla Appointment Type: Career Appointment Percent: 60% Union: HX Contract Total Openings: 1 Work Schedule: Variable, 8 hour shifts As a fed...","tags":null,"skills":null,"important_notes":null,"reference":null,"reference_string":"120139","industry_url":null,"industry_name":"Education","pre_screen_id":0,"job_type":1,"job_type_name":"Standard","compensation_amount":"0.0","minimum_compensation":0,"compensation_comment":"","custom_apply_text":null,"custom_apply_link":"Apply Online","custom_apply_url":"https://jobs.ucsd.edu/Apply/BeforeApply.aspx?jid=120139&sid=99974.0","employment_types":[0],"EmploymentTypesCSV":"Not Specified","recruiters":[{"level":0,"company":null,"recruiter_id":237525,"title":"Manager","telephone":"(999) 999-9999","email_display":"Hide","email_format":"HTML","email_attachment":"Yes","crm_version":null,"umbrella":false,"password_expired":false,"email":"[email protected]","location":{"id":0,"address_id":0,"external_id":null,"metro":[],"street":null,"street2":null,"city":null,"state_name":null,"state_abbr":null,"zip":0,"latitude":0,"longitude":0,"country":null,"country_abbr":null,"is_remote":false,"has_geocode_override":false,"city_state_abbr":"","location_id":[],"miles":0,"city_state":"","latitude_longitude":"0,0","name":null,"id_value":null,"area_id":0,"postalcode":null,"is_main_office":false,"street_city_state":", , ","is_active":false,"sublocality":null},"session_guid":"00000000-0000-0000-0000-000000000000","is_mfa_enabled":false,"verification_hash":null,"retry_verification":false,"visitor_id":0,"key":null,"status":0,"guid":null,"first_name":"Scrape","last_name":"User","phones":[],"birth_date":"1900-01-01T00:00:00","photo":{"versions":{"version_125x125":"https://dot1p08cqyhea.cloudfront.net/default_pictures/125x125.png","version_80x80":"https://dot1p08cqyhea.cloudfront.net/default_pictures/80x80.png","version_50x50":"https://dot1p08cqyhea.cloudfront.net/default_pictures/50x50.png","version_20x20":"https://dot1p08cqyhea.cloudfront.net/default_pictures/20x20.png","version_30x30":"https://dot1p08cqyhea.cloudfront.net/default_pictures/30x30.png"},"has_photo":false,"original_url":"https://dot1p08cqyhea.cloudfront.net/pictures//","preview_url":"https://dot1p08cqyhea.cloudfront.net/pictures//"},"guest_id":null,"full_name":"Scrape User","login_session":{"object_type":0,"recruiter_id":0,"jobseeker_id":0,"date_stamp":"0001-01-01T00:00:00","date_stamp_string":"0001-01-01T00:00:00","hash_key":null,"cache_statistics":null},"cache_statistics":null}],"locations":[{"id":23167860,"address_id":482223,"external_id":"","metro":["california","sandiego"],"street":"9500 Gilman Drive","street2":null,"city":"La Jolla","state_name":"California","state_abbr":"CA","zip":92093,"latitude":32.877058,"longitude":-117.230578,"country":"United States of America","country_abbr":"US","is_remote":false,"has_geocode_override":false,"city_state_abbr":"La Jolla, CA","location_id":[],"miles":0,"city_state":"La Jolla, California","latitude_longitude":"32.877058000000000,-117.230578000000000","name":"Campus","id_value":null,"area_id":0,"postalcode":"","is_main_office":false,"street_city_state":"9500 Gilman Drive, La Jolla, CA","is_active":false,"sublocality":""}],"lat_long":"32.877058000000000,-117.230578000000000","geo":["32.877058000000000,-117.230578000000000"],"areaid":null,"questions":[],"categories":[{"id":21,"url":"science-biotech-research","name":"Science/Biotech/Research","keywords":null},{"id":507,"url":"healthcare-lab-hematology-pathology","name":"Healthcare - Lab/Hematology/Pathology","keywords":null},{"id":534,"url":"healthcare-all-categories","name":"Healthcare -  ALL CATEGORIES","keywords":null}],"logo_img_url":"/company/logos/logo157272.gif","job_duration":28,"industry_id":16,"job_source":1,"job_level":4,"job_source_id":"21460814","job_entry_level":false,"job_indexed_time":"2022-12-25T08:10:06.315","timestamp":"2022-12-25T01:10:06.268","score":null,"company_logo":{"company_id":0,"MaxFileSize":5242880,"company_photo_versions":{},"photo_versions":[],"has_image":false,"original_file_name":null,"original_file_path":null},"utc_expiration_date":"2023-01-16T00:00:00","utc_start_date":"2022-12-19T00:00:00","external_reference_id":null,"brand":null,"widget_id":null,"job_shifts":null,"cem_account_name":null,"dynamic_data":{},"external_posting_date":null,"external_location_id":null,"external_recruiter_name":null,"cost_center":null,"remote_type":null,"cache_statistics":null},{"distance":0,"distanceUnits":null,"id":"ae428e40-7588-11ed-80ce-643700856fc1","guid":"ae428e40-7588-11ed-80ce-643700856fc1","isGuidGenerated":false,"active":0,"company_id":157272,"company_name":"UC San Diego","company_type":"E","company_unique_name":"uc-san-diego-demo272","jobseeker_blacklist":[],"applied_jobseekers":[],"company_image_url":null,"job_url":"/clinical-lab-scientist-specialist-120483/job/21869984","custom_categories":[{"id":0,"url":null,"name":"Medical/Healthcare - Health","keywords":null},{"id":0,"url":null,"name":"Laboratory Services - Health","keywords":null}],"custom_fields":[],"CustomCategoriesString":["Medical/Healthcare - Health","Laboratory Services - Health"],"to_custom_site":false,"job_details":{"id":"ae428e40-7588-11ed-80ce-643700856fc1","job_title_url":"/clinical-lab-scientist-specialist-120483/job/21869984","title":"Clinical Lab Scientist / Specialist - 120483","job_cource":1,"company_name":"UC San Diego","locations":[{"id":23580175,"address_id":482227,"external_id":"","metro":["california","sandiego"],"street":"9300 Campus Point Dr","street2":null,"city":"La Jolla","state_name":"California","state_abbr":"CA","zip":92037,"latitude":32.838775,"longitude":-117.253243,"country":"United States of America","country_abbr":"US","is_remote":false,"has_geocode_override":false,"city_state_abbr":"La Jolla, CA","location_id":[],"miles":0,"city_state":"La Jolla, California","latitude_longitude":"32.838775000000000,-117.253243000000000","name":"Jacobs Medical Center","id_value":null,"area_id":0,"postalcode":"","is_main_office":false,"street_city_state":"9300 Campus Point Dr, La Jolla, CA","is_active":false,"sublocality":""}],"salary_amount":"0.0","job_status":null,"job_type":1,"expiration_date":"2023-01-03T10:09:01.797","content_short":" Payroll Title: CLIN LAB SCI SPEC Department: ITL - Immunogenetics and Transplantation Lab(42773 Hiring Salary Range $48.61 - $64.02 / Hour Worksite: La Jolla Medical Center Appointment Type: Career Appointment Percent: 100% Union: HX Contract Total Openi..."},"start_date":"2022-12-06T10:09:01.797","original_posting_date":"0001-01-01T00:00:00","updated_date":"2022-12-25T01:09:10.12","expiration_date":"2023-01-03T10:09:01.797","headline":"Clinical Lab Scientist / Specialist - 120483","headline_titlecase":"Clinical Lab Scientist / Specialist - 120483","content":null,"content_short":" Payroll Title: CLIN LAB SCI SPEC Department: ITL - Immunogenetics and Transplantation Lab(42773 Hiring Salary Range $48.61 - $64.02 / Hour Worksite: La Jolla Medical Center Appointment Type: Career Appointment Percent: 100% Union: HX Contract Total Openi...","tags":null,"skills":null,"important_notes":null,"reference":null,"reference_string":"120483","industry_url":null,"industry_name":"Education","pre_screen_id":0,"job_type":1,"job_type_name":"Standard","compensation_amount":"0.0","minimum_compensation":0,"compensation_comment":"","custom_apply_text":null,"custom_apply_link":"Apply Online","custom_apply_url":"https://jobs.ucsd.edu/Apply/BeforeApply.aspx?jid=120483&sid=100312.0","employment_types":[0],"EmploymentTypesCSV":"Not Specified","recruiters":[{"level":0,"company":null,"recruiter_id":237525,"title":"Manager","telephone":"(999) 999-9999","email_display":"Hide","email_format":"HTML","email_attachment":"Yes","crm_version":null,"umbrella":false,"password_expired":false,"email":"[email protected]","location":{"id":0,"address_id":0,"external_id":null,"metro":[],"street":null,"street2":null,"city":null,"state_name":null,"state_abbr":null,"zip":0,"latitude":0,"longitude":0,"country":null,"country_abbr":null,"is_remote":false,"has_geocode_override":false,"city_state_abbr":"","location_id":[],"miles":0,"city_state":"","latitude_longitude":"0,0","name":null,"id_value":null,"area_id":0,"postalcode":null,"is_main_office":false,"street_city_state":", , ","is_active":false,"sublocality":null},"session_guid":"00000000-0000-0000-0000-000000000000","is_mfa_enabled":false,"verification_hash":null,"retry_verification":false,"visitor_id":0,"key":null,"status":0,"guid":null,"first_name":"Scrape","last_name":"User","phones":[],"birth_date":"1900-01-01T00:00:00","photo":{"versions":{"version_125x125":"https://dot1p08cqyhea.cloudfront.net/default_pictures/125x125.png","version_80x80":"https://dot1p08cqyhea.cloudfront.net/default_pictures/80x80.png","version_50x50":"https://dot1p08cqyhea.cloudfront.net/default_pictures/50x50.png","version_20x20":"https://dot1p08cqyhea.cloudfront.net/default_pictures/20x20.png","version_30x30":"https://dot1p08cqyhea.cloudfront.net/default_pictures/30x30.png"},"has_photo":false,"original_url":"https://dot1p08cqyhea.cloudfront.net/pictures//","preview_url":"https://dot1p08cqyhea.cloudfront.net/pictures//"},"guest_id":null,"full_name":"Scrape User","login_session":{"object_type":0,"recruiter_id":0,"jobseeker_id":0,"date_stamp":"0001-01-01T00:00:00","date_stamp_string":"0001-01-01T00:00:00","hash_key":null,"cache_statistics":null},"cache_statistics":null}],"locations":[{"id":23580175,"address_id":482227,"external_id":"","metro":["california","sandiego"],"street":"9300 Campus Point Dr","street2":null,"city":"La Jolla","state_name":"California","state_abbr":"CA","zip":92037,"latitude":32.838775,"longitude":-117.253243,"country":"United States of America","country_abbr":"US","is_remote":false,"has_geocode_override":false,"city_state_abbr":"La Jolla, CA","location_id":[],"miles":0,"city_state":"La Jolla, California","latitude_longitude":"32.838775000000000,-117.253243000000000","name":"Jacobs Medical Center","id_value":null,"area_id":0,"postalcode":"","is_main_office":false,"street_city_state":"9300 Campus Point Dr, La Jolla, CA","is_active":false,"sublocality":""}],"lat_long":"32.838775000000000,-117.253243000000000","geo":["32.838775000000000,-117.253243000000000"],"areaid":null,"questions":[],"categories":[{"id":21,"url":"science-biotech-research","name":"Science/Biotech/Research","keywords":null},{"id":507,"url":"healthcare-lab-hematology-pathology","name":"Healthcare - 
</script>
Asked By: Baby Yoda

||

Answers:

I recall find_all is case-sensitive. In your description you are using "clinical lab scientist", but what I’m seeing in the doc is "Clinical Lab Scientist"

Answered By: ansa
tag = doc.find_all("clinical lab scientist")

That’s not how find_all() works. The first argument is the tag name of the elements you’re looking for:

tables = doc.find_all('table')

Your code found no results because there is no <clinical lab scientist> HTML element.

Usually, you would also pass a second argument to look for specific attribute values (such as id or class) to help narrow down the elements you want.

For example, if you wanted to find all <div class="groovy"> elements, you would say:

divs = doc.find_all('div', attrs={'class': 'groovy'})

So, you need to know the specific html element in which the "clinical lab scientist" text appears, so you can search on that element.

On the other hand, if you don’t really care about the HTML structure of the page, you might be able to do a simple string search on the html, without using BeautifulSoup at all.

Answered By: John Gordon

There are few things that you can do, depending on your expectations of the format of that page.
Say, you start with

    url = "https://employment.ucsd.edu/jobs?page_size=500&page_number=1&keyword=clinical%20lab%20scientist&location_city=Remote&location_city=San%20Diego&location_city=Encinitas&location_city=Murrieta&location_city=La%20Jolla&location_city=Not%20Specified&location_city=Vista&sort_by=score&sort_order=DESC"
    response = requests.get(url).text
    doc = BeautifulSoup(response, "html.parser")

Now, if you know that the tag is "input" and the attribute is always "value" then things are straightforward:

    tag = doc.find_all(
        "input",
        value="clinical lab scientist")

If you expect any tag (not necessarily "input") then you may omit the tag name:

    tag = doc.find_all(
        value="clinical lab scientist")

If you don’t know if the value is case sensitive, then you can use a regular expression:

    tag = doc.find_all(
        value=re.compile("^clinical lab scientist$", re.IGNORECASE))

Lastly, you can even do it if you are uncertain of the attribute name (we were using "value" thus far). You can use a lambda instead of tag name, and implement it so that it would pass if ANY attribute of that tag has a matching value:

    tag = doc.find_all(
        lambda tag: [v for v in tag.attrs.values()
                     if v == "clinical lab scientist"])
  • with or without using regular expressions.

Something like that. Using "any" in last snippet may optimize things.

Answered By: ansa