Fetch URL hidden under the search button that is inside a form [Beautifulsoup]

The source of e.g. https://schoolinfo.leicester.gov.uk/SchoolDetails.html?r=1 (first entry) is https://schoolinfo.leicester.gov.uk/api/schools/1, where 1 is a reference to the BaseID. On how to find this source, see Update below. So, you can try something as follows:

import requests
import json

search = 'https://schoolinfo.leicester.gov.uk/api/schoolsearch/?schoolPhase=&schoolName=&pn=0&pa=900&schoolType='

# get list with dicts for each school
data = json.loads(requests.get(search).json())

# use key `BaseID` to get the ID for each school
ids = [i['BaseID'] for i in data]

# create empty `list` to store dicts for each school
schools = list()

# iterate over the `IDs` (use `i(ndex)` to keep track of entry for error message)
for i, _id in enumerate(ids):
    
    # create url string with `ID` added at end
    api = f'https://schoolinfo.leicester.gov.uk/api/schools/{_id}'
    
    # try to fetch data, on error: print error message
    try:
        school = json.loads(requests.get(api).json())

        # append school `dict` to list of schools
        schools.append(school)
    except:
        print(f'No data for school '{data[i]["SchoolName"]}' (ID: {_id})')

This will result in a list with 110 dictionaries. One for each school. The data for "Hope Hamilton C of E Primary School" (actually a duplicate; the only one) is missing. Indeed, if you click the link, the entry is just empty. So, the above will print:

No data for school 'Hope Hamilton C of E Primary School' (ID: 609)
No data for school 'Hope Hamilton C of E Primary School' (ID: 609)

You could create a df with this info. E.g.

df = pd.DataFrame(schools)

print(df.iloc[0])

SchoolName                                     Abbey Mead Primary Academy
BaseID                                                                  1
SchoolType                                                        Academy
DfeNumber                                                            2337
Ward                                                             Belgrave
RoleDesc                                                   Head Principal
Head                                                       Mr Gary Aldred
Address                 Abbey Mead Primary Academy, 109 Ross Walk, Lei...
Postcode                                                          LE4 5HH
BaseURL                                          http://www.abbey-tmet.uk
SchoolInspectionsURL     https://reports.ofsted.gov.uk/provider/21/147148
SchoolReportsURL        https://www.compare-school-performance.service...
SchoolVacanciesURL      https://www.eteach.com/microsite/ourjobs.aspx?...
Telephone                                                   0116 266 1809
Fax                                                          0116 2611543
Email                                       [email protected]
Sponsor                                        The Mead Educational Trust
UPRN                                                               147148
SchoolPhase                                                       Primary
MapUrl                 <div id="sample" style="width: 500px; height: ..."
AgeRange                                                             3-11
OverSubscribed                                                        Yes
ArrangementURL          https://www.leicester.gov.uk/schools-and-learn...
PAN                                                         Reception: 90
Name: 0, dtype: object

Update

Let me just add how you can figure out that https://schoolinfo.leicester.gov.uk/SchoolDetails.html?r=1 is based on https://schoolinfo.leicester.gov.uk/api/schools/1:

1. First go to the url https://schoolinfo.leicester.gov.uk/SchoolDetails.html?r=1 in Chrome. Open Chrome DevTools (Ctrl + Shift + J), and select "Sources" from the navigation bar.
2. To the left, you should now see a folder. We are interested in the first subfolder "(cloud) https://schoolinfo.leicester.gov.uk/". Inside it, navigate to "Scripts" and open the file "script.js".
3. Use "Ctrl + F" to open the search bar, and here I just searched for "api". You’ll get 1 match:

{
url: "/api/schools/" + uniqueRef,
success: function (result) {
    sObj = JSON.parse(result);
}

That’s the info we want!