Web scraping: HTTPError: HTTP Error 400: Bad Request


I’m trying to scrape this site: [https://www.icriq.com/fr/][1]

I need to search by company name, and get the details of the company as shown in [this example company page][2].

I wrote the following code:

import requests
from bs4 import BeautifulSoup

api_url ='https://www.icriq.com/pls/owa_rib/ribw_recherche.rech_rap'

headers= {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0"}

res = requests.post(api_url,data=body_first_page,headers=headers)

soup = BeautifulSoup(res.text,'lxml')

The resulting soup has the following href, associated with ‘AGRIMETAL INC.’:


However, when I tried to use it to make another request using the following function:

def get_soup(url):
    req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
    html_page = urlopen(req).read()
    soup = BeautifulSoup(html_page, 'html.parser')
    return soup

I got an HTTPError: HTTP Error 400: Bad Request error
[1]: https://www.icriq.com/fr/
[2]: https://www.icriq.com/pls/owa_rib/ribwaff1.afficher_profil?p_id_req=60354349&p_cle=NOWLSUZQKM

Asked By: Mohamed Hedeya



If you use /pls/owa_rib/ribwaff1.afficher_profil?p_id_req=60354405&p_cle=8POJTR9O2P. Then just replace & by & .

Then assemble the url like this : https://www.icriq.com/pls/owa_rib/ribwaff1.afficher_profil?p_id_req=60354405&p_cle=8POJTR9O2P

Answered By: Corentin Lecroq