Web scraping: HTTPError: HTTP Error 400: Bad Request

Question:

I’m trying to scrape this site: [https://www.icriq.com/fr/][1]

I need to search by company name, and get the details of the company as shown in [this example company page][2].

I wrote the following code:

import requests
from bs4 import BeautifulSoup


api_url ='https://www.icriq.com/pls/owa_rib/ribw_recherche.rech_rap'

headers= {
    "Content-Type":"application/x-www-form-urlencoded",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/110.0"}

body_first_page="p_lang=fr&p_portail=&p_inclus_req=N&p_ecoresp=N&p_ind_lpdq=N&p_mot_cle=agrimetal&p_type_rech=NOM&p_tab_alim_atpro=-1"
res = requests.post(api_url,data=body_first_page,headers=headers)

soup = BeautifulSoup(res.text,'lxml')

The resulting soup has the following href, associated with ‘AGRIMETAL INC.’:

href="/pls/owa_rib/ribwaff1.afficher_profil?p_id_req=60354405&p_cle=8POJTR9O2P"

However, when I tried to use it to make another request using the following function:

def get_soup(url):
    req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
    #time.sleep(10)
    html_page = urlopen(req).read()
    #time.sleep(10)
    soup = BeautifulSoup(html_page, 'html.parser')
    return soup

I got an HTTPError: HTTP Error 400: Bad Request error
[1]: https://www.icriq.com/fr/
[2]: https://www.icriq.com/pls/owa_rib/ribwaff1.afficher_profil?p_id_req=60354349&p_cle=NOWLSUZQKM

Asked By: Mohamed Hedeya

||

Answers:

If you use /pls/owa_rib/ribwaff1.afficher_profil?p_id_req=60354405&p_cle=8POJTR9O2P. Then just replace & by & .

Then assemble the url like this : https://www.icriq.com/pls/owa_rib/ribwaff1.afficher_profil?p_id_req=60354405&p_cle=8POJTR9O2P

Answered By: Corentin Lecroq