AIOHTTP replacing %3A with :

Question

import yarl
async with cs.get(yarl.URL(f"https://ipqualityscore.com/api/json/url/{self.token}/{url}",encoded=True)) as r:

Hello, i’m having this issue where AIOHTTP is converting characters like %3A to the original :. i need to use the %3A version in the API req, if not, it raises 404

My code:

for link in results:
    url = urllib.parse.quote(link, safe = '')
    print(url)
    ## ^^ 1st ^^

    async with aiohttp.ClientSession() as cs:

        print(f"https://ipqualityscore.com/api/json/url/{self.token}/{url}")
        ## ^^ 2nd ^^

        async with cs.get(f"https://ipqualityscore.com/api/json/url/{self.token}/{url}") as r:
            text = await r.json()
            print(text)

URL it should’ve used:

https://ipqualityscore.com/api/json/url/PRIVATE_TOKEN/https%3A%2F%2Fstreancommunuty.ru%2Ftradoffer%2Fnew%2F%3Fpartner%3D1284276379%26token%3DiMDdLkoe

error raised (and url used):

aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8', url=URL('https://ipqualityscore.com/api/json/url/PRIVATE_TOKEN/https:%2F%2Fstreancommunuty.ru%2Ftradoffer%2Fnew%2F%3Fpartner=1284276379&token=iMDdLkoe')

Asked By: LeoCx1000

||

Source

Answer 1

EDIT:

Minimal working code based on @Weeble answer.

It uses yarl with encoded=True to stop requoting %3A to :

import urllib.parse
import aiohttp
import asyncio
import yarl

import os
token = os.getenv('IPQUALITYSCORE_TOKEN')

link = 'https://streancommunuty.ru/tradoffer/new/?partner=1284276379&token=iMDdLkoe'

async def main(link):
    url = urllib.parse.quote(link, safe='')
    print('--- url ---')
    print(url)
    
    async with aiohttp.ClientSession() as cs:
        
        yarl_url = yarl.URL(f"https://ipqualityscore.com/api/json/url/{token}/{url}", encoded=True)
                            
        async with cs.get(yarl_url) as r:
            #print('--- text ---')
            #text = await r.text() 
            #print(text)
            print('--- data ---')
            data = await r.json()
            print(data)
            print('--- url ---')
            print(r.url)
            
loop = asyncio.get_event_loop()
loop.run_until_complete(main(link))

EDIT:

I found Request url of client session gets malformed #3424 which shows that it uses module yarl` which automatically reqoute some chars. It is doing to help create correct URL but in your situation it makes only problem. It may need to change source code to stop it.

Below is old version which doesn’t resolve all problems.

OLD: (it doesn’t resolve main problem)

Code doesn’t raise error if I quote link two times

url = urllib.parse.quote(link, safe='')  # first time
url = urllib.parse.quote(url)            # second time

but I don’t have TOKEN so I get message Invalid or unauthorized key from server and I can’t check if this resolves all problems.

Minimal working code for test.

import urllib.parse
import aiohttp
import asyncio

token = 'PRIVATE_TOKEN'
link = 'https://streancommunuty.ru/tradoffer/new/?partner=1284276379&token=iMDdLkoe'

async def main(link):
    url = urllib.parse.quote(link, safe='')
    url = urllib.parse.quote(url)
    print('--- url ---')
    print(url)
    
    async with aiohttp.ClientSession() as cs:
    
        async with cs.get(f"https://ipqualityscore.com/api/json/url/{token}/{url}") as r:
            #print('--- text ---')
            #text = await r.text()
            #print(text)
            data = await r.json()
            print('--- data ---')
            print(data)
            
loop = asyncio.get_event_loop()
loop.run_until_complete(main(link))

Result:

--- url ---
https%253A%252F%252Fstreancommunuty.ru%252Ftradoffer%252Fnew%252F%253Fpartner%253D1284276379%2526token%253DiMDdLkoe
--- data ---
{'success': False, 'message': 'Invalid or unauthorized key. Please check the API key and try again.', 'request_id': '4DqddqGpINmFBAI'}

Answered By: furas

Answer 2

First of all, are you sure this is what you want to do? I ask because while : is a reserved character in URLs, it is not used as a delimiter in the path component of a URL, and so whether or not it is percent-encoded it should mean the exact same thing to the web server. Are you certain that whether the : is percent-encoded is the only thing causing your problem? That said, it’s possible this particular web server isn’t following the RFC properly, in which case it might be you need to work around it.

If it is what you want to do, I think you need to prevent aiohttp from normalizing the URL. From the answer to that question, it sounds like you could do something like this:

import yarl

...

ipqs_url = yarl.URL(
    f"https://ipqualityscore.com/api/json/url/{self.token}/{url}",
    encoded=True)
await ctx.send(ipqs_url)

Similarly, you can pass a yarl.URL object to cs.get.

Answered By: Weeble

AIOHTTP replacing %3A with :

Question:

My code:

URL it should’ve used:

error raised (and url used):

Answers: