How to requests all sizes in stock – Python

The sizes for the default color of shoe are shown in html. Alongside this are the urls for the other colors. You can extract these into a dictionary and loop, making requests and pulling the different colors and their availability, which I think is what you are actually requesting, as follows (note: I have kept quite generic to avoid hardcoding keys which change across requests):

import requests, re, json

def get_color_results(link):
    headers = {"User-Agent": "Mozilla/5.0"}
    r = requests.get(link, headers=headers).text
    data = json.loads(re.search(r'({"enrichedEntity".*size.*)</script', r).group(1))
    results = []
    color = ""
    for i in data["graphqlCache"]:
        if "ern:product" in i:
            if "product" in data["graphqlCache"][i]["data"]:
                if "name" in data["graphqlCache"][i]["data"]["product"]:
                    results.append(data["graphqlCache"][i]["data"]["product"])
                if (
                    color == ""
                    and "color" in data["graphqlCache"][i]["data"]["product"]
                ):
                    color = data["graphqlCache"][i]["data"]["product"]["color"]["name"]
    return (color, results)


link = "https://www.zalando.dk/nike-sportswear-air-max-90-sneakers-ni112o0bt-a11.html"
final = {}
color, results = get_color_results(link)
colors = {
    j["node"]["color"]["name"]: j["node"]["uri"]
    for j in [
        a
        for b in [
            i["family"]["products"]["edges"]
            for i in results
            if "family" in i
            if "products" in i["family"]
        ]
        for a in b
    ]
}
final[color] = {
    j["size"]: j["offer"]["stock"]["quantity"]
    for j in [i for i in results if "simples" in i][0]["simples"]
}

for k, v in colors.items():
    if k not in final:
        color, results = get_color_results(v)
        final[color] = {
            j["size"]: j["offer"]["stock"]["quantity"]
            for j in [i for i in results if "simples" in i][0]["simples"]
        }

print(final)

---

Explanatory notes from chat:

1. Use chrome browser to navigate to link

2. Press Ctrl + U to view page source

3. Press Ctrl + F to search for 38.5 in html

   The first match is the long string you already know about. The string is long and difficult to navigate in page source and identify which tag it is part of. There are a number of ways I could identify the right script from these, but for now, an easy way would be:

from bs4 import BeautifulSoup as bs

link = 'https://www.zalando.dk/nike-sportswear-air-max-90-sneakers-ni112o0bt-a11.html'
headers = {'User-Agent':'Mozilla/5.0'}
r = requests.get(link, headers = headers)
soup = bs(r.text, 'lxml')

for i in soup.select('script[type="application/json"]'):
    if '38.5' in i.text:
        print(i)
        break

Slower method would be:

soup.find("script", text=re.compile(r'.*38.5.*'))

4. Whilst I used bs4 to get the right script tag contents, this was so I knew the start and end of the string denoting the JavaScript object I wanted to use re to extract, and then to deserialize into a JSON object with json; this in a re-write to use re rather than bs4 i.e. use re on entire response text, from the request, and pass a regex pattern which would pull out the same string

5. I put the entire page source in a regex tool and wrote a regex to return that same string as identified above. See that regex here

6. Click on right hand side, match 1 group 1, to see highlighted the same string being returned from regex as you saw with BeautifulSoup. Two different ways of getting the same string containing the sizes

7. That is the string which I needed to examine, as JSON, the structure of. See in json viewer here

8. You will notice the JSON is very nested with some keys to dictionaries that are likely dynamic, meaning I needed to write code which could traverse the JSON and use certain more stable keys to pull out the colours available, and for the default shoe colour the sizes and availability

9. There is an expand all button in that JSON viewer. You can then search with Ctrl + F for 38.5 again

10a) I noticed that size and availability were for the default shoe colour

10b) I also noticed that within JSON, if I searched by one of the other colours from the dropdown, I could find URIs for each colour of show listed

11. I used Wolf as my search term (as I suspected less matches for that term within the JSON)

You can see one of the alternate colours and its URI listed above

12. I visited that URI and found the availability and shoe sizes for that colour in same place as I did for the default white shoes

13. I realised I could make an initial request and get the default colour and sizes with availability. From that same request, extract the other colours and their URIs

14. I could then make requests to those other URIs and re-use my existing code to extract the sizes/availability for the new colours

15. This is why I created my get_color_results() function. This was the re-usable code to extract the sizes and availability from each page

16. results holds all the matches within the JSON to certain keys I am looking for to navigate to the right place to get the sizes and availabilities, as well as the current colour

17. This code traverses the JSON to get to the right place to extract data I want to use later

results = []
color = ""
for i in data["graphqlCache"]:
    if "ern:product" in i:
        if "product" in data["graphqlCache"][i]["data"]:
            if "name" in data["graphqlCache"][i]["data"]["product"]:
                results.append(data["graphqlCache"][i]["data"]["product"])
            if (
                color == ""
                and "color" in data["graphqlCache"][i]["data"]["product"]
            ):
                color = data["graphqlCache"][i]["data"]["product"]["color"]["name"]

18. The following pulls out the sizes and availability from results:

{
    j["size"]: j["offer"]["stock"]["quantity"]
    for j in [i for i in results if "simples" in i][0]["simples"]
}

19. For the first request only, the following gets the other shoes colours and their URIs into a dictionary to later loop:

colors = {
    j["node"]["color"]["name"]: j["node"]["uri"]
    for j in [
        a
        for b in [
            i["family"]["products"]["edges"]
            for i in results
            if "family" in i
            if "products" in i["family"]
        ]
        for a in b
    ]
}

20. This bit gets all the other colours and their availability:

for k, v in colors.items():
    if k not in final:
        color, results = get_color_results(v)
        final[color] = {
            j["size"]: j["offer"]["stock"]["quantity"]
            for j in [i for i in results if "simples" in i][0]["simples"]
        }

21. Throughout, I update the dictionary final with the found colour and associated size and availabilities