How can I grab only divs containing a specific tag inside

Question

I am trying to scrape this website (https://mentalmars.com/gear/brightside) for a very specific line(s) of text. From my current code I am grabbing every div with the correct class but the content inside of, and quantity of these divs can change depending on the search. Therefore, I am trying to grab only divs containing an h3 tag with the text "Special Weapon Effect" inside of it. I have researched as much of the bs4 documentation as my beginner brain can handle and even tried converting each one of these divs into list objects and just parsing through them individually to search for my specific tag but I ran into trouble converting ResultSet elements and have been stumped since then. Any possible help / pointers would be greatly appreciated. (P.S. I’ll mention again that I am fairly new to most of these concepts and have little to no prior experience.)

import requests
from bs4 import BeautifulSoup
inp = 'brightside'
#
headers = {'User-Agent': '...'}
url = f'https://mentalmars.com/gear/{inp}/'
r = requests.get(url, headers)
soup = BeautifulSoup(r.content, 'html.parser')
x = soup.find_all('div', class_="wpb_text_column wpb_content_element")

EXAMPLE RESPONSE (This is containing the tag I want) :

<div class="wpb_text_column wpb_content_element">
  <div class="wpb_wrapper">
    <h3>Special Weapon Effect:</h3>
      <p><span style="color: #ff0000;">“It’s killing me.”</span></p>
    <ul>
      <li>On Weapon Throw it will spawn 4 shotguns that each have their own unique  element. These will home into a nearby enemy and start shooting at it. Each shotgun will explode in their respected element.</li>
      <li>The x18 and x20 versions will consume 2 ammo per shot</li>
    </ul>
  </div>
</div>

EXAMPLE RESPONSE 2 (This is the example of multiple divs having the same class that I need but with different data inside):

<div class="wpb_text_column wpb_content_element">
  <div class="wpb_wrapper">
    <h3>Weapon Stats:</h3>
    <table>
      <tbody>
        <tr>
          <td><strong>Level:</strong></td>
          <td>60+</td>
        </tr>
        <tr>
          <td><strong>Damage:</strong></td>
          <td>1778×13</td>
        </tr>
         <tr>
          <td><strong>Accuracy:</strong></td>
          <td>49%</td>
         </tr>
         <tr>
          <td><strong>Handling:</strong></td>
          <td>61%</td>
        </tr>
         <tr>
          <td><strong>Reload Time:</strong></td>
          <td>2.0s</td>
        </tr>
        <tr>
          <td><strong>Fire Rate:</strong></td>
          <td>1.43/s</td>
        </tr>
        <tr>
          <td><strong>Magazine Size:</strong></td>
          <td>8</td>
        </tr>
      </tbody>
    </table>
  </div>
</div>

Asked By: riley9

||

Source

Answer 1

If you want to search only for "Special Effect", you can search by this string. For example:

import requests
from bs4 import BeautifulSoup

url = "https://mentalmars.com/gear/brightside"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

special_effect = soup.select_one(
    'div:has(>h3:-soup-contains("Special Weapon Effect"))'
)
print(
    *special_effect.get_text(strip=True, separator="n").split("n"), sep="n"
)

Prints:

Special Weapon Effect:
“It’s killing me.”
On Weapon Throw it will spawn 4 shotguns that each have their own unique element. These will home into a nearby enemy and start shooting at it. Each shotgun will explode in their respected element.
The x18 and x20 versions will consume 2 ammo per shot

div:has(>h3:-soup-contains("Special Weapon Effect")) means:

Give me <div> tag that has directly under it <h3> with text "Special Weapon Effect"

Answered By: Andrej Kesely

Answer 2

Another no-complexity solution would be:

import requests
from bs4 import BeautifulSoup

r = requests.get('https://mentalmars.com/gear/brightside')
soup = BeautifulSoup(r.text, 'html.parser')
element = soup.find('h3', string='Special Weapon Effect:').parent
print(element.get_text(strip=True))

This prints out:

Special Weapon Effect:“It’s killing me.”On Weapon Throw it will spawn 4 shotguns that each have their own unique element. These will home into a nearby enemy and start shooting at it. Each shotgun will explode in their respected element.The x18 and x20 versions will consume 2 ammo per shot

Answered By: platipus_on_fire

How can I grab only divs containing a specific tag inside

Question:

Answers: