Web scraping table from UniProt database

Question

I have a list of UniProt IDs and would like to use BeautifulSoup to scrap a table containing the structure information. The url I am using is as follows: https://www.uniprot.org/uniprot/P03496, with accession "P03496".

A snippet of the html code is as follows.

<div class="main-aside">
    <div class="content entry_view_content up_entry swissprot">
        <div class="section" id="structure">
            <protvista-uniprot-structure accession="P03468">
                <div class="protvista-uniprot-structure">
                    <div class="class=" protvista-uniprot-structure__table">
                        <protvista-datatable class="feature">
                            <table>...</table>
                        </protvista-datatable>
                    </div>
                </div>
            </protvista-uniprot-structure>
        </div>
    </div>
</div>

The information I require is contained between the <table>...</table> tag.

I tried

from bs4 import BeautifulSoup
import requests

url='https://www.uniprot.org/uniprot/P03468'
r=requests.get(url)
url=r.content
soup = BeautifulSoup(url,'html.parser')
soup.find("protvista-datatable", {"class": "feature"})
print(soup)

Asked By: ann

||

Source

Answer 1

Content is provided dynamically and is not contained in your soup if you take a deeper look. It do not need BeautifulSoupto get data, your tabel is based on, simply use their api / rest interface to get structured data as JSON:

import requests
url='https://rest.uniprot.org/uniprot/P03468'
## fetch the json response
data = requests.get(url).json()
## pick needed data e.g. 
data['uniProtKBCrossReferences']

Output

[{'database': 'EMBL',
  'id': 'J02146',
  'properties': [{'key': 'ProteinId', 'value': 'AAA43412.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'AF389120',
  'properties': [{'key': 'ProteinId', 'value': 'AAM75160.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'EF467823',
  'properties': [{'key': 'ProteinId', 'value': 'ABO21711.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'CY009446',
  'properties': [{'key': 'ProteinId', 'value': 'ABD77678.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'K01031',
  'properties': [{'key': 'ProteinId', 'value': 'AAA43415.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'RefSeq',
  'id': 'NP_040981.1',
  'properties': [{'key': 'NucleotideSequenceId', 'value': 'NC_002018.1'}]},
 {'database': 'PDB',
  'id': '6WZY',
  'properties': [{'key': 'Method', 'value': 'X-ray'},
   {'key': 'Resolution', 'value': '1.50 A'},
   {'key': 'Chains', 'value': 'C=181-190'}]},...]

Answered By: HedgeHog

Answer 2

There’s a Python package, Unipressed, by Michael Milton (@multimeric) that allows programmatic access query UniProt’s new REST API.
Example:

from unipressed import UniprotkbClient
UniprotkbClient.fetch_one("P03468")["uniProtKBCrossReferences"]

Output

[{'database': 'EMBL',
  'id': 'J02146',
  'properties': [{'key': 'ProteinId', 'value': 'AAA43412.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'AF389120',
  'properties': [{'key': 'ProteinId', 'value': 'AAM75160.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'EF467823',
  'properties': [{'key': 'ProteinId', 'value': 'ABO21711.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'CY009446',
  'properties': [{'key': 'ProteinId', 'value': 'ABD77678.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'EMBL',
  'id': 'K01031',
  'properties': [{'key': 'ProteinId', 'value': 'AAA43415.1'},
   {'key': 'Status', 'value': '-'},
   {'key': 'MoleculeType', 'value': 'Genomic_RNA'}]},
 {'database': 'RefSeq',
  'id': 'NP_040981.1',
  'properties': [{'key': 'NucleotideSequenceId', 'value': 'NC_002018.1'}]},
 {'database': 'PDB',
  'id': '6WZY',
  'properties': [{'key': 'Method', 'value': 'X-ray'},
   {'key': 'Resolution', 'value': '1.50 A'},
   {'key': 'Chains', 'value': 'C=181-190'}]}, ... ]

See more examples of using Unipressed to access Uniprot’s new REST API here in my reply to Biostar’s post ‘Accessing UNIPROT using REST API’. See using Unipressed for ID mapping here and here.

Answered By: Wayne

Web scraping table from UniProt database

Question:

Answers:

Output

Output