Better strategy to update mysql table on distinct column from another table

Question

I have 2 mysql-tables (MariaDB) – games and lineups where the latter is filled from a third source.

The logic behind them is: Each entry in the first table ‘games’ should have corresponding multiple entries of players participating in the respective game in the second table ‘lineups’.

So it looks like this:

games

matchid (Unique Key)  some_data     ...        requestedPlayers
100                   "foobar"      "bar"      1
101                   "foobar"      "bar"      1
102                   "foobar"      "bar"      0


lineups

matchid     playerid    some_data     ...
100         1           ...           ...
100         2           ...           ...
101         1           ...           ...
101         2           ...           ...

So when the players for a matchid are already in the lineups table, the boolean column requestedPlayers should be set to 1. My python code then reads the matchids into self.gameids from the table matches with a simple:
SELECT matchID FROM games WHERE requestedPlayers = 0. Then it loops through the IDs, retrieves data from 2nd source, parses it, and collects in in self.lineupdata.

The corresponding code looks like this:

   def lineups(self,number:int=1000):
        self._read_gameIds()          # `SELECT matchID FROM games WHERE requestedPlayers = 0` 
        try: 
            for gid in self.gameids:
                if number ==0: break
                print(gid)
                self._request_match(gid)
                number -= 1
        finally:
            if len(self.lineupdata) >0:
                self._write_lineups()
                self._update_gameIds()

I do this in batches (batch size regulated by number) and the code crashes regularily as the data is parsed and it isn’t clean, so a lot of unexpected things happen. By catching the exceptions I finally write the retrieved data into the database (self._write_lineups()) and in a last step update the games table column requestedLineups with this mySQL-statement:

'''UPDATE games g, lineups l
            SET g.requestedLineUp = 1
            WHERE g.matchID IN (SELECT DISTINCT matchID FROM lineups)'''

This worked fine for some time, but as lineups grows, this takes more and more time so I am looking for an alternative strategy now. So my questions are:

Is there efficient way to UPDATE the requestedLineUp-Column?

Or is there a different way the whole process needs to be done? Maybe it would be better to write into lineups after each matchid and not collect them before writing. Then I could update the respective matchid-row in matches in the same step. Is there a best parctice from a mysql-perspective?

Asked By: J_Scholz

||

Source

Answer 1

UPDATE games
SET requestedLineUp = EXISTS ( SELECT NULL
                               FROM lineups 
                               WHERE games.matchID = lineups.matchID 
                               )
/* WHERE requestedPlayers = 0        -- ??? */;

Answered By: Akina

Better strategy to update mysql table on distinct column from another table

Question:

Answers: