SQLalchemy not find table for creating foreign key

The solution is to replace the strings with actual columns:

Column('person_id', Integer, ForeignKey(tbl_person.c.id), primary_key=True)

In case of Declarative, I solved this problem by simply importing the class that was ‘could not be found’.

Conclusion

This exception was caused because that there is no record of parent table in MetaData instance, and this table needs to be retrieved in the DB.
Invoke the function "reflect" of class MetaData to obtain all existed tables on the DB. It should be used like this

def upgrade(migrate_engine):

    meta.bind = migrate_engine

    meta.reflect() # <------ Obtain all tables here.

    aggregate_metadata.create()
    aggregate_hosts.create()

Description

A table is in a different file from another table that has a foreign key associate with it.
In this case, sqlalchemy will fail to find the corresponding table when create tables, as shown below:

sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column ‘aggregate_metadata.aggregate_id’ could not find table ‘aggregates’ with which to generate a foreign key to target column ‘id’

For example:

# File 002_Add_aggregates_table.py
# ========================================
...

meta = MetaData()

aggregates = Table('aggregates', meta,
    ...
    Column('id', Integer, primary_key=True, nullable=False),
    mysql_engine='InnoDB',
    mysql_charset='utf8'
)

def upgrade(migrate_engine):
    meta.bind = migrate_engine
    aggregates.create()

# File 003_Add_aggregate_metadata_hosts.py
# ========================================
...

meta = MetaData()

aggregate_metadata = Table('aggregate_metadata', meta,
    ...
    Column('aggregate_id', Integer, ForeignKey('aggregates.id'), # <------ ForeignKey
           nullable=False),
    mysql_engine='InnoDB',
    mysql_charset='utf8'
    )

def upgrade(migrate_engine):

    meta.bind = migrate_engine
    aggregate_metadata.create()

Root cause

Let’s locate at the point that throw the exception

  File "/opt/xxx/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 1113, in __get__
    obj.__dict__[self.__name__] = result = self.fget(obj)
  File "/opt/xxx/.local/lib/python3.6/site-packages/sqlalchemy/sql/schema.py", line 2394, in column
    tablekey,
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'aggregate_metadata.aggregate_id' could not find table 'aggregates' with which to generate a foreign key to target column 'id'

We could find the corresponding code and debug it:

# File: sqlalchemy/sql/schema.py
2358     def column(self):
            ...
2371
2372        if isinstance(self._colspec, util.string_types):
2373
2374           parenttable, tablekey, colname = self._resolve_col_tokens()

# =========> BeginDebug
2375           raise Exception(
2376               'imgrass->n'
2377               '  - parenttable: %sn'
2378               '  - parenttable.metadata: %sn'
2379               '  - tablekey: %sn'
2380               '  - colname: %s' % (
2381                   parenttable,
2382                   parenttable.metadata,
2383                   tablekey,
2384                   colname
2385                   )
2386               )
# =========> EndDebug

2387
2388           if tablekey not in parenttable.metadata:
2389              raise exc.NoReferencedTableError(
2390                 "Foreign key associated with column '%s' could not find "
2391                 "table '%s' with which to generate a "
2392                 "foreign key to target column '%s'"
2393                 % (self.parent, tablekey, colname),
2394                 tablekey,
2395              )

Then we could get below exceptions:

Exception: imgrass->
  - parenttable: aggregate_metadata
  - parenttable.metadata: MetaData(bind=Engine(mysql+pymysql://imgrass:***@172.17.0.1/demo))
  - tablekey: aggregates
  - colname: id

So, the parenttable.metadata is an instance of class MetaData, and the tablekey is a table name.
We could reasonably guess that the table aggregates should be included in the instance of class MetaData.
Considering that the definition of this table is in another file, and the MetaData instance has the connection way of DB(bind=xxx), there should be a function in class MetaData to obtain all tables in the DB.

In MetaData, we could find this function

# File: sqlalchemy/sql/schema.py

class MetaData(SchemaItem):
    ...

    def reflect(...):
        r"""Load all available table definitions from the database.
        ...

From its description, we could guess its function, let’s apply it in my script:

# File 003_Add_aggregate_metadata_hosts.py
# ========================================
...

def upgrade(migrate_engine):

    meta.bind = migrate_engine

# ==================> BeginUpdate
    meta.reflect()
# ==================> EndUpdate

    aggregate_metadata.create()
    aggregate_hosts.create()

It’s okay!

Although the top voted answer solves the issue, replacing strings with objects forces you to define tables in a specific order (which can be non-trivial for very large databases). From the SQLAlchemy docs:

The advantage to using a string is that the in-python linkage between [different tables] is resolved only when first needed, so that table objects can be easily spread across multiple modules and defined in any order.

You can continue to use strings by passing the schema to the ForeignKey. For example, instead of doing:

tbl_estate_agent = Table(
   'estate_agent', meta,
   Column('person_id', Integer, ForeignKey("person.id"), primary_key = True),
   Column('prize_range_id', Integer, ForeignKey("prize_range.id"), nullable = False),
   schema = 'public')

One can do:

tbl_estate_agent = Table(
   'estate_agent', meta,
   Column('person_id', Integer, ForeignKey("public.person.id"), primary_key = True),
   Column('prize_range_id', Integer, ForeignKey("public.prize_range.id"), nullable = False),
   schema = 'public')