Duplicating model instances and their related objects in Django / Algorithm for recusrively duplicating an object

I haven’t tried it in django but python’s deepcopy might just work for you

EDIT:

You can define custom copy behavior for your models if you implement functions:

__copy__() and __deepcopy__()

I think you’d be happier with a simpler data model, also.

Is it really true that a Page is in some Chapter but a different book?

userMe = User( username="me" )
userYou= User( username="you" )
bookMyA = Book( userMe )
bookYourB = Book( userYou )

chapterA1 = Chapter( book= bookMyA, author=userYou ) # "me" owns the Book, "you" owns the chapter?

chapterB2 = Chapter( book= bookYourB, author=userMe ) # "you" owns the book, "me" owns the chapter?

page1 = Page( book= bookMyA, chapter= chapterB2, author=userMe ) # Book and Author aggree, chapter doesn't?

It seems like your model is too complex.

I think you’d be happier with something simpler. I’m just guessing at this, since I don’t your know entire problem.

class Book(models.Model)
    name = models.CharField(...)

class Chapter(models.Model)
    name = models.CharField(...)
    book = models.ForeignKey(Book)

class Page(models.Model)
    author = models.ForeignKey('auth.User')
    chapter = models.ForeignKey(Chapter)

Each page has distinct authorship. Each chapter, then, has a collection of authors, as does the book. Now you can duplicate Book, Chapter and Pages, assigning the cloned Pages to the new Author.

Indeed, you might want to have a many-to-many relationship between Page and Chapter, allowing you to have multiple copies of just the Page, without cloning book and Chapter.

If there’s just a couple copies in the database you’re building, I’ve found you can just use the back button in the admin interface, change the necessary fields and save the instance again. This has worked for me in cases where, for instance, I need to build a “gimlet” and a “vodka gimlet” cocktail where the only difference is replacing the name and an ingredient. Obviously, this requires a little foresight of the data and isn’t as powerful as overriding django’s copy/deepcopy – but it may do the trick for some.

Here’s an easy way to copy your object.

Basically:

(1) set the id of your original object to None:

book_to_copy.id = None

(2) change the ‘author’ attribute and save the ojbect:

book_to_copy.author = new_author

book_to_copy.save()

(3) INSERT performed instead of UPDATE

(It doesn’t address changing the author in the Page–I agree with the comments regarding re-structuring the models)

this is an edit of http://www.djangosnippets.org/snippets/1282/

It’s now compatible with the Collector which replaced CollectedObjects in 1.3.

I didn’t really test this too heavily, but did test it with an object with about 20,000 sub-objects, but in only about three layers of foreign-key depth. Use at your own risk of course.

For the ambitious guy who reads this post, you should consider subclassing Collector (or copying the entire class to remove this dependency on this unpublished section of the django API) to a class called something like “DuplicateCollector” and writing a .duplicate method that works similarly to the .delete method. that would solve this problem in a real way.

from django.db.models.deletion import Collector
from django.db.models.fields.related import ForeignKey

def duplicate(obj, value=None, field=None, duplicate_order=None):
    """
    Duplicate all related objects of obj setting
    field to value. If one of the duplicate
    objects has an FK to another duplicate object
    update that as well. Return the duplicate copy
    of obj.
    duplicate_order is a list of models which specify how
    the duplicate objects are saved. For complex objects
    this can matter. Check to save if objects are being
    saved correctly and if not just pass in related objects
    in the order that they should be saved.
    """
    collector = Collector({})
    collector.collect([obj])
    collector.sort()
    related_models = collector.data.keys()
    data_snapshot =  {}
    for key in collector.data.keys():
        data_snapshot.update({ key: dict(zip([item.pk for item in collector.data[key]], [item for item in collector.data[key]])) })
    root_obj = None

    # Sometimes it's good enough just to save in reverse deletion order.
    if duplicate_order is None:
        duplicate_order = reversed(related_models)

    for model in duplicate_order:
        # Find all FKs on model that point to a related_model.
        fks = []
        for f in model._meta.fields:
            if isinstance(f, ForeignKey) and f.rel.to in related_models:
                fks.append(f)
        # Replace each `sub_obj` with a duplicate.
        if model not in collector.data:
            continue
        sub_objects = collector.data[model]
        for obj in sub_objects:
            for fk in fks:
                fk_value = getattr(obj, "%s_id" % fk.name)
                # If this FK has been duplicated then point to the duplicate.
                fk_rel_to = data_snapshot[fk.rel.to]
                if fk_value in fk_rel_to:
                    dupe_obj = fk_rel_to[fk_value]
                    setattr(obj, fk.name, dupe_obj)
            # Duplicate the object and save it.
            obj.id = None
            if field is not None:
                setattr(obj, field, value)
            obj.save()
            if root_obj is None:
                root_obj = obj
    return root_obj

EDIT: Removed a debugging “print” statement.

Django does have a built-in way to duplicate an object via the admin – as answered here:
In the Django admin interface, is there a way to duplicate an item?

In Django 1.5 this works for me:

thing.id = None
thing.pk = None
thing.save()

Using the CollectedObjects snippet above no longer works but can be done with the following modification:

from django.contrib.admin.util import NestedObjects
from django.db import DEFAULT_DB_ALIAS

and

collector = NestedObjects(using=DEFAULT_DB_ALIAS)

instead of CollectorObjects

Simple non generic way

Proposed solutions didn’t work for me, so I went the simple, not clever way. This is only useful for simple cases.

For a model with the following structure

Book
 |__ CroppedFace
 |__ Photo
      |__ AwsReco
            |__ AwsLabel
            |__ AwsFace
                  |__ AwsEmotion

this works

def duplicate_book(book: Book, new_user: MyUser):
    # AwsEmotion, AwsFace, AwsLabel, AwsReco, Photo, CroppedFace, Book

    old_cropped_faces = book.croppedface_set.all()
    old_photos = book.photo_set.all()

    book.pk = None
    book.user = new_user
    book.save()

    for cf in old_cropped_faces:
        cf.pk = None
        cf.book = book
        cf.save()

    for photo in old_photos:
        photo.pk = None
        photo.book = book
        photo.save()

        if hasattr(photo, 'awsreco'):
            reco = photo.awsreco
            old_aws_labels = reco.awslabel_set.all()
            old_aws_faces = reco.awsface_set.all()
            reco.pk = None
            reco.photo = photo
            reco.save()

            for label in old_aws_labels:
                label.pk = None
                label.reco = reco
                label.save()

            for face in old_aws_faces:
                old_aws_emotions = face.awsemotion_set.all()
                face.pk = None
                face.reco = reco
                face.save()

                for emotion in old_aws_emotions:
                    emotion.pk = None
                    emotion.aws_face = face
                    emotion.save()
    return book

I tried a few of the answers in Django 2.2/Python 3.6 and they didn’t seem to copy one-to-many and many-to-many related objects. Also, many included hardcoding / incorporated foreknowledge of the data structures.

I wrote a way to do this in a more generic fashion, handling one-to-many and many-to-many related objects. Comments included, and I’m looking to improve upon it if you have suggestions:

def duplicate_object(self):
    """
    Duplicate a model instance, making copies of all foreign keys pointing to it.
    There are 3 steps that need to occur in order:

        1.  Enumerate the related child objects and m2m relations, saving in lists/dicts
        2.  Copy the parent object per django docs (doesn't copy relations)
        3a. Copy the child objects, relating to the copied parent object
        3b. Re-create the m2m relations on the copied parent object

    """
    related_objects_to_copy = []
    relations_to_set = {}
    # Iterate through all the fields in the parent object looking for related fields
    for field in self._meta.get_fields():
        if field.one_to_many:
            # One to many fields are backward relationships where many child 
            # objects are related to the parent. Enumerate them and save a list 
            # so we can copy them after duplicating our parent object.
            print(f'Found a one-to-many field: {field.name}')

            # 'field' is a ManyToOneRel which is not iterable, we need to get
            # the object attribute itself.
            related_object_manager = getattr(self, field.name)
            related_objects = list(related_object_manager.all())
            if related_objects:
                print(f' - {len(related_objects)} related objects to copy')
                related_objects_to_copy += related_objects

        elif field.many_to_one:
            # In testing, these relationships are preserved when the parent
            # object is copied, so they don't need to be copied separately.
            print(f'Found a many-to-one field: {field.name}')

        elif field.many_to_many:
            # Many to many fields are relationships where many parent objects
            # can be related to many child objects. Because of this the child
            # objects don't need to be copied when we copy the parent, we just
            # need to re-create the relationship to them on the copied parent.
            print(f'Found a many-to-many field: {field.name}')
            related_object_manager = getattr(self, field.name)
            relations = list(related_object_manager.all())
            if relations:
                print(f' - {len(relations)} relations to set')
                relations_to_set[field.name] = relations

    # Duplicate the parent object
    self.pk = None
    self.save()
    print(f'Copied parent object ({str(self)})')

    # Copy the one-to-many child objects and relate them to the copied parent
    for related_object in related_objects_to_copy:
        # Iterate through the fields in the related object to find the one that 
        # relates to the parent model.
        for related_object_field in related_object._meta.fields:
            if related_object_field.related_model == self.__class__:
                # If the related_model on this field matches the parent
                # object's class, perform the copy of the child object and set
                # this field to the parent object, creating the new
                # child -> parent relationship.
                related_object.pk = None
                setattr(related_object, related_object_field.name, self)
                related_object.save()

                text = str(related_object)
                text = (text[:40] + '..') if len(text) > 40 else text
                print(f'|- Copied child object ({text})')

    # Set the many-to-many relations on the copied parent
    for field_name, relations in relations_to_set.items():
        # Get the field by name and set the relations, creating the new
        # relationships.
        field = getattr(self, field_name)
        field.set(relations)
        text_relations = []
        for relation in relations:
            text_relations.append(str(relation))
        print(f'|- Set {len(relations)} many-to-many relations on {field_name} {text_relations}')

    return self

I had no luck with any of the answers here with Django 2.1.2, so I created a generic way of performing a deep copy of a database model that is heavily based on the answers posted above.

The key differences from the answers above is that ForeignKey no longer has an attribute called rel, so it has to be changed to f.remote_field.model etc.

Furthermore, because of the difficulty of knowing the order the database models should be copied in, I created a simple queuing system that pushes the current model to the end of the list if it is unsuccessfully copied. The code is postet below:

import queue
from django.contrib.admin.utils import NestedObjects
from django.db.models.fields.related import ForeignKey

def duplicate(obj, field=None, value=None, max_retries=5):
    # Use the Nested Objects collector to retrieve the related models
    collector = NestedObjects(using='default')
    collector.collect([obj])
    related_models = list(collector.data.keys())

    # Create an object to map old primary keys to new ones
    data_snapshot = {}
    model_queue = queue.Queue()
    for key in related_models:
        data_snapshot.update(
            {key: {item.pk: None for item in collector.data[key]}}
        )
        model_queue.put(key)

    # For each of the models in related models copy their instances
    root_obj = None
    attempt_count = 0
    while not model_queue.empty():
        model = model_queue.get()
        root_obj, success = copy_instances(model, related_models, collector, data_snapshot, root_obj)

        # If the copy is not a success, it probably means that not
        # all the related fields for the model has been copied yet.
        # The current model is therefore pushed to the end of the list to be copied last
        if not success:

            # If the last model is unsuccessful or the number of max retries is reached, raise an error
            if model_queue.empty() or attempt_count > max_retries:
                raise DuplicationError(model)
            model_queue.put(model)
            attempt_count += 1
    return root_obj

def copy_instances(model, related_models, collector, data_snapshot, root_obj):

# Store all foreign keys for the model in a list
fks = []
for f in model._meta.fields:
    if isinstance(f, ForeignKey) and f.remote_field.model in related_models:
        fks.append(f)

# Iterate over the instances of the model
for obj in collector.data[model]:

    # For each of the models foreign keys check if the related object has been copied
    # and if so, assign its personal key to the current objects related field
    for fk in fks:
        pk_field = f"{fk.name}_id"
        fk_value = getattr(obj, pk_field)

        # Fetch the dictionary containing the old ids
        fk_rel_to = data_snapshot[fk.remote_field.model]

        # If the value exists and is in the dictionary assign it to the object
        if fk_value is not None and fk_value in fk_rel_to:
            dupe_pk = fk_rel_to[fk_value]

            # If the desired pk is none it means that the related object has not been copied yet
            # so the function returns unsuccessful
            if dupe_pk is None:
                return root_obj, False

            setattr(obj, pk_field, dupe_pk)

    # Store the old pk and save the object without an id to create a shallow copy of the object
    old_pk = obj.id
    obj.id = None

    if field is not None:
        setattr(obj, field, value)

    obj.save()

    # Store the new id in the data snapshot object for potential use on later objects
    data_snapshot[model][old_pk] = obj.id

    if root_obj is None:
        root_obj = obj

return root_obj, True

I hope it is of any help 🙂

The duplication error is just a simple exception extension:

class DuplicationError(Exception):
    """
    Is raised when a duplication operation did not succeed

    Attributes:
        model -- The database model that failed
    """

    def __init__(self, model):
        self.error_model = model

    def __str__(self):
        return f'Was not able to duplicate database objects for model {self.error_model}'

Here is a somewhat simple-minded solution. This does not depend on any undocumented Django APIs. It assumes that you want to duplicate a single parent record, along with its child, grandchild, etc. records. You pass in a whitelist of classes that should actually be duplicated, in the form of a list of names of the one-to-many relationships on each parent object that point to its child objects. This code assumes that, given the above whitelist, the entire tree is self-contained, with no external references to worry about.

This solution doesn’t do anything special for the author field above. I’m not sure if it would work with that. Like others have said, that author field probably shouldn’t be repeated in different model classes.

One more thing about this code: it is truly recursive, in that it calls itself for each new level of descendants.

from collections import OrderedDict

def duplicate_model_with_descendants(obj, whitelist, _new_parent_pk=None):
    kwargs = {}
    children_to_clone = OrderedDict()
    for field in obj._meta.get_fields():
        if field.name == "id":
            pass
        elif field.one_to_many:
            if field.name in whitelist:
                these_children = list(getattr(obj, field.name).all())
                if children_to_clone.has_key(field.name):
                    children_to_clone[field.name] |= these_children
                else:
                    children_to_clone[field.name] = these_children
            else:
                pass
        elif field.many_to_one:
            if _new_parent_pk:
                kwargs[field.name + '_id'] = _new_parent_pk
        elif field.concrete:
            kwargs[field.name] = getattr(obj, field.name)
        else:
            pass
    new_instance = obj.__class__(**kwargs)
    new_instance.save()
    new_instance_pk = new_instance.pk
    for ky in children_to_clone.keys():
        child_collection = getattr(new_instance, ky)
        for child in children_to_clone[ky]:
            child_collection.add(duplicate_model_with_descendants(child, whitelist=whitelist, _new_parent_pk=new_instance_pk))
    return new_instance

Example usage:

from django.db import models

class Book(models.Model)
    author = models.ForeignKey('auth.User')

class Chapter(models.Model)
    # author = models.ForeignKey('auth.User')
    book = models.ForeignKey(Book, related_name='chapters')

class Page(models.Model)
    # author = models.ForeignKey('auth.User')
    # book = models.ForeignKey(Book)
    chapter = models.ForeignKey(Chapter, related_name='pages')

WHITELIST = ['books', 'chapters', 'pages']
original_record = models.Book.objects.get(pk=1)
duplicate_record = duplicate_model_with_descendants(original_record, WHITELIST)

I experimented the Stephen G Tuggy’s solution and I found it very clever but, unfortunatly, it won’t work in some special situations.

Let’s suppose the following scenario:

class FattAqp(models.Model):    
    descr = models.CharField('descrizione', max_length=200)
    ef = models.ForeignKey(Esercizio, ...)
    forn = models.ForeignKey(Fornitore, ...)

class Periodo(models.Model):
    #  id usato per identificare i documenti
    # periodo rilevato in fattura
    data_i_p = models.DateField('data inizio', blank=True)
    idfatt = models.ForeignKey(FattAqp, related_name='periodo')

class Lettura(models.Model):
    mc_i = models.DecimalField(max_digits=7, ...)
    faqp = models.ForeignKey(FattAqp, related_name='lettura')
    an_im = models.ForeignKey('cnd.AnagImm', ..)

class DettFAqp(models.Model):
    imponibile = models.DecimalField(...)
    voce = models.ForeignKey(VoceAqp, ...)
    periodo = models.ForeignKey(Periodo, related_name='dettfaqp')

In this case, if we try to deep-copy a FattAqp instance, ef, forn, an_im and voce fields will not correctly set; on the other hand idfatt, faqp, periodo will.

I solved the problem by adding one more parameter to the function and with a slight modification to the code. I tested it with Python 3.6 and Django 2.2
Here is it:

def duplicate_model_with_descendants(obj, whitelist, _new_parent_pk=None, static_fk=None):
    kwargs = {}
    children_to_clone = OrderedDict()
    for field in obj._meta.get_fields():
        if field.name == "id":
            pass
        elif field.one_to_many:
            if field.name in whitelist:
                these_children = list(getattr(obj, field.name).all())

                if field.name in children_to_clone:
                    children_to_clone[field.name] |= these_children
                else:
                    children_to_clone[field.name] = these_children
            else:
                pass
        elif field.many_to_one:
            name_with_id = field.name + '_id'
            if _new_parent_pk:
                kwargs[name_with_id] = _new_parent_pk

            if name_with_id in static_fk:
                kwargs[name_with_id] = getattr(obj, name_with_id)

        elif field.concrete:
            kwargs[field.name] = getattr(obj, field.name)
        else:
            pass
    new_instance = obj.__class__(**kwargs)
    new_instance.save()
    new_instance_pk = new_instance.pk
    for ky in children_to_clone.keys():
        child_collection = getattr(new_instance, ky)
        for child in children_to_clone[ky]:
            child_collection.add(
                duplicate_model_with_descendants(child, whitelist=whitelist, _new_parent_pk=new_instance_pk,static_fk=static_fk))

Example usage:

original_record = FattAqp.objects.get(pk=4)
WHITELIST = ['lettura', 'periodo', 'dettfaqp']
STATIC_FK = ['fornitore_id','ef_id','an_im_id', 'voce_id']
duplicate_record = duplicate_model_with_descendants(original_record, WHITELIST, static_fk=STATIC_FK)

Elaborated based on previous answers:

def derive(obj):
    import copy
    from django.contrib.admin.utils import NestedObjects
    from django.db import DEFAULT_DB_ALIAS
    from django.db.models.fields.related import ForeignKey
    """
        Derive a new model instance from previous one,
        and duplicate all related fields to point to the new instance
    """
    obj2 = copy.copy(obj)
    obj2.pk = None
    obj2.save()
    collector = NestedObjects(using=DEFAULT_DB_ALIAS)
    collector.collect([obj])
    collector.sort()
    related_models = collector.data.keys()
    data_snapshot = {}

    for key in collector.data.keys():
        data_snapshot.update({
            key: dict(
                zip(
                    [item.pk for item in collector.data[key]],
                    [item for item in collector.data[key]]
                )
            )
        })

    duplicate_order = reversed(related_models)

    for model in duplicate_order:
        # Find all FKs on model that point to a related_model.
        fks = []
        for f in model._meta.fields:
            if isinstance(f, ForeignKey) and f.rel.to in related_models:
                fks.append(f)
        # Replace each `sub_obj` with a duplicate.
        if model not in collector.data:
            continue
        sub_objects = collector.data[model]
        for obj in sub_objects:
            for fk in fks:
                dupe_obj = copy.copy(obj)
                setattr(dupe_obj, fk.name, obj2)
                dupe_obj.pk = None
                dupe_obj.save()
    return obj2

There is an option to create a duplicate/clone/save-as-new in django admin.

1. Create a ModelAdmin class of the model you want to clone in admin.py
2. In the class add an admin action like:

 @admin.register(Book)
 class BookAdmin(models.ModelAdmin):
     save_as = True

and this will create a “Save as New” button in your admin panel to completely clone the model object with all it’s related fields.

Suggestion of Julio Marins works! Thnx!

For Django >= 2.* this line:

if isinstance(f, ForeignKey) and f.rel.to in related_models:

Should be replaced with:

if isinstance(f, ForeignKey) and f.remote_field.model in related_models:

django-clone library works perfectly for me with ManyToMany relationships. Just:

1. Make the model you want to clone a subclass of CloneModel:

from django.db import models
from model_clone.models import CloneModel

class MyModel(CloneModel):
    name = models.CharField(max_length=50)
    tags = models.ManyToManyField(Tag)

    # You must specify all the ManyToManyField fields
    _clone_m2m_fields = ['tags']

2. Then just call make_clone method:

obj = MyModel.objects.get(pk=some_pk)
cloned = obj.make_clone()

You can also define specific values for the cloned object. Read the docs for more!