How to store aggregation pipeline in mongoDB?

Question

I am developing a rule engine using MongoDB and Python. I want to store Rules in MongoDB in a Rule database “myrulesdb” and want to use it to run aggregation pipeline on a different collection. Rule saved in MongoDB is

{
"_id" : ObjectId("57f46e843166d426a20d5e08"),
"Rule" : "[{"$match":{"Name.First":"Sunil"}},{"$limit":5}]",
"Description" : "You live long life"
}

Python Code:

import pymongo
from pymongo import MongoClient

client = MongoClient('localhost', 27017)

db = client['myDB']
coll = db['myColl']

ruledb = client['myrulesdb']
rulecoll = ruledb['myrules']

rule1 = rulecoll.find_one({}, {"Rule":1, "_id":0})

pipe = rule1['Rule']
print(pipe)

data = coll.aggregate(pipeline=pipe)

I am getting correct string when i print pipe.

[{"$match":{"Name.First":"Sunil"}},{"$limit":5}]

But when i pass this to aggregate() function, it gives me following error:

data = coll.aggregate(pipeline=pipe)
in aggregate raise TypeError("pipeline must be a list")
TypeError: pipeline must be a list

After retrieving pipeline from database, how can i pass it to aggregate() function?

Asked By: Sunil Jha

||

Source

Answer 1

I don’t know why you are doing this but the culprit here is the value of the Rule field which is string. If you try a little debuging with the print() function your will see that type(pipe) yields <class 'str'>

You can fix this by loading the value using json.loads like this:

>>> import json
>>> r = {"Rule" : "[{"$match":{"Name.First":"Sunil"}},{"$limit":5}]"}
>>> pipe = json.loads(r['Rule'])
>>> pipe
[{'$match': {'Name.First': 'Sunil'}}, {'$limit': 5}]
>>> type(pipe)
<class 'list'>

Answered By: styvane

Answer 2

This is a late answer (but maybe useful for others looking at this question) and only in case you might consider using Java instead of Python. Please disregard my answer if not.
You can use the small free library https://github.com/MongoPipe/mongopipe-core to do the pipeline saving, parameterized running, automatic migration, etc.

Answered By: doncristiano

How to store aggregation pipeline in mongoDB?

Question:

Answers: