logo
Splunk Team

pymongo json_util.dumps overwrite objectid representation

  • Last Update :
  • Techknowledgy :

As a work around, I was able to accomplish the same result with re regular expressions:

import re

def remove_oid(string):
   while True:
   pattern = re.compile('{\s*"\$oid":\s*(\"[a-z0-9]{1,}\")\s*}')
match = re.search(pattern, string)
if match:
   string = string.replace(match.group(0), match.group(1))
else:
   return string

string = json_dumps(mongo_query_result)
string = remove_oid(string)

This is my way of doing it, using only the provided json library

json.dumps(list(products.find({})),
   default = lambda o: str(o))

Suggestion : 2

This is my way of doing it, using only anycodings_bson the provided json library,I looked a bit in the documentation over anycodings_pymongo here, and here are the findings:,Is there something I am missing to convert anycodings_pymongo PyMongo query result to:,The problem with that is it doesn't match anycodings_pymongo the results I get in prod env. I am using anycodings_pymongo PyMongo just to build test cases, the actual anycodings_pymongo prod format is

bson.json_util provides functions to convert anycodings_pymongo to either canonical or relaxed JSON format. anycodings_pymongo However, both of them stick to the same anycodings_pymongo representation of the ObjectId:

from PyMongo
import MongoClient
from bson.objectid
import ObjectId
from bson
import json_util
from bson.json_util
import RELAXED_JSON_OPTIONS
from bson.json_util
import CANONICAL_JSON_OPTIONS, DEFAULT_JSON_OPTIONS

db = MongoClient(URL)['DB_NAME']
mongo_query_result = db.collection.find_one({
   '_id': ObjectId('ID')
}, {
   '_id': 1
})

# returns {
   '_id': ObjectId('ID')
}

print(json_util.dumps(mongo_query_result, json_options = RELAXED_JSON_OPTIONS))
print(json_util.dumps(mongo_query_result, json_options = CANONICAL_JSON_OPTIONS))
print(json_util.dumps(mongo_query_result, json_options = DEFAULT_JSON_OPTIONS))

# Results {
   "_id": {
      "$oid": "ID"
   }
} {
   "_id": {
      "$oid": "ID"
   }
} {
   "_id": {
      "$oid": "ID"
   }
}

# Desired Output {
   "_id": "ID"
}

The problem with that is it doesn't match anycodings_pymongo the results I get in prod env. I am using anycodings_pymongo PyMongo just to build test cases, the actual anycodings_pymongo prod format is

{
   '_id': "ID",
   ..etc
}

Is there something I am missing to convert anycodings_pymongo PyMongo query result to:

{
   '_id': 'ID',
   ..
}
# not {
   '_id': {
      '$oid': 'ID'
   },
   ..
}

This is my way of doing it, using only anycodings_bson the provided json library

json.dumps(list(products.find({})),
   default = lambda o: str(o))

As a work around, I was able to anycodings_bson accomplish the same result with re anycodings_bson regular expressions:

import re

def remove_oid(string):
   while True:
   pattern = re.compile('{\s*"\$oid":\s*(\"[a-z0-9]{1,}\")\s*}')
match = re.search(pattern, string)
if match:
   string = string.replace(match.group(0), match.group(1))
else:
   return string

string = json_dumps(mongo_query_result)
string = remove_oid(string)

Suggestion : 3

This essentially replaces the CANONICAL_JSON to normalized one and remove the key-value to just a value.,Although this gets the job done, it is not ideal, since I am manipulating JSON as a string, and very much error prone, plus doesn't work in Date or other format.,Connect to remote mongo server works fine but when connecting through Go, it gives err: context deadline exceeded,Best way to develop this function in react js instead of this current .map

As a work around, I was able to accomplish the same result with re regular expressions:

import re

def remove_oid(string):
   while True:
   pattern = re.compile('{\s*"\$oid":\s*(\"[a-z0-9]{1,}\")\s*}')
match = re.search(pattern, string)
if match:
   string = string.replace(match.group(0), match.group(1))
else:
   return string

string = json_dumps(mongo_query_result)
string = remove_oid(string)

This is my way of doing it, using only the provided json library

json.dumps(list(products.find({})),
   default = lambda o: str(o))

Suggestion : 4

This module provides two helper methods dumps and loads that wrap the native json methods and provide explicit BSON conversion to and from JSON. JSONOptions provides a way to control how JSON is emitted and parsed, with the default being the Relaxed Extended JSON format. json_util can also generate Canonical or legacy Extended JSON when CANONICAL_JSON_OPTIONS or LEGACY_JSON_OPTIONS is provided, respectively.,In this mode, dumps() produces Canonical Extended JSON, a type preserving format. Consider using this for things like testing, where one has to precisely specify expected types in JSON. In particular, the int, Int64, and float numeric types are encoded with type wrappers.,Alternatively, you can manually pass the default to json.dumps(). It won’t handle Binary and Code instances (as they are extended strings you can’t provide custom defaults), but it will be faster as there is less recursion.,If your application does not need the flexibility offered by JSONOptions and spends a large amount of time in the json_util module, look to python-bsonjs for a nice performance improvement. python-bsonjs is a fast BSON to MongoDB Extended JSON converter for Python built on top of libbson. python-bsonjs works best with PyMongo when using RawBSONDocument.

>>> from bson.json_util
import loads
   >>>
   loads('[{"foo": [1, 2]}, {"bar": {"hello": "world"}}, {"code": {"$scope": {}, "$code": "function x() { return 1; }"}}, {"bin": {"$type": "80", "$binary": "AQIDBA=="}}]')[{
      'foo': [1, 2]
   }, {
      'bar': {
         'hello': 'world'
      }
   }, {
      'code': Code('function x() { return 1; }', {})
   }, {
      'bin': Binary(b '...', 128)
   }]
>>> from bson
import Binary, Code
   >>>
   from bson.json_util
import dumps
   >>>
   dumps([{
         'foo': [1, 2]
      },
      ...{
         'bar': {
            'hello': 'world'
         }
      },
      ...{
         'code': Code("function x() { return 1; }")
      },
      ...{
         'bin': Binary(b "")
      }
   ])
'[{"foo": [1, 2]}, {"bar": {"hello": "world"}}, {"code": {"$code": "function x() { return 1; }"}}, {"bin": {"$binary": {"base64": "AQIDBA==", "subType": "00"}}}]'
>>> from bson
import Binary, Code
   >>>
   from bson.json_util
import dumps, CANONICAL_JSON_OPTIONS
   >>>
   dumps([{
            'foo': [1, 2]
         },
         ...{
            'bar': {
               'hello': 'world'
            }
         },
         ...{
            'code': Code("function x() { return 1; }")
         },
         ...{
            'bin': Binary(b "")
         }
      ],
      ...json_options = CANONICAL_JSON_OPTIONS)
'[{"foo": [{"$numberInt": "1"}, {"$numberInt": "2"}]}, {"bar": {"hello": "world"}}, {"code": {"$code": "function x() { return 1; }"}}, {"bin": {"$binary": {"base64": "AQIDBA==", "subType": "00"}}}]'
>>> from bson
import Binary, Code
   >>>
   from bson.json_util
import dumps, LEGACY_JSON_OPTIONS
   >>>
   dumps([{
            'foo': [1, 2]
         },
         ...{
            'bar': {
               'hello': 'world'
            }
         },
         ...{
            'code': Code("function x() { return 1; }", {})
         },
         ...{
            'bin': Binary(b "")
         }
      ],
      ...json_options = LEGACY_JSON_OPTIONS)
'[{"foo": [1, 2]}, {"bar": {"hello": "world"}}, {"code": {"$code": "function x() { return 1; }", "$scope": {}}}, {"bin": {"$binary": "AQIDBA==", "$type": "00"}}]'
>>> from bson.json_util
import CANONICAL_JSON_OPTIONS
   >>>
   CANONICAL_JSON_OPTIONS.tz_aware
True
   >>>
   json_options = CANONICAL_JSON_OPTIONS.with_options(tz_aware = False, tzinfo = None) >>>
   json_options.tz_aware
False

Suggestion : 5

json_util provides two helper methods, dumps and loads, that wrap the native json methods and provide explicit BSON conversion to and from json.,python-bsonjs does not depend on PyMongo and can offer a nice performance improvement over json_util:,If all you need is serializing mongo results into json, it is possible to use the json module, provided you define custom handlers to deal with non-serializable fields types. One advantage is that you have full power on how you encode specific fields, like the datetime representation.,Here is a handler which encodes dates using the iso representation and the id as an hexadecimal string:

Simple usage

from bson.json_util
import loads, dumps
record = db.movies.find_one()
json_str = dumps(record)
record2 = loads(json_str)

if record is:

{
   "_id": ObjectId("5692a15524de1e0ce2dfcfa3"),
   "title": "Toy Story 4",
   "released": ISODate("2010-06-18T04:00:00Z")
}

then json_str becomes:

{
   "_id": {
      "$oid": "5692a15524de1e0ce2dfcfa3"
   },
   "title": "Toy Story 4",
   "released": {
      "$date": 1276833600000
   }
}

modify the DEFAULT_JSON_OPTIONS object. In this case, the options will be used for all subsequent call to dumps:

 from bson.json_util
 import DEFAULT_JSON_OPTIONS
 DEFAULT_JSON_OPTIONS.datetime_representation = 2
 dumps(record)

specify a JSONOptions in a call to dumps using the json_options parameter:

 # using strict
 dumps(record, json_options = bson.json_util.STRICT_JSON_OPTIONS)

 # using a custom set of options
 from bson.json_util
 import JSONOptions
 options = JSONOptions() # options is a copy of DEFAULT_JSON_OPTIONS
 options.datetime_representation = 2
 dumps(record, json_options = options)

modify the DEFAULT_JSON_OPTIONS object. In this case, the options will be used for all subsequent call to dumps:

 from bson.json_util
 import DEFAULT_JSON_OPTIONS
 DEFAULT_JSON_OPTIONS.datetime_representation = 2
 dumps(record)

specify a JSONOptions in a call to dumps using the json_options parameter:

 # using strict
 dumps(record, json_options = bson.json_util.STRICT_JSON_OPTIONS)

 # using a custom set of options
 from bson.json_util
 import JSONOptions
 options = JSONOptions() # options is a copy of DEFAULT_JSON_OPTIONS
 options.datetime_representation = 2
 dumps(record, json_options = options)

Suggestion : 6

json_util provides two helper methods, dumps and loads, that wrap the native json methods and provide explicit BSON conversion to and from json., modify the DEFAULT_JSON_OPTIONS object. In this case, the options will be used for all subsequent call to dumps: from bson.json_util import DEFAULT_JSON_OPTIONS DEFAULT_JSON_OPTIONS.datetime_representation = 2 dumps(record) , specify a JSONOptions in a call to dumps using the json_options parameter: # using strict dumps(record, json_options=bson.json_util.STRICT_JSON_OPTIONS) # using a custom set of options from bson.json_util import JSONOptions options = JSONOptions() # options is a copy of DEFAULT_JSON_OPTIONS options.datetime_representation=2 dumps(record, json_options=options) ,document_class: BSON documents returned by loads() will be decoded to an instance of this class. Must be a subclass of collections.MutableMapping. Defaults to dict.

Simple usage

from bson.json_util
import loads, dumps
record = db.movies.find_one()
json_str = dumps(record)
record2 = loads(json_str)

if record is:

{
   "_id": ObjectId("5692a15524de1e0ce2dfcfa3"),
   "title": "Toy Story 4",
   "released": ISODate("2010-06-18T04:00:00Z")
}

then json_str becomes:

{
   "_id": {
      "$oid": "5692a15524de1e0ce2dfcfa3"
   },
   "title": "Toy Story 4",
   "released": {
      "$date": 1276833600000
   }
}

modify the DEFAULT_JSON_OPTIONS object. In this case, the options will be used for all subsequent call to dumps:

 from bson.json_util
 import DEFAULT_JSON_OPTIONS
 DEFAULT_JSON_OPTIONS.datetime_representation = 2
 dumps(record)

specify a JSONOptions in a call to dumps using the json_options parameter:

 # using strict
 dumps(record, json_options = bson.json_util.STRICT_JSON_OPTIONS)

 # using a custom set of options
 from bson.json_util
 import JSONOptions
 options = JSONOptions() # options is a copy of DEFAULT_JSON_OPTIONS
 options.datetime_representation = 2
 dumps(record, json_options = options)

modify the DEFAULT_JSON_OPTIONS object. In this case, the options will be used for all subsequent call to dumps:

 from bson.json_util
 import DEFAULT_JSON_OPTIONS
 DEFAULT_JSON_OPTIONS.datetime_representation = 2
 dumps(record)

specify a JSONOptions in a call to dumps using the json_options parameter:

 # using strict
 dumps(record, json_options = bson.json_util.STRICT_JSON_OPTIONS)

 # using a custom set of options
 from bson.json_util
 import JSONOptions
 options = JSONOptions() # options is a copy of DEFAULT_JSON_OPTIONS
 options.datetime_representation = 2
 dumps(record, json_options = options)

Suggestion : 7

I'm using pymongo to seed a database with old information from a different system, and I have a lot of queries like this:, 4 days ago MongoDB: insert documents with specific id instead of auto generated ObjectID; Flask PyMongo string back to ObjectID; Configure pymongo to use string _id instead of ObjectId; … , 2 days ago Flask PyMongo string back to ObjectID; Configure pymongo to use string _id instead of ObjectId; Flask Pymongo Objectid link not working; Convert string back to ObjectID; ObjectId … ,Any objects I create with the Meteor specification will use the string format, and not the ObjectId format. I don't want to have mixing of id types in my application, because it's causing me interoperability headaches.


studentId = studentsRemote.insert({
   'price': price
})

from pymongo.son_manipulator
import SONManipulator class ObjectIdManipulator(SONManipulator): def transform_incoming(self, son, collection): son[u '_id'] = str(son[u '_id']) return son db.add_son_manipulator(ObjectIdManipulator())


Trending Technology


Most popular in python