wiki:Plugins/mongodb-log
Last modified 5 years ago Last modified on 21.08.2013 14:34:54

MongoDB Logging Plugin

This plugin implements a generic logger that stores all blackboard data, images, and point clouds to a MongoDB database. The concepts are explained in the paper "A Generic Robot Database and its Application in Fault Analysis and Performance Evaluation". Also confer the MongoDB logging project page for more general information and example graphs.

Requires

BlackBoard Interfaces

No particular interface is required, but all that exist can be logged.

Config Values

Currently the plugin will immediately start to log everything to the database.

PathTypeDescriptionDefaultR
/plugins/mongolog/databasestringDatabase configured in mongodb? plugin to use for loggingfflog*
/plugins/mongorrd/databases/mongologstringSet automatically when starting to allow for RRD graph generation.AUTO
/plugins/mongolog/enable_loggerboolEnable recording of text log messages to the database?true

Provides

BlackBoard Interfaces

None.

Other

Database containing the logged data, by default is is named fflog.

Usage Instructions

This assumes a working setup of MongoDB and the mongodb plugin?. Afterwards simply load the plugin and recording will start instantly. Note that for now it only checks for images and point clouds once on startup. Data of plugins loaded after mongodb-log producing such data will not be logged.

Database details

Transforms

Transforms are grouped together in chunks of several seconds to reduce the number of insert operations on the database. On a PR2, for example, an average of more than 3500 transforms is published per second. Storing each and every transform in a separate document can quickly lead to congestion (on a single-host database, in particular with a spinning hard drive). Therefore, the plugin periodically takes the transform listener and stores a certain period of time with one document per transform link (source to target frame). By default, this interval is two seconds. In case of the PR2, this decreases the average number of inserts per second to about 50.

Database Queries

This section contains some example database queries which have been useful for evaluation etc.

Query averages on transforms frequencies

The following queries gathers data about the number of transform and transform documents per second. The distinction results from the grouping of transforms into documents explained above. The results are written to new collections with one entry for each second at which transforms were stored and as value the number of transform (documents) stored in that very second.

Note that the examples always operate on all the available data, i.e. it assumes that only a single run or all runs of interest are current stored in the database. Add a query to limit the time range if required.

Transform documents per second

// this assumes you are operating on the recording database, fflog by default
db.tf.mapReduce(
  // map
  function() {
    emit(Math.floor(this.timestamp_to / 1000), {count: 1});
  },
  // reduce
  function(key, values) {
    rv = { count: 0};
    for (var i = 0; i < values.length; ++i) {
      rv.count += values[i].count;
    };
    return rv;
  },
  // output document
  {out: { replace: "tfmr"}})

Transforms per second

// this assumes you are operating on the recording database, fflog by default
db.tf.mapReduce(
  // map
  function() {
    for (var i = 0; i < this.transforms.length; ++i) {
      emit(Math.floor(this.transforms[i].timestamp / 1000), {count: 1});
    }
  },
  // reduce
  function(key, values) {
    rv = { count: 0};
    for (var i = 0; i < values.length; ++i) {
      rv.count += values[i].count;
    };
    return rv;
  },
  // output document
  {out: { replace: "tfmr"}})

Average number

This aggregate query gives the overall average for the whole data currently in the database. This can be merged into the above map-reduce queries above, but we kept it separate for simplicity. The meaning of the returned value (average number of transforms or transform documents) depends only on the input collection. Adjust it as required (replace tfmr).

// this assumes you are operating on the recording database, fflog by default
db.tfmr.group({
  initial: {count:0, total: 0},
  reduce: function(doc,out) {
    out.count += 1;
    out.total += doc.value.count;
  },
  finalize: function(out) {
    out.avg = out.total / out.count;
  }
})