Blog
OpenEnergyMonitor

Preparing emoncms.org feeds for conversion to timestore

One of the central principles behind the Timestore data storage approach is that datapoints are stored at a fixed interval. Because you know that there is a datapoint every 10s you dont need to store the timestamp for each datapoint, you only need to store the timestamp for the first datapoint. The timestamp for every other datapoint can be worked out i.e:

timestamp = start + position * interval.

Storing time series data in this way makes it really easy and very fast to query. The tests so far have shown timestore to be several magnitudes faster while also using significantly less disk space.

If your logging to emoncms.org there is now an interface that can be used to select the interval rate that you would like your feeds converted too, its linked from the feeds page.

On the feed/convert page:

The new interval column states the average interval rate of the existing feed and is calculated simply as the end time minus the start time divided by the number of datapoints in the feed. This interval rate can be skewed if the monitor dropped off for a period, so its worth double checking that it is correct.

You may wish to change your interval rate, if your logging temperature data at 5s intervals and 60s is enough to see the changes in temperature you want to see then select 60s as this reduces the disk use of the feed considerably.

To set the interval rate you wish your feed to be converted to, click on the pencil button to bring up in-line editing:

Click on the drop down menu under convert to and select from the list the interval you wish to use. Click on the tick button to complete.

Repeat the above steps for every feed you wish to convert and then once your done click on Add feeds to conversion queue button that is at the bottom of the conversion page.

As mentioned in the last post, its going to take up to a month for the server to convert through all the feeds so you wont see anything straight away. I will update with conversion progress.

I've put all the code for the above on github under an emoncmsorg branch of emoncms, the conversion scripts can also be found in the repo usefulscripts To engage in discussion regarding this post, please post on our Community Forum.