From 1.3 minutes to 196ms: timestore on emoncms.org


John Cantor who uses the openenergymonitor system for heatpump performance monitoring got in contact the other day saying that graphs on emoncms.org where taking a really long time to load. I said that the work on using timestore will soon speed things up significantly. He asked if there was a chance to convert the feeds before he went up to work on a system in Scotland as being able to zoom with speed through the graphs would make a big difference.

Clicking on the Y (Year) button in the rawdata visualisation has slowed down so much over the last year that it now takes over a minute to load the view.


The same request on a raspberrypi takes 2.9 seconds, The thinking to why its so much slower on emoncms.org is because of the use of the php loop doing individual requests to mysql which are added to the mysql process queue and have to wait in line amongst all the other mysql inserts and updates caused by all the updating feeds, the php loop method was initially faster than an equivalent mysql implementation but clearly not for long.

Anyway I figured we could get timestore up and running on emoncms.org straight away if we run timestore in parallel with the current mysql storage and then slowly convert feeds over to timestore over time. The present input processing and visualisation implementation would be kept as is for the time being only initially using timestore to replace the realtime datatype rather than both realtime and daily.

Conversion from mysql to timestore takes a long time especially as the process has to be throttled down in order to allow enough resources on the server for normal operation to continue. For example the conversion of a 568 day long, 5s datapoint interval feed (~152mb) takes about half an hour loading the server to an average 15min (quite high) load of 4.69 at that rate it will take about a month to convert all feeds over to timestore.

The results though are worth it!:


196ms!

1.3 minutes down to 196ms is a 398x speed increase! and its also possible to see another advantage of timestore the effect of the average layers showing a slight winter summer curve. A big thankyou and credit to Mike Stirling for developing timestore enabling this kind of performance, if you haven't already check out the timestore project website:


John Cantor has shared the Scottish heatpump dashboard publicly, take a look to try out the new timestore feeds:


Im currently working on an interface in emoncms to allow selection of the feed datapoint interval (5s, 10s, 30s etc) to convert to before the feeds are added to the conversion queue, more on this soon.

OEM Gateway by Jerome Lafréchoux

I would like to highlight the good and interesting work that Jerome Lafréchoux has been doing on developing a gateway that is intended to go beyond the current RFM2Pi interface towards something that can be used on any linux system whether a pi or any computer to take say input from a serial RFM12Pi and forward it either to local instances of emoncms or remote instances of any service (without necessarily needing a local install of emoncms).

Here's an exerpt from Jerome's last post on the github topic on gateway development:
"The standalone gateway seems complete enough for me to be published, although I'd rather have a little feedback before recommending it.
It works basically the same way. Here are the main features:
  • Can be parameterized either via emoncms GUI (with current limitations of the interface) or through a configuration file. Useful for those who don't want to install a local emoncms just for the configuration of their gateway. This also allows to add features without the need for complex GUI edition.
  • Can read inputs from the RFM2Pi, but also inputs from the serial port of the form "NodeID val1 val2 ...", or even from a socket. This socket possibility means any application on the same machine or on the network can send data to the gateway. This even solves the inter-applications issue. It is better than writing to a file, in my opinion.
  • Extensibility. By design, it should be easy to create new inputs (data sources) or outputs (another server's API). It uses classes, so a new input (listener) or output (buffer) can be created through inheritance/overriding.
See readme file for more explanations.
I should add a socket use sample, but it's pretty basic and require very few code. Here's a python example:
import socket
HOST = 'raspberrypi' # The remote host, can be 'localhost' just the same
PORT = 50011 # The port chosen in the gateway config
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.send('14 15 26\r\n')
s.close() 
Currently, each buffer instantiated in the gateway buffers a given amount of data when network is down. This is limited (we don't want to eat up all memory). Next step could be to add the possibility to buffer samples in a file (with care for SD card wearout)."

To try it out see the github repository here: https://github.com/Jerome-github/oem_gateway. Jerome is looking for testers and would appreciate feedback on it.

EmonTx Shield SMT progress

The latest emontx shield SMT board revision (v2.4) arrived a couple of days ago.

This revision carries on from the developments described in the previous blog post here: http://openenergymonitor.blogspot.co.uk/2013/06/emontx-shield-smt-with-new-layout-in.html

The improvements in v2.4 are:
  • The DS18B20 connector is bought out to the edge with screw terminals.
  • There's an FTDI connector as the shield covers the FTDI connector on the NanodeRF.
  • The jumpers are 2.54mm headers rather than surface pads that are soldered, making them easier to change.
  • The RFM12 footprint is moved further in on the board so that there is space for the Ciseco SRF with its on board chip antenna


There was one small but crucial mistake, I got the RX and TX on the FTDI connector the wrong way around!!

Next step: switch the RX and TX around and send the board off for production.

DIY Casing

We are working on new designs which which will be better designed to fit into cases in the meantime John Cantor  (from heatpumps.co.uk) who has been using a modified OpenEnergyMonitor system for monitoring the performance of air and ground source heatpumps shares a photo of his experiences of boxing up the emonTx and Nanode RF SMT

emonTx setup for temperature monitoring and nanode RF SMT cased up using standard RS cases

These cases from RS were used:

ABS moulded box, 120x65x40mm, ivory
RS Stock No.502-641
ABS moulded box, 112x62x31mm, ivory
RS Stock No.502-635
both £3.45+vat.


Ciseco SRF and RFM12B Power Consumption Investigation

Following on yesterdays post on the Hope RF RFM12B power consumption I decided to do a comparison with the Ciseco SRF Radio. An RFu328 (miniature ATmega328 in XBEE footprint) was used to mount and interface with the two radios. The same 3.3V power supply was used with both modules

Ciseco RFu328 with SRF and RFM12B

The scope was connected up to measure the voltage drop across a shunt resistor as follows:

Oscilloscope probe measuring voltage drop across a 10R series resistor

Hope RF RFM12B

Here is the RFM12B current consumption trace while sending 4 integers using the JeeLib packet structure. Using this packet structure each integer takes up 2 bytes, therefore 4 integers is 8 bytes plus 1 byte containing the node ID, this gives a total packet size of 9 bytes. Transmission takes 2.7ms and the current consumption in the time is about 24mA @ 3.3V. This gives a power and energy consumption of 24mA * 3.3V = 79.2mW * 2.7mS = 0.214mJ = 214uJ


Ciseco SRF


A SRF V1.0a with serial firmware was used for this test.

The SRF is serial based. Ciseco have standardized on a communication structure called LLAP (Lightweight Local Automation Protocol)

A LLAP packet consists of one start byte 'a' , two bytes for the node ID then 9 bytes for the message. Encoding as HEX each LLAP packet can give us space for two integers. Each integer has a range of -32767 to 32767 which is fine for our standard emonTx setup which has a maximum reading of 25000W (100A x 250Vrms).

To transmit four integers from the emonTx (3 x power and 1 x voltage) would require two LLAP packets which each contains 12 char characters which gives a packet size of 12 bytes transmitted twice giving a total of 24 bytes.

Here is a current capture waveform of the SRF transmitting two LLAP packets, it's rather more interesting than the RFM12B, I would love to know exactly what the SRF is doing at each spike and dip.


Average power consumption of 20.8mA


Transmission of two LLAP packets takes 15ms

Transmission of two LLAP packets takes 15ms with an average current of consumption of 20.8mA. This gives and power and energy consumption of 20.8mA * 3.3V = 68.6mW * 15ms =1mJ.

This is 4.7 times more energy than the RFM1B for the transmission of the same four integers. This is mainly due to the efficient nature of the JeeLib packet structure sending the integers as binary rather than serial characters as in the case of the SRF. Transmitting four integers as HEX characters in two LLAP packets takes 24 bytes as opposed to the 9 bytes needed for the same four integers in the RFM12B JeeLib packet structure. Taking this into account the SRF consumes 41uJ per byte where the RFM12B consumes 23uJ per byte, this is around 1.8 times more power byte for byte than the RFM12B.


SRF startup 50mA spike

An interesting observation is that the SRF exhibits a rather high current spike of about 50mA as it's turned on / comes out of sleep. As this spike only lasts for only about 100nS it won't contribute that much to the overall power consumption. 

Energy Consumed While Sleeping

The energy used the the RF modules needs to be put in perspective with the overall consumption of the system. An emonTx running on batteries or low power temperate node will spend much of it's time sleeping, the ATmega328 consumes 4.3uA when sleeping and the SRF and RFM12B consume about the same when sleeping 0.2-0.3uA, giving an overall sleep mode power draw of 4.6uA or 0.0046mA.

Sleeping for 10s

Assuming a the case of a wireless node which sleeps for 10s in between readings. This gives a energy consumption of 0.0046mA * 3.3V = 0.0152mW * 10s = 0.152mJ = 152uJ.

If this node was using an RFM1B 1.4 times more energy would be consumed in the 3ms that the RFM12B is active while transmitting the data via RF then in the proceeding 10s when the node is sleeping

If the temperature node was using an SRF 6.6 times more energy would be consumed in the 15ms that the SRF is active while transmitting the data via RF then in the proceeding 10s when the node is sleeping.

Sleeping for 10 min

Assuming a the case of a wireless node which sleeps for 10s in between readings. This gives a energy consumption of 0.0046mA * 3.3V = 0.0152mW * (60s *10) =  9.13mJ.

The energy consumed while sleeping now becomes the greatest consumer. The energy consumed during sleeping for 10s is 43 times greater than the energy required by the RFM12B to transmit the data or 9 times greater than the energy required by the SRF to transmit the data.


Conclusion

If a ATmega328 based 'sleepy' node sleeps for 14s or more the energy used during sleeping will equal or greater the energy used by the RFM12B (to transmit four integers). If the nodes sleeps for 1 min or more the energy used during sleeping will equal or greater then energy used by the SRF (using serial LLAP to transmit four integers).

LLAP serial on the SRF not the most power efficient way to transmit integers compared to the RFM12B using the JeeLib packet structure. Power consumption of the SRF can be reduced at the expense of human readability of the data packets. I plan to investigate this further, see questions to answer below:

Questions to answer:

Does the extra energy consumed by the SRF result in increased range over the RFM1B?

The SRF by default is set at 10dBm transmission power (compared to 0dBM for RFM12B), this can be reduced all the way down to -30dBm in various increments, how much will this reduce energy consumption and range? Is there a sweet spot? The RFM12B transmits at 0dBm, how will the range of the SRF transmitting at 0dBM compare to the SRF? 

The SRF currently transmits at 9600 baud rate, this can be increased to 115200, will this reduce the time taken to complete a transmission and therefore energy used. How much will this effect loss of packets and range? Ciseco SRF setup documentation. 

Is it possible to interface directly with the SRF to transmit the raw packets not using serial?

Can power consumption of SRF be improved with new firmware?

I hear it's possible to use the CC chip on the SRF to offload the WDT to wake up the ATmega328 using a hardware interrupt, this could result in sleep current draw of around 0.3uA. I'm keen to investigate this.

New Oscilloscope - RFM12B Power Consumption

We have finally taken the plunge and have upgraded our measurement facilitates.

The main (and most costly!) acquisition has been a Rigol DS2072 70Mhz digital scope.



We also upgraded our old cheapo multimeter to a more accurate model which as well as the usual multimeter functions can measure AC power and frequency and has a USB link (Uni-T UT71E) and to complete the setup a variable DC voltage and current bench top power supply (Rapid HY3003D).

Test setup

Uni-T UT71E True RMS 40K count multimeter

I thought an interesting investigation and a good starting point for getting to grips with the scope would be to measure the power consumption during an RFM12B transmission.

With the standard passive probes the scope only measures voltage so a 10R 0.25W shunt resistor was used in series and the voltage drop measured over this. A 10R resistor works nicely since the voltage drop can easily be converted into current by dividing by 10 which is done by the probe which is 10:1 passive probe. Any other value could be used and the conversion to current done in software on the scope. The units for current (A) can be set on the scope. The shut resistor must be connected on the GND side of the circuit as to make one side of the resistor 0V or else use differential probes (or two standard probes with some clever settings on the scope).

Scope probe connection, measuring voltage drop across a 10R resistor

The maximum current which a shunt resistor of a particular power rating can handle can be calculated with Imax = sqrt( Pmax / R).

A 0.25W 1R shunt resistor can handle up to 500mA
A 0.25W 10R resistor can handle up to 158mA
A 0.25W 100R resistor can handle up to 50mA

All measurements were taken with a 3.3V supply voltage using and RFu328 on a prototype emonTx V3.


These traces show the ATmega328 coming out of sleep then transmitting 4 integers. Transmission of 4 integers takes 3ms during which the current draw alternates between 23-28mA for 3ms.   

Testing the scope's capabilities using the waveform zoom function
The built in measurement functions on the scope are pretty amazing, these of the sort of measurements it's possible to obtain for a particular signal. 

 
We intend to eventually setup a proper AC power test rig. In the meantime I intend to continue playing with more of the scopes functions, I've only scratched the surface so far. I'm particularly interested in hooking it up to my laptop for further data processing. The scope has both USB and Ethernet connectivity and an open standard protocol. As well as the software bundled with the scope which look pretty decent but is only for windows a quick google shows that much of the work has already been done to import the data from the scope on Linux.

The EEVblog review of the Rigol scope give more of an idea of it's full capability: 


http://www.eevblog.com/forum/testgear/first-impressions-and-review-of-the-rigol-ds2072-ds2000-series-dso/

http://www.eevblog.com/forum/projects/software-tips-and-tricks-for-rigol-ds200040006000-ultravision-dsos/
I hear it's also possible to perform a light software hack to convert the scope to 100Mhz :-)  http://hackaday.com/2013/07/02/unlocking-a-rigol-scope-once-again/




Emoncms powered by timestore

I've been making some good progress with a timestore powered version of emoncms, its available on github here: https://github.com/emoncms/emoncms/tree/timestore. The basics now work, its not ready for stable switchover, but if your interested in experimenting, its good to go for trying out. There are instructions for installation in the github repo readme.

To recap: timestore is time-series database designed specifically for time-series data developed by Mike Stirling.


Faster Query speeds
With timestore feed data query requests are about 10x faster (2700ms using mysql vs 210ms using timestore). The initial benchmarks I mentioned in previous blog posts show timestore request time to be around 45ms so I still need to investigate the slightly slower performance which may be on the emoncms end rather than timestore.

Reduced Disk use
Disk use is much smaller, A test feed stored in an indexed mysql table used 170mb, stored using timestore which does not need an index and is based on a fixed time interval the same feed used 42mb of disk space.

In-built averaging
Timestore also has an additional benefit of using averaged layers which ensures that requested data is representative of the window of time each datapoint covers.


Due to the integration of averaging in timestore there are significant implications for input processing and visualisations. Rather than compute kwh/d data from power data in an emoncms input process this can now be moved to timestore by querying timestore for average power in a day and converting this value to kwh/d by multiplying the average power by 0.024.

This means we no longer have two feeds one for kwh/d data and another for power data which changes the way many of the visualisations need to work. Visualisations such as simple zoomer need to query the same feed at different average levels.

To fully integrate timestore in emoncms and reach a similar level of input process and visualisation options is a big task. To simplify things, the current release has cut down functionality with only a few input processes and visualisations.

To create feeds with a fixed datapoint interval I added an interval selector to the input processing configure input page:

The data point interval is displayed as a field in feeds page:


Example of timestore working displaying solar production here over the last few hours:

Try it out following the instructions here for installation on a raspberrypi:

Next I intend to test the histogram datatype, ensure that works ok. The input processors for plus, minus, divide and multiply by another input also need looking at and then the more complex kwhd to power feed type zoomer visualisations. If a timestore version of emoncms is useful for you and your interested in helping to get it working well, I would be glad to have your help.


Great emoncms dashboards by Paul Reed, Jürgen and Tom + update on backup

There's a new forum on the forums for showcasing emoncms dashboard creations (thanks to Paul Reed for the idea), here are a few screenshots of dashboards created by Paul Reed and Jürgen:

I really like the additions of the icons, you can do this by inserting html in the text widget.


Some really nice looking multigraph data, forum post by Jürgen: http://openenergymonitor.org/emon/node/2557

Feel free to share your dashboard screenshots or links on the emoncms showcase forum:

Emoncms backup
A couple of additions to the script linked to in the last blog post: http://openenergymonitor.blogspot.com/2013/07/a-new-backup-system-for-emoncmsorg.html

Completes file backup correctly and better verbose output:

To run the backup script from a service, there's a service script available here:

To install the service script:
sudo cp /home/username/backup/emonbackup /etc/init.d/ 
sudo chmod 755 /etc/init.d/emonbackup 
sudo update-rc.d emonbackup defaults

Start the service:
sudo /etc/init.d/emonbackup start log

A new backup system for emoncms.org

The current emoncms.org backup system works by using two separate servers with data being synced from the main one to the second backup server using the same implementation used in the emoncms sync module http://openenergymonitor.blogspot.co.uk/2013/05/emoncmsorg-backup.html

A couple of weeks after getting all that setup BigV announced a new feature: archive storage which they say is ideal for backups. Archive storage works in much the same way as connecting a second external drive to a computer. Archive storage on bigv is guaranteed to be on a separate storage pool from the main vm disc which is good news as it wasn't guaranteed that the two separate server method used seperate storage pools (If I understand correctly)

Another advantage to the new archive storage is that its much cheaper to run, costing £2/month for 50GB rather than £16/month for another vm and 30GB extra space.

A simple php script is used to perform the backup, it makes use of the direct file access stuff I recently learnt about (http://openenergymonitor.blogspot.co.uk/2013/07/more-direct-file-storage-research.html) to do incremental backup file copy using php file access commands making it potentially very fast, it can backup files at full file copy speeds:

It accesses the mysql data directly, copying the content of the mysql feed data file i.e:  /var/lib/mysql/emoncms/feed_1.MYD to the backup drive.

On another slightly related note the data in feed_1.MYD is stored simply as:

1 byte for null flag
4 bytes for the timestamp
4 bytes for the float data value
repeated...

and because the readings are inserted one after the other in ascending time we can actually use the mysql feed data files directly with the direct file get_feed_data method to get 10-20x query speed improvement for generating visualisations: https://github.com/emoncms/experimental/blob/master/storage/directfiles/get_feed_data.php

We could even get rid of the mysql feed table index's to save disk space although that will probably slow down mysql updates (I need to look into it). This could be a short term measure before a full timestore emoncms implementation is complete which provides many other benefits.

Probably the best student placement opportunity ever!? developing an open energy management system at the Center for Alternative Technology

If your a student, looking for a student placement and interested in open source hardware and software, energy monitoring and sustainability this might be of real interest to you. The Center for Alternative Technology are offering an exciting student placement developing an open source energy control and management system for the CAT site, intended to ultimately replace two large proprietary Building Management/SCADA systems that will help them use the site more effectively as a living laboratory for sustainable energy systems.



Here's the full project description from Adam Taylor at CAT:

This placement will assist with the initial development and demonstration of a wireless open source energy management and monitoring system

Project description and person specification

Why?
At the centre we have two proprietary Building Management/SCADA systems that are used for monitoring and heating control; for historical reasons the two systems are not linked in anyway, leading to operational problems. Also as the site is effectively a living laboratory, constantly evolving in its use and layout, the proprietary nature of the systems lead to areas not being integrated or lead to expensive integrations.

Because of the lack of complete site wide monitoring, our knowledge of energy usage is restricted to a site wide level, not the office level that would be useful. Also because of the various none linked systems, heating operation is down to a member of staff switching it on appropriately early to warm the room up, and remembering to switch it off after the room has been finished with. With a detailed understanding of how much energy is actually being used, in comparison with our site energy model, as well as being able to automate the operation of the heating, has the potential to save CAT significant amounts of carbon, time and money.

Once the initial development and demonstration of the system Is complete, the project can then be opened up to the wider open source community, for further development. The demonstration system would also be expanded to cover the whole site, eventually replacing the two existing proprietary systems.

What?
Due to the proprietary and incomplete nature of the existing systems at CAT, a project was begun a few years ago with openenergymonitor.org developers to develop a system to gather data from our various electrical generators, and present it to the public visitors in a understandable way. This proposed project will build on the work of the open energy monitoring project, to develop an open energy management and monitoring system.

The project would initially be looking into the feasibility, and best method of implementation of the proposed plan, to use known well developed technologies such as Arduino micro controllers and xBee wireless RF modules, as part of a meshed wireless energy management and monitoring system.

The main part of the project would be to develop and install a relatively small demonstration of the system, but one that demonstrates all the fundamental requirements of the system. The fundamental requirements are to:
  • Monitor and record a variety of types of sensor readings; e.g. pulsed output anemometers, and variable resistance thermocouples 
  • Operate external systems; e.g. boilers, pumps and radiators 
  • Display recorded data in a useful manor to the general public; e.g. an electronic sign showing the performance of a solar thermal system 
  • Be able to be programmed easily with new control strategies and room bookings 
  • Be developed under open source principles 
  • Be fully documented 
Who?

The applicant should be someone with an interest in open source electronic development. The project requires someone with a rigorous and detailed approach to their work, who has an interest in working in the field of environmental monitoring.

The applicant will be working within CAT’s Estates and Technical department, specifically alongside the Engineering team. The multi-discipline teams contain experienced qualified mechanical, electrical, control and heating Engineers, as well as a plumber, electrician and builder. Guidance and support will be provided throughout the project by the team, but a fair amount of previous experience with electronics and software development will be required.

This is also an ideal opportunity for someone who wishes to learn, in a hands-on manner, about renewable energy generating technologies, eco building, mechanical or control engineering. As well as the energy management system project, the applicant will be helping out with the day to day duties of the teams, including fixing and maintaining the site buildings, heating, electrical supplies, control systems and displays. There will also be opportunities to be involved in any of the projects being undertaken by the teams during the placement. Projects either on-going, or due to start in the next twelve months include:
  • Specifying, installing and commissioning a new biomass boiler for the WISE building 
  • Connecting the new WISE boiler, the new biomass training centre and site community cottages onto existing site heat main, including reconfiguration of SCADA control strategy to account for new loads and heat sources 
  • Upgrading the two main hydro turbines on site 
  • On-going maintenance and upgrade work to the site cliff railway 
S.M.A.R.T. Targets 

Specific
goal:
The goal of the project during the year placement is for the applicant to develop and demonstrate and document, a wireless open source energy management and monitoring system. This system can then be expanded in the future so that it can be used to monitor and reduce site energy usage.

Measurable goals:
The system must be able to perform four basic requirements. All four basic requirements will be restricted to an individual node on the network for the demonstration, and therefore can be developed separately. The four basic requirements are to:
  • Monitor and record a variety of types of sensor readings; e.g. pulsed output anemometers, and variable resistance thermocouples 
  • Operate external systems; e.g. boilers, pumps and radiators 
  • Display recorded data in a useful manor to the general public; e.g. an electronic sign showing the performance of a solar thermal system 
  • Be able to be programmed easily with new control strategies and room bookings 
Attainable and Realistic goals: The project is a significant undertaking, but also one that is realistic and attainable. Within the Engineering team at CAT, exists all the skills and knowledge to undertake the project, all of which will be available to the applicant to help guide and steer when required. The project also has a real world end use as an objective, which will provide significant motivation to the applicant.

Timely goals:
The placement is limited to a period of one year, which provides a fixed end point to the project.


How to apply & Role description
http://content.cat.org.uk/index.php/vacancies?download=123%3Afunded-student-placement-building-energy-monitoring-system-developer-fixed-term

We've done a fair bit of work with the Center for Alternative Technology over the years, recently running a openenergymonitor course from there and in 2010-2011 working on a emoncms powered microgrid display and pulse counter/data logger for grid import/export.

Microgrid display project at CAT [1] [2] [3]
CAT OpenEnergyMonitor course

We will be happy to support you, go through things that may be helpful both on hardware and software design, the openenergymonitor lab is based 30 miles north of CAT and we often go down to CAT and Machynlleth.

CAT is one of Europe's leading sustainability center's, it is a university, a visitors center a pulisher of books, reports such as zerocarbonbritain and a place were practical solutions, sustainable technologies are tried out, tested and demonstrated.

Application deadline is the 15th of July