Blog
OpenEnergyMonitor

Notify on inactive feeds

Here's a simple feature that sends an email whenever emoncms feeds become inactive for more than two hours, designed to stop those instances where you login to emoncms after a month and realise that something went wrong and it hasn't been updating for a while.

The notify control page can be accessed via the Extras menu. It can be disabled or enabled and an email address that is different from you main account email can be used if needed.

A cron job runs every 2 hours on the server to check if any feeds have become inactive for more than 2 hours but less than 4 hours. This ensures only one email is sent and historical inactivity is ignored.

Notify is now running on emoncms.org, you may want to check that your email address is correct before turning it on.

To install notify on your raspberrypi or own server see installation details here:
http://github.com/emoncms/notify


PCB Manufacturer Meeting

A couple of months ago I met up with Stacey Driver who is the Business Development Manager of PCB manufacturer Stickleback Manufacturing Ltd. We have been working with Stickleback for several years now, over that time they have grown in size and moved to a new factory. It's great to support (in the small way we can) a UK based manufacturer.

Here's a copy of Stickleback's recent newsletter where our meeting is mentioned alongside details of their new water treatment plant they have recently installed. It's great to hear that they are continuing to invest in their manufacturing processes and reducing their environmental impact. This links in well with my previous blog post on Ethical and Sustainable Electronics 


Stickleback Chat - Official news of Stickleback PCBs

Image

Water Recycling Treatment Plant

As part of our continuing energy efficiency scheme we are thrilled to announce that we have installed a water recycling plant for our factory in Rochester, Kent. The introduction of the unit will significantly reduce the volume of waste water produced as a direct result of our processing.
Acids and alkalines used in our manufacturing processes are neutralised under controlled conditions. The ph. of the solution is carefully controlled allowing the water to be reused. This reduces the volume of water drawn by our company and removes the possibility of any polluting effects.
More and more companies are investing in waste water treatment. For Stickleback, we see it as part of our corporate responsibility to preserve and protect the natural resources we have wherever possible. We can’t wait until it is all put together and up and running.
Image

Open Energy Monitor and Google Campus

Stacey Driver had a fantastic meeting with our long-standing, valued customer, Glyn Hudson, at the Google Campus.
Glyn is a highly ambitious and motivated young entrepreneur, who is proud to be manufacturing a British product. The Open source hardware product, manufactured by Open Energy monitoring, allows us to relate our use of energy to our energy systems and embrace the challenge of sustainable energy.
We’ve worked with Glyn from the prototype stage through to production and his quantities continue to grow as his products are sent worldwide.
We are proud to support growing business’; start-ups and SME’s and have developed smart prototyping systems to help minimise the required outlay at these costly development stages.
Contact Stacey today on [email protected] or 01634 272416 if you think we could become part of your businesses fantastic journey.
We would be delighted to discuss where Stickleback may be able to help!


Ethical and Sustainable Electronics

Like many of us I love technology and enjoy using the latest gadgets, but this often sits uncomfortably on my conscience since I also care deeply about the environment and try and do my best to reduce my energy consumption and carbon footprint where possible.

The choices we make as consumers do make a difference, the stuff we buy accounts for about 40% of our current energy consumption, this is for the average person, it's probably a fair bit higher for 'techies'! The energy used to make the stuff we buy is more than the energy we use for personal transport and more than our energy demands for heating and cooling [1 – P.94]. Hi-tech electronic items have a large embodied energy carbon footprint and often contain 'conflict materials' such as gold (contacts) , tantalum (capacitors), tungsten etc. Manufacturing a PC requires 11 times it's own weight in raw materials to produce compared to 1-2 times for a car [1 - P.94]

Fair Trade Graphic from Nager IT

In many areas of purchasing there is an ethical and lower carbon choice available, like buying local vegetables and fair trade coffee and chocolate etc. However when purchasing electronic items I have always assumed there was not this option, that no product is more ethical or lower carbon than another (they are all just as bad!).

I would like to highlight two projects that I have discovered lately that are working together to try and improve the state of electronics manufacture. I should state that I have no connection with these projects other than reading their websites.

The first is a German project called Nager-IT who have produced a "Faire Maus" or Fair Mouse.

They have designed the USB mouse using low carbon materials and have gone to great lengths to source ethically produced components and have pushed for transparency throughout the supply chain (something which is almost non existent in electronic production at the moment). Checkout this graphic they have produced illustrating their transparent supply chain:



 The mouse they have produced can be purchased through their website.

The second project is called FairPhone, they are a social enterprise in the Netherlands with the aim of " designing, creating and producing our first smartphone and taking the next crucial step in uncovering the story behind the sourcing, production, distribution and recycling of electronics." They are developing and aiming to start stating in Autumn this year a 'fairer' android powered smartphone.

Graphic from http://www.fairphone.com website

It's uplifting to hear of small independent companies taking on the big players in the consumer electronics industry by offering a 'fairer' alternative. We as consumers should do our best to show that there is demand for more ethically produced electronic projects.

I have also discovered that GreenPeace produce an independent ranking of the large electronic companies based on how 'green' they are.

Graphic from GreenPeace


[1] – Sustainable Energy Without the Hot Air – David MacKay [2009]

How this relates to OpenEnergyMonitor 

Steps we have taken so far

  • Choose RHOS and components which are free of conflict materials
  • Choose low embodied energy materials for enclosures which can be easily recycled
  • Use standard parts which can be easily re programmed and used for other purposes in the future
  • Use recyclable packaging (paper bags which are naturally antistatic) for our shipping 
  • Choose low power equipment, LED lighting, recycled paper for our office and switch things off at the end of the day! 
  • Choose surface freight (boat) rather than planes to ship stock (mainly power adapters) from China, this takes 3 months rather than 3 days but worth it for the 32 times less energy consumed [1 - P.92]. 
  • Choose a PCB manufacture who are based in the UK who uses lead free techniques, complies to the highest environmental industry standard and is actively investing in techniques and equipment to reduce wastage and environmental impact (e.g water treatment and recycling) 
  • Strive to optimise electrical consumption in our hardware to be as low was possible 

Steps we would like to take in the future 

  • Produce an end-to-end supply chain map of our manufacture (possibly using http://sourcemap.com)
  • Try and calculate the embodied energy that goes into producing our designs and constantly try and find ways we can reduce this
  • Sell through local distributors to reduce air based shipping and investigate possibility of local manufacture
  • Use recycled plastic / other low embodied energy martial in the moulding of the housing for our next version of the emonGLCD and temperature monitoring node 
  • Investigate how we can build recyclability into the design, checkout the video below from the Great Recovery Project:

Preparing emoncms.org feeds for conversion to timestore

One of the central principles behind the Timestore data storage approach is that datapoints are stored at a fixed interval. Because you know that there is a datapoint every 10s you dont need to store the timestamp for each datapoint, you only need to store the timestamp for the first datapoint. The timestamp for every other datapoint can be worked out i.e:

timestamp = start + position * interval.

Storing time series data in this way makes it really easy and very fast to query. The tests so far have shown timestore to be several magnitudes faster while also using significantly less disk space.

If your logging to emoncms.org there is now an interface that can be used to select the interval rate that you would like your feeds converted too, its linked from the feeds page.

On the feed/convert page:

The new interval column states the average interval rate of the existing feed and is calculated simply as the end time minus the start time divided by the number of datapoints in the feed. This interval rate can be skewed if the monitor dropped off for a period, so its worth double checking that it is correct.

You may wish to change your interval rate, if your logging temperature data at 5s intervals and 60s is enough to see the changes in temperature you want to see then select 60s as this reduces the disk use of the feed considerably.

To set the interval rate you wish your feed to be converted to, click on the pencil button to bring up in-line editing:

Click on the drop down menu under convert to and select from the list the interval you wish to use. Click on the tick button to complete.

Repeat the above steps for every feed you wish to convert and then once your done click on Add feeds to conversion queue button that is at the bottom of the conversion page.

As mentioned in the last post, its going to take up to a month for the server to convert through all the feeds so you wont see anything straight away. I will update with conversion progress.

I've put all the code for the above on github under an emoncmsorg branch of emoncms, the conversion scripts can also be found in the repo usefulscripts

From 1.3 minutes to 196ms: timestore on emoncms.org


John Cantor who uses the openenergymonitor system for heatpump performance monitoring got in contact the other day saying that graphs on emoncms.org where taking a really long time to load. I said that the work on using timestore will soon speed things up significantly. He asked if there was a chance to convert the feeds before he went up to work on a system in Scotland as being able to zoom with speed through the graphs would make a big difference.

Clicking on the Y (Year) button in the rawdata visualisation has slowed down so much over the last year that it now takes over a minute to load the view.


The same request on a raspberrypi takes 2.9 seconds, The thinking to why its so much slower on emoncms.org is because of the use of the php loop doing individual requests to mysql which are added to the mysql process queue and have to wait in line amongst all the other mysql inserts and updates caused by all the updating feeds, the php loop method was initially faster than an equivalent mysql implementation but clearly not for long.

Anyway I figured we could get timestore up and running on emoncms.org straight away if we run timestore in parallel with the current mysql storage and then slowly convert feeds over to timestore over time. The present input processing and visualisation implementation would be kept as is for the time being only initially using timestore to replace the realtime datatype rather than both realtime and daily.

Conversion from mysql to timestore takes a long time especially as the process has to be throttled down in order to allow enough resources on the server for normal operation to continue. For example the conversion of a 568 day long, 5s datapoint interval feed (~152mb) takes about half an hour loading the server to an average 15min (quite high) load of 4.69 at that rate it will take about a month to convert all feeds over to timestore.

The results though are worth it!:


196ms!

1.3 minutes down to 196ms is a 398x speed increase! and its also possible to see another advantage of timestore the effect of the average layers showing a slight winter summer curve. A big thankyou and credit to Mike Stirling for developing timestore enabling this kind of performance, if you haven't already check out the timestore project website:


John Cantor has shared the Scottish heatpump dashboard publicly, take a look to try out the new timestore feeds:


Im currently working on an interface in emoncms to allow selection of the feed datapoint interval (5s, 10s, 30s etc) to convert to before the feeds are added to the conversion queue, more on this soon.

OEM Gateway by Jerome Lafréchoux

I would like to highlight the good and interesting work that Jerome Lafréchoux has been doing on developing a gateway that is intended to go beyond the current RFM2Pi interface towards something that can be used on any linux system whether a pi or any computer to take say input from a serial RFM12Pi and forward it either to local instances of emoncms or remote instances of any service (without necessarily needing a local install of emoncms).

Here's an exerpt from Jerome's last post on the github topic on gateway development:
"The standalone gateway seems complete enough for me to be published, although I'd rather have a little feedback before recommending it.
It works basically the same way. Here are the main features:
  • Can be parameterized either via emoncms GUI (with current limitations of the interface) or through a configuration file. Useful for those who don't want to install a local emoncms just for the configuration of their gateway. This also allows to add features without the need for complex GUI edition.
  • Can read inputs from the RFM2Pi, but also inputs from the serial port of the form "NodeID val1 val2 ...", or even from a socket. This socket possibility means any application on the same machine or on the network can send data to the gateway. This even solves the inter-applications issue. It is better than writing to a file, in my opinion.
  • Extensibility. By design, it should be easy to create new inputs (data sources) or outputs (another server's API). It uses classes, so a new input (listener) or output (buffer) can be created through inheritance/overriding.
See readme file for more explanations.
I should add a socket use sample, but it's pretty basic and require very few code. Here's a python example:
import socket
HOST = 'raspberrypi'    # The remote host, can be 'localhost' just the same
PORT = 50011            # The port chosen in the gateway config
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.send('14 15 26\r\n')
s.close() 
Currently, each buffer instantiated in the gateway buffers a given amount of data when network is down. This is limited (we don't want to eat up all memory). Next step could be to add the possibility to buffer samples in a file (with care for SD card wearout)."

To try it out see the github repository here: https://github.com/Jerome-github/oem_gateway. Jerome is looking for testers and would appreciate feedback on it.

EmonTx Shield SMT progress

The latest emontx shield SMT board revision (v2.4) arrived a couple of days ago.

This revision carries on from the developments described in the previous blog post here: http://openenergymonitor.blogspot.co.uk/2013/06/emontx-shield-smt-with-new-layout-in.html

The improvements in v2.4 are:
  • The DS18B20 connector is bought out to the edge with screw terminals.
  • There's an FTDI connector as the shield covers the FTDI connector on the NanodeRF.
  • The jumpers are 2.54mm headers rather than surface pads that are soldered, making them easier to change.
  • The RFM12 footprint is moved further in on the board so that there is space for the Ciseco SRF with its on board chip antenna


There was one small but crucial mistake, I got the RX and TX on the FTDI connector the wrong way around!!

Next step: switch the RX and TX around and send the board off for production.

DIY Casing

We are working on new designs which which will be better designed to fit into cases in the meantime John Cantor  (from heatpumps.co.uk) who has been using a modified OpenEnergyMonitor system for monitoring the performance of air and ground source heatpumps shares a photo of his experiences of boxing up the emonTx and Nanode RF SMT

emonTx setup for temperature monitoring and nanode RF SMT cased up using standard RS cases

These cases from RS were used:

ABS moulded box, 120x65x40mm, ivory
RS Stock No.502-641
ABS moulded box, 112x62x31mm, ivory
RS Stock No.502-635
both £3.45+vat.

Ciseco SRF and RFM12B Power Consumption Investigation

Following on yesterdays post on the Hope RF RFM12B power consumption I decided to do a comparison with the Ciseco SRF Radio. An RFu328 (miniature ATmega328 in XBEE footprint) was used to mount and interface with the two radios. The same 3.3V power supply was used with both modules

Ciseco RFu328 with SRF and RFM12B

The scope was connected up to measure the voltage drop across a shunt resistor as follows:

Oscilloscope probe measuring voltage drop across a 10R series resistor

Hope RF RFM12B

Here is the RFM12B current consumption trace while sending 4 integers using the JeeLib packet structure. Using this packet structure each integer takes up 2 bytes, therefore 4 integers is 8 bytes plus 1 byte containing the node ID, this gives a total packet size of 9 bytes. Transmission takes 2.7ms and the current consumption in the time is about 24mA @ 3.3V. This gives a power and energy consumption of 24mA * 3.3V = 79.2mW * 2.7mS = 0.214mJ = 214uJ


Ciseco SRF


A SRF V1.0a with serial firmware was used for this test.

The SRF is serial based. Ciseco have standardized on a communication structure called LLAP (Lightweight Local Automation Protocol). 

A LLAP packet consists of one start byte 'a' , two bytes for the node ID then 9 bytes for the message. Encoding as HEX each LLAP packet can give us space for two integers. Each integer has a range of -32767 to 32767 which is fine for our standard emonTx setup which has a maximum reading of 25000W (100A x 250Vrms).

To transmit four integers from the emonTx (3 x power and 1 x voltage) would require two LLAP packets which each contains 12 char characters which gives a packet size of 12 bytes transmitted twice giving a total of 24 bytes.

Here is a current capture waveform of the SRF transmitting two LLAP packets, it's rather more interesting than the RFM12B, I would love to know exactly what the SRF is doing at each spike and dip.


Average power consumption of 20.8mA


Transmission of two LLAP packets takes 15ms

Transmission of two LLAP packets takes 15ms with an average current of consumption of 20.8mA. This gives and power and energy consumption of 20.8mA * 3.3V = 68.6mW * 15ms =1mJ.

This is 4.7 times more energy than the RFM1B for the transmission of the same four integers. This is mainly due to the efficient nature of the JeeLib packet structure sending the integers as binary rather than serial characters as in the case of the SRF. Transmitting four integers as HEX characters in two LLAP packets takes 24 bytes as opposed to the 9 bytes needed for the same four integers in the RFM12B JeeLib packet structure. Taking this into account the SRF consumes 41uJ per byte where the RFM12B consumes 23uJ per byte, this is around 1.8 times more power byte for byte than the RFM12B.


SRF startup 50mA spike

An interesting observation is that the SRF exhibits a rather high current spike of about 50mA as it's turned on / comes out of sleep. As this spike only lasts for only about 100nS it won't contribute that much to the overall power consumption. 

Energy Consumed While Sleeping

The energy used the the RF modules needs to be put in perspective with the overall consumption of the system. An emonTx running on batteries or low power temperate node will spend much of it's time sleeping, the ATmega328 consumes 4.3uA when sleeping and the SRF and RFM12B consume about the same when sleeping 0.2-0.3uA, giving an overall sleep mode power draw of 4.6uA or 0.0046mA.

Sleeping for 10s

Assuming a the case of a wireless node which sleeps for 10s in between readings. This gives a energy consumption of 0.0046mA * 3.3V = 0.0152mW * 10s = 0.152mJ = 152uJ.

If this node was using an RFM1B 1.4 times more energy would be consumed in the 3ms that the RFM12B is active while transmitting the data via RF then in the proceeding 10s when the node is sleeping

If the temperature node was using an SRF 6.6 times more energy would be consumed in the 15ms that the SRF is active while transmitting the data via RF then in the proceeding 10s when the node is sleeping.

Sleeping for 10 min

Assuming a the case of a wireless node which sleeps for 10s in between readings. This gives a energy consumption of 0.0046mA * 3.3V = 0.0152mW * (60s *10) =  9.13mJ.

The energy consumed while sleeping now becomes the greatest consumer. The energy consumed during sleeping for 10s is 43 times greater than the energy required by the RFM12B to transmit the data or 9 times greater than the energy required by the SRF to transmit the data.


Conclusion

If a ATmega328 based 'sleepy' node sleeps for 14s or more the energy used during sleeping will equal or greater the energy used by the RFM12B (to transmit four integers). If the nodes sleeps for 1 min or more the energy used during sleeping will equal or greater then energy used by the SRF (using serial LLAP to transmit four integers).

LLAP serial on the SRF not the most power efficient way to transmit integers compared to the RFM12B using the JeeLib packet structure. Power consumption of the SRF can be reduced at the expense of human readability of the data packets. I plan to investigate this further, see questions to answer below:

Questions to answer:

Does the extra energy consumed by the SRF result in increased range over the RFM1B?

The SRF by default is set at 10dBm transmission power (compared to 0dBM for RFM12B), this can be reduced all the way down to -30dBm in various increments, how much will this reduce energy consumption and range? Is there a sweet spot? The RFM12B transmits at 0dBm, how will the range of the SRF transmitting at 0dBM compare to the SRF? 

The SRF currently transmits at 9600 baud rate, this can be increased to 115200, will this reduce the time taken to complete a transmission and therefore energy used. How much will this effect loss of packets and range? Ciseco SRF setup documentation. 

Is it possible to interface directly with the SRF to transmit the raw packets not using serial?

Can power consumption of SRF be improved with new firmware?

I hear it's possible to use the CC chip on the SRF to offload the WDT to wake up the ATmega328 using a hardware interrupt, this could result in sleep current draw of around 0.3uA. I'm keen to investigate this.

New Oscilloscope - RFM12B Power Consumption

We have finally taken the plunge and have upgraded our measurement facilitates.

The main (and most costly!) acquisition has been a Rigol DS2072 70Mhz digital scope.



We also upgraded our old cheapo multimeter to a more accurate model which as well as the usual multimeter functions can measure AC power and frequency and has a USB link (Uni-T UT71E) and to complete the setup a variable DC voltage and current bench top power supply (Rapid HY3003D).

Test setup

Uni-T UT71E True RMS 40K count multimeter

I thought an interesting investigation and a good starting point for getting to grips with the scope would be to measure the power consumption during an RFM12B transmission.

With the standard passive probes the scope only measures voltage so a 10R 0.25W shunt resistor was used in series and the voltage drop measured over this. A 10R resistor works nicely since the voltage drop can easily be converted into current by dividing by 10 which is done by the probe which is 10:1 passive probe. Any other value could be used and the conversion to current done in software on the scope. The units for current (A) can be set on the scope. The shut resistor must be connected on the GND side of the circuit as to make one side of the resistor 0V or else use differential probes (or two standard probes with some clever settings on the scope).

Scope probe connection, measuring voltage drop across a 10R resistor

The maximum current which a shunt resistor of a particular power rating can handle can be calculated with Imax = sqrt( Pmax / R).

A 0.25W 1R shunt resistor can handle up to 500mA
A 0.25W 10R resistor can handle up to 158mA
A 0.25W 100R resistor can handle up to 50mA

All measurements were taken with a 3.3V supply voltage using and RFu328 on a prototype emonTx V3.

 

These traces show the ATmega328 coming out of sleep then transmitting 4 integers. Transmission of 4 integers takes 3ms during which the current draw alternates between 23-28mA for 3ms.   

Testing the scope's capabilities using the waveform zoom function
The built in measurement functions on the scope are pretty amazing, these of the sort of measurements it's possible to obtain for a particular signal. 

 
We intend to eventually setup a proper AC power test rig. In the meantime I intend to continue playing with more of the scopes functions, I've only scratched the surface so far. I'm particularly interested in hooking it up to my laptop for further data processing. The scope has both USB and Ethernet connectivity and an open standard protocol. As well as the software bundled with the scope which look pretty decent but is only for windows a quick google shows that much of the work has already been done to import the data from the scope on Linux.

The EEVblog review of the Rigol scope give more of an idea of it's full capability: 


http://www.eevblog.com/forum/testgear/first-impressions-and-review-of-the-rigol-ds2072-ds2000-series-dso/

http://www.eevblog.com/forum/projects/software-tips-and-tricks-for-rigol-ds200040006000-ultravision-dsos/
I hear it's also possible to perform a light software hack to convert the scope to 100Mhz :-)  http://hackaday.com/2013/07/02/unlocking-a-rigol-scope-once-again/