Emoncms.org backup

I'm happy to announce that emoncms.org is fully backed up and has incremental backup implemented, all data is incrementally backed up once every 24 hours, a backup cron job runs hourly syncing 640 feeds each time so 15360 feeds every 24 hours. emoncms.org disk use is currently growing at a rate of about 300MB a day and the transfer format is csv which gives you an idea of the volume of data that the backup implementation needs to sync. The total volume of data I have synced so far using this is 49 GB.

The backup implementation uses many of the things already developed as part of the sync module which allows you to download feed data from a remote server. I've put the full emoncms backup script in the tools repository on github here:


For the above script to run, you need to first copy the users and feeds table from the master server to the backup server using the more common backup procedure of using mysqldump and scp, the steps to do this are described in the header comments of the backup script.

This method of backing up is much faster than using rsync which I originally tried for incremental backup as it does not go through each feed looking for changes it just checks when was the last datapoint in the backed up feed and downloads every new datapoint from the master server recorded after that time, one disadvantage of this is that any changes to feed data using the datapoint editor tools in emoncms will not get updated to the backup server. It would be good though to make it possible to delete data on the backup server if its deleted off the master server, as disk space is expensive and if you delete data off emoncms.org you would expect no copy to remain from a data privacy point of view.

I implemented the backup system like this because I had most of what I needed already in place in the sync module and so it was the quickest way for me to get this up and running using what I already knew. I'm aware the database replication can be performed with mysql replication, where a transaction log is stored on the master server and transferred to and then executed on the backup server. I'm interested in exploring this option too and if anyone can tell me that using mysql replication will offer significant performance benefits over the method above and why, that would certainly motivate me to look at it sooner.

I'm still reluctant to guarantee data on emoncms.org as both vm servers are in the same datacenter and they are part of bigv cloud which could even mean that both share the same disk (which would invalidate one of the reasons for a separate backup to protect against disk failure) although bigv suggested that this is unlikely as there are plenty of tails. They recognise this as a weakness and something they hope to change soon.

So if you want extra peace of mind I suggest installing a local installation of emoncms on your own computer and downloading your data periodically using the sync module, I do this both for extra backup and so that I can access the raw data for trying new visualisation and data analysis, processing ideas. I will write a guide on how to do this soon. The sync module is available here:


I'm interested in being transparent about how emoncms.org is hosted, so that rather than give opaque promises you can asses things like how its backed up for yourself. You often here people say that no system is absolutely secure and completely safe from failure so I hope that by being transparent about this you can see what has been done. I'm relatively new to administering web services and I'm sure if your a more experienced web admin reading this you may know how this can be done better, I would appreciate hearing how you think it could be improved.

12 input pulse counter idea

A while ago now Glyn and I worked on a design for a 12 input pulse counter, we where doing some work at the Centre for Alternative technology, a stripboard version was built and is in continued use monitoring grid import/export, chp and diesel generator (the last two not actually in active use).

We wrote up about it here:

After visiting CAT again recently and discussing a project they hope to do, it got me thinking again about the 12 input pulse counter. In non-domestic buildings that already have pulse output meters on many of the circuits and a meter room with all the meters in one place, a multiple input pulse counter may be the most effective way to add automatic meter reading.

I've wanted to make a PCB for the 12 input pulse counter for a while so I though I'd do a little work on it this morning, here's a screenshot of where I've got to so far:

Here are the features Im thinking it will have:
  • 12-input pulse counter
  • Optional pull down resistor with option for SMT or through hole, see building blocks pages linked above for why pull down resistors are required.
  • Input status LED, driven by pulse signal.
  • Dedicated ATmega for pulse counting
  • Serial connection to second ATmega used for ethernet or/and rfm12 comms.
  • Enclosed in a DIN rail mounted enclosure.
Here's the eagle design so far:

I used the rfm12pi board design as a starting point as it already had the basic atmega + rfm12 circuit in place. 

One thing I'm still wondering about is whether to add a second optional resistor between the terminals and the pull down resistor which would provide the option of having a voltage divider on the input for stepping down from higher pulse voltages like 24V.

More to come soon..

Continuation of emontx testing - feed comparison tool

Glyn has been running a parallel test of the new emontx v3 vs the old emontx v2 for over a month now. See Glyn's original post introducing emontx v3 here: http://openenergymonitor.blogspot.co.uk/2013/02/emontx-smt-update-introducing-emontx-v3.html

In testing a new emontx version here are a few questions that we would like to answer:

How does emonTx v3 accuracy compare at lower power's?
Is there any difference in low power readings with or without the powered from AC-AC adapter feature?
Can differences be explained by calibration error?
Are there any other measurement variations that need investigating?

To make it easier to compare the parallel test power feeds I though Id create a visualisation tool in emoncms that made it easier to see the difference between the feeds.

If difference is caused by calibration error then applying a calibration to the measured data should bring the difference down close to zero.

Any deviations in measurements that remain should be non-calibration errors, and they will appear off to one side of the linear PowerX vs PowerY plot.

The above visualisation can be viewed here:

This visualisation tool is available in the emoncms visualisations list if youd like to try this on your own monitor, even comparing say two different CT channels on a single emontx.

There are some issues I need to fix with the visualisation tool implementation that gives rise to some incorrect comparisons at some scales to do with the way it selects datapoint id's to compare.

In the next post I will explore differences between the two parallel test power feeds.

On another topic:
One of the interesting things I did yesterday was use the raspberrypi emoncms module on my ubuntu laptop. I used a jeelink connected to the usb port of the laptop and then configured raspberrypi_run.php to connect on port /dev/ttyUSB4 instead of the default raspberrypi port, this could be a useful configuration for anyone who just wants to log data from the nodes locally to their laptop and as Jerome pointed out here, maybe the emoncms raspberrypi module should just be called the emoncms linux board module or just serial interface module.

Testing DS18B20 temperature sensing on emonTx v3

Hello, so I thought Id try something new on this blog, I'm going to try writing short blog posts about progress and general work on things day to day, these are not intended to mark significant developments or milestones which many of the other blog posts have tended to be, something more like a log book that will hopefully give insight into the development process.

Testing DS18B20 temperature sensing on emonTx v3

Glyn and I with help from Robert are continuing with testing emontx V3, there have been a few niggling issues that we're working on but its pretty much there. This morning I tested the DS18B20 temperature sensor connection, it all worked fine, one of the nice things of the new emontx is that there are screw terminals for connecting up the temperature sensor which makes connecting up the encapsulated DS18B20 temperature sensors much easier.
DS18B20 connected to emontx v3
Another new addition for temperature sensing is the ability to switch the sensor power pin on and off for use when powering the emontx of batteries (this avoids the hack introduced in the low power temperature node that uses the emontx v2.2 pcb). Anyway the circuit was originally designed with analog 5 (used as digital pin 19) being used for data and digital 5 used for power. While reading about using analog inputs as digital pins I came across this note on the arduino site:

The Atmega datasheet also cautions against switching analog pins in
close temporal proximity to making A/D readings (analogRead) on other
analog pins. This can cause electrical noise and introduce jitter in
the analog system. It may be desirable, after manipulating analog pins
(in digital mode), to add a short delay before using analogRead() to
read other analog pins.

It turns out one can just switch the atmega pins used around so analog 5 (digital 19) is used for power and digital 5 is used for the data which will be higher frequency switching and so good to bring that off the analog pin. As power switching is only needed in battery operation one could make a temperature measurement after the analog read section just before putting the emontx to sleep which should not cause interference.

Update (15/06/13): On future emonTx V3 PCB revisions 3.x ADC 5 (Dig 19) and Dig5 have been swapped round on the PCB to fix this issue. Dig 5 is now DS18B20 one-wire signal and ADC5 (Dig19) is now DS18B20 power.

If no DS18B20 temperature sensor is connected the ports can be used for other functions. ADC 5 can be used as a analogue input/output and Dig 5 (with R27/R24 4.7K pull up removed) can be used as a general Digital I/O with PWM capabilities. 

The DS18B20 data 4.7K pull-up resistor on Dig 5 and the Dig 2 IRQ 10K pull-down resistor have been designed to take a thru-hole resistor if the SMT resistor was removed to make it possible to change the value of these resistors. 

Open Source Hardware Users Group (OSHUG) #26 Meetup

Last night I attended OSHUG event #26 in London and gave a short talk on Low Power Wireless sensors as well as the brief overview of the OpenMonitorProject and a quick look at what we've been working on recently.

There are a couple of new screenshots in the presentation from the open-source SAP building modelling emoncms module that Trystan is been working in with Manchester @CarbonCoop recently. This is an exciting bit of development. The idea is that the building model could be matched up to the monitoring results (temperature profile & heat input) of a building, the model could then be used to investigate the effect of undertaking improvement measures such as wall, loft insulation or external cladding etc.. Finally after completion of improvement measures the monitoring data can again be used to evaluate the actual performance of the improvement measures.

Osug #26 low power wireless sensors from OpenEnergyMonitor

It was great to meet everyone and put faces to (twitter) names! Big thanks to Andrew Back @9600 for organising the event and @skpang_uk for sponsoring the event...and buying me a beer afterwards!  

One Year of Solar PV Monitoring

On the 23rd November 2011 we have had a 2.9Kwp solar PV installed. 

We recently got our first payment from the UK's governments Feed In Tariff scheme. This has prompted me to take stock of how the system has been performing and how the data collected using the OpenEnergyMonitor energy monitoring system compares to the utility company's billing meter:

From the 23rd Nov 2011 - 23rd Nov 2012 the billing utility meter on the solar PV system has recorded a  generation of 2069 kWh. Over the same period we consumed 3588 Kwh, 57% of our electrical energy needs have been met by the solar PV. 

At this rate it looks like we're on track for the system to pay back in 7-8 years, maybe even less if we have some more sun in the next few years, fingers crossed! 

For the same time period the OpenEnergyMonitor monitoring system has recorded a generation of 2029 kWh, giving the energy monitor an accuracy of 98%! 

The monitoring system is a standard emonTx with an AC-AC adapter taking Real Power readings

One Year of PV Generation
Kwh/d Electricity Consumption (orange) Overlaid with Generation (blue) 

When the system was installed the solar PV company estimated we would produce 2434 Kwh per year, we generated 15% less than this estimate. This could partly be attributed to the very poor summer we experienced in 2012.

This year is already looking promising, on the 2nd of May 2013 our system generated a record (for us) of 18.3 Kwh 1.9 times more than we consumed on that day (9.7 Kwh):

Record Generation on 2nd May 2013
Fingers crossed for a sunny 2013 summer!

Carbon Coop

The weekend before last I went to Manchester to help Matt Fawcett and Jonathan Atkinson of Carbon coop with the EcoHome_Lab openenergymonitor energy monitor building workshop. I haven’t blogged about Carbon Co-op before so Id like to give a bit of an intro to who they are and why I think what they are doing is particularly exiting.

I first met Matt who is responsible for the monitoring at Carbon Coop at a Cleanwebuk event in London after initially exchanging a few emails. Matt explained that they are working on a retrofit programme to try to prove that it is possible to cut between 60-80% of energy demand from homes because this is what we need to save.

They want monitoring to be at the centre of everything they do, so that their members can see how minor changes impact energy usage as well as more major retrofit changes. They also want to monitor internal air quality as well as temperature for early identification of any problems with the insulation and airtightness measures. By monitoring humidity, CO2 as well as temperature Matt explained that its possible to isolate issues around damp/cooking etc from issues around ventilation.

They are really keen to use open source technology because that then gives them the freedom to configure the system as they need and then get at the data. They are keen advocates for open data and open source software and hardware in general and carry out their own work in an open way.

Since then Carbon Coop have installed 18 openenergymonitor energy monitors in houses across Manchester, held two energy monitor workshops including last weekends and are going forward with retrofitting 15 houses to between 60 and 80% carbon reduction.

Building monitors at the Carbon Coop EcoHomeLab workshop 19th-20th April 
What I really like about Carbon Coop is the way they are working on every part of the process: They start with the larger picture of what we need to do in terms of sustainable energy in housing and then go from there to work on an individual household basis, first carrying out a detailed whole house assessment based on the governments SAP (standard assessment procedure) to identify where most energy is being lost, which improvements would have the biggest effect and what to do first. They’re using the energy monitoring to provide feedback to households as described above. From there they go on to help households carry out work on their houses, from finding the best way to finance retrofit measures to working with architects and local tradespeople. Once the retrofit building works are complete they continue to monitor to ensure that the changes reach the predicted energy targets. They do all this using an open source approach, they are also a co-operative that is open for people to join. They are a really switched on group of people, carrying out their work in a thorough, thoughtful, evidence based way.

Every-time I go to visit them I come back inspired to develop things further. One of the highlights of the last visit was listening to Eleni Kalkantzi who works for URBED (Carbon Coop's technical partner) talk about the whole house assessment methodology and reports she, Charlie Baker and Marianne Heaslip have developed. A huge amount of work has clearly gone into these reports and the assessment method, they have an example report and report on the assessment method here:

See their example assessment here: http://carbon.coop/assessments/CC_EXAMPLE_ASSESSMENT2.pdf

Report on the assessment method: http://carbon.coop/assessments/URBED_ASSESSMENT_REPORT.pdf

They see the next step being to integrate monitoring data directly in to the SAP model on which their reporting is based to reduce the number of assumptions made. This is where the open source emoncms SAP model that we have been developing comes in but I will come back to that in another post.

Introducing the RFM12Pi V2 - Raspberry Pi Wireless Expansion Board

Since, November last year we have more and more been leaning towards using a Raspberry Pi as our web connected base station of choice. See here for our original blog post on using the Raspberry Pi as an emoncms logging  and visualisation server.

Here is the latest version of the RFM12Pi Raspberry Pi wireless expansion board. Like the V1 the RFM12Pi V2 allows the Raspberry Pi to receive and transmit data via RFM12B 433/868mhz from and to other wireless nodes such as the emonTx, emonGLCD and Remote Temperature nodes.

RFM12Pi V2 - Raspberry Pi Expansion Board

The RFM12Pi V2 is now for sale in the OpenEnergyMonitor online shop, it comes pre-assembled and ready loaded with firmware, when used in conjunction with the read-to-go emoncms Raspberry Pi SD card image it makes for a truly plug-and-play energy monitor web-connected base-station setup. We're also selling a PCB only version and of course the hardware design, Arduino based firmware all Raspberry Pi emoncms setup is all open-source.

For the ultimate super-quick and easy setup we have also made available pre-loaded Emoncms Raspberry Pi SD cards
Emoncms Raspberry Pi Pre-loaded SD card 

The new features of the RFM12Pi V2 are:
  • ATmega328 based, has more memory and hardware serial support
  • Using the ATmega328 hardware serial (+optiboot) and AVRdude (pre-installed on SD card image) it's possible to upgrade the firmware on the RFM12Pi V2 direct from Raspberry Pi, instructions are on the wiki.
  • Runs on ATmega328 internal 8Mhz oscillator (to save unnecessary components)
  • SMT used, board layout is optimised for pick-and-place assembly
  • Lower profile to fit inside most Raspberry Pi enclosures
See the the RFM12Pi V2 wiki for more technical details regarding the hardware and firmware.

To get up and running all that's needed is to connect the RFM12Pi onto the Raspberry Pi's GPIO pins, insert the pre-build SD card and bootup the Pi. Once booted up browse to the Pi's local IP address or the host name (http://emoncmspi) and then configure the RFM12B network setting from within emoncms.

emoncms Raspberry Pi Module Config Page

Monitoring data is logged to the Raspberry Pi's SD card and can be mirrored to emoncms.org or any other emoncms server for backup and remote viewing. The Raspberry Pi is powerful enough to be able to serve emoncms graphs and widgets.

emoncms dashboard served from the Raspberry Pi - viewed on a Nexus 7 Tablet 

Pictures from the OpenEnergyMonitor CAT course

A couple of weeks ago we held our first openenergymonitor course at the Center for Alternative Technology in Mid Wales. It was well attended with 17 of us in total including tutors: myself, Carlos and Suneil. We where also lucky to have Robin Emley come down to demo his MK2 Surplus PV diverter and lend a helping hand.

Here are a few pictures of everyone getting stuck in with building emontx's and rfm12pi adapter boards:

Carlos and Sebastian who came all the way from Germany going through the Raspberry PI setup process:

Robin Emley helping out above and his solar pv diversion setup below:

The course was held in the beautiful Welsh Institute for Sustainable Education (WISE)  building at CAT:

I'm in Manchester this coming weekend (20th April) helping Matt Fawcett and the rest of the Carbon Coop team with their: Ecohome lab launch workshop and build day. Looking forward to it.

Course at the Centre for Alternative Technology

by Carlos Alonso Gabizón

Open Source Tools For Energy Monitoring

25th - 28th March

When working towards sustainable energy it is important that we work towards it in a way that adds up. Energy monitoring is a vital tool for helping us do this from measuring our home electricity consumption to working out the performance of sustainable heating systems.
During this short workshop we will explore and build monitoring solutions that are based on open source technology developed as part of the openenergymonitor.org online research project. Solutions that can be configured for a wide range of applications including:
  • Household Electricity monitoring
  • Heat-pump monitoring
  • Solar PV generation monitoring
The workshop will focus around a practical case study of monitoring electricity generation and use at the Centre for Alternative Technology designing a public energy display to engage visitors that can be accessed both at the site and on the web.

Workshop content outline


Course tutors

Carlos Alonso Gabizón – Is the course leader. Worked on microgrid energy display project at CAT and open source software for extracting energy data from PV inverter equipment. Carlos has an electronic engineering and computer science background.
Trystan Lea – As a core developer on the openenergymonitor project Trystan will be introducing the hardware and software workshops. Trystan codeveloped with Glyn Hudson many of the main elements of the openenergymonitor system with a focus on software and firmware including emoncms an open source web application for energy visualisation. Trystan also runs the openenergymonitor kit shop with Glyn Hudson.Trystan has a Physics background with a BSc in Physics at Cardiff University.
Suneil Tagore – Will be giving an introduction to open source 3D fabrication/printing using the reprap. Suneil has also worked on DC monitoring for theHugh Piggot wind turbine. Suneil has a background in Engineering andcomputer science with an MEng in Engineering at UCL London and a MSc inComputer Science at Cardiff University.

Course cost and how to book

The course cost is £520. Concession: £340 (unemployed, retired or full time student). This covers accommodation, food and course tuition.
You can reserve a place on the course online at: courses.cat.org.uk/ or by phone calling the CAT Courses Department on 01654 704 952.

What will you get from this course?

Firstly, the knowledge and practical skills to build and set up a monitor solution based in the openenergymonitor kit and how to expand its functionality beyond what is done in the course.
Secondly, you will be able to install emonCMS, set it up and create really nice visualizations.
And lastly but more exciting than the previous, you will take home your own energy monitor completely ready to use!!!