Data portability challenges

Continuing from yesterday's post on making emoncms data portable.

There are some challenges with implementing data portability for energy data. 5s energy feed data can grow to be quite large, I have several feeds that are approaching 100Mb and a total account size of about 700Mb and thats only just approaching one year.

Moving large quantities of data uses server resources including: cpu, memory and bandwidth. is hosted on a BigV virtual machine, where you get 1GiB Ram, 25GiB Disk space, 200GiB transfer for £10 a month, which is a fantastic deal for those specs.

As exporting or importing large feeds takes time its important that it does not hog resources on the server making the server unusable for other users, any import or export code needs to use a small percentage of the server resources and ideally the amount of resources used could be adjustable. We also need to keep an eye on the data transfer limits.

In the next post I will post up the initial working feed export code.

Making emoncms data portable

One of the features I've wanted in emoncms for some time is the ability to move feed data between different server's with ease.

I want to be able to log in the emoncms instance running on my local machine and go to a page where I can enter the authentication details and location of my account on a remote emoncms server such as at which point it would bring up a list of the feeds that are available and I could either download particular feeds or just click download all and it would automatically download the whole account.

This would allow me to get at my data when I'm offline and it could serve as a personal backup of the data. Another need for this feature is we have an old remote emoncms instance that has some old energy data on, this feature would make it really easy to move the data to the new instance.

This feature could also work in a similar way to a ubuntu one or dropbox like service, there could be an automatic syncing option.

This would make the energy data very portable, data potability helps to create an open web, ensures that we can be in full control of our energy data, we can remove it from a service if we no longer want to be with a particular service, use it in other applications and for other applications that may not have been thought of in the original application design.

Over the next few blog posts I'm going to try and explore how this can be implemented. I've already got some working code, but a lot of work will need to be done to make this a nice well implemented feature, so if you can help with the coding or/and provide insight into better ways of doing things, that would be most welcome.

Cleanweb UK

Last Thursday Glyn and I attended the Ignite Cleanweb UK event in London, an event exploring the power and potential of the web to help us solve our environmental challenges.

The format of the talks where all 5 minute, 20 slides, 15 seconds per slide which was a challenge to create but makes for a really fascinating event where you here about a lot of different projects in a short space of time. Videos of all the presentations talks are up on the web here:

Here's a video of our presentation on OpenEnergyMonitor:

The slides on their own can be viewed on slideshare:

Thanks a lot to James Smith (@floppy) for organising the event and inspiring us with the cleanweb uk manifesto at the end.

EmonTx and NanodeRF Code guides

These EmonTx and NanodeRF code guides go through the main components required to put a full EmonTx and NanodeRF (or ENC28J60 based OpenKontrolGateway) firmware together.

If your just starting out, you might find going through these guide examples first better than starting straight in with the fully fledged firmware examples.

Cleanweb UK - Ignite Evening - London 13th Sep

Thursday, September 13, 2012

7:00pm, at Forward Technology in London

Trystan and I are looking forward to attending and speaking at the first Cleanweb UK Ignite evening in London. It looks like it will be a great event, there will be 12 short (5min) fast paced talks to do with sustainability and the internet. 

Please come along and joins us, registration and full event details here:

Reading Watt-hour data from an Elster A100C electricity meter using IrDA by Dave Berkeley part 2

As mentioned before if you have an Elster meter this is a fantastic way to read the exact accumulated watt hours that you have generated or used and can compliment and cross check a CT based measurement.

Dave Berkeley has now packaged the code into a nice library which makes it really easy to use. We have uploaded the library to a repository on github here:

How to build it

1) To build the TSL261R based sensor for use with the emontx, follow the jack plug connection diagram on the emontx reference page here:

We've put together a small kit available in the shop of all the components needed to connect it to an emontx including the TSL261R a male 3.5mm jack, 1m of cable and heat-shrink.

2) Plug the sensor into the pulse input on the emontx (digital pin 3, interrupt pin 1)

3) Download the ElsterMeterReader library and place in the arduino libraries folder.

4) Open the Arduino IDE and run the following example:

#include "elster.h"

void meter_reading(unsigned long r)

ElsterA100C meter(meter_reading);

void setup()
  Serial.println("IRDA Meter reader");
void loop()
  // Decode the meter stream
  const int byte_data = meter.decode_bit_stream();
  if (byte_data != -1) {

5) If you now go to the serial window and place the sensor over the IrDA port (bottom-left of the meter) you should see the Watt-hours elapsed printed out to the serial window:

Thanks a lot to Dave Berkeley for this, for more information see his project page:

Open Kontrol Gateway - emonBase

For a while now we have been using the NanodeRF as our emonBase web-connected base station. The NanodeRF is great and has met our needs well. However using the ENC28J60 Ethernet chip does have some limitations.

We have through about having an emonBase option which supports the Wiznet Ethernet chip. This is the chip that is used in the official Arduino Ethernet. On the Wiznet chip the TCP/IP stack is implemented on the chip as opposed to on the ATmega328 as in the case of the ENC28J60, this means ATmega328 has a greatly reduced load, more application memory is freed up and the code is much simpler and brought in line with the Arduino Ethernet code examples.

In testing we have found the Wiznet chip to be more stable at holding a connection over a long period.

However the Wiznet chip is more expensive than the ENC28J60, what we need is a flexible emonBase unit which is interchangeable between the two Ethernet chips, enter the Open Kontrol Gateway (OKG)!

The OKG is a very flexible web-connected base station based on the ATmega328 which is totally Arduino IDE compatible (Arduino UNO bootloader). The OKG has been designed and manufactured by our friend Miles who runs Ciseco Plc.

The OKG PCB supports multiple wireless options such as RFM12B, XBEE, XRF, RN-XV Wifi. There is also PCB footprints for expansions like a RTC, SD card and SRAM. Finally what attracted us to the OKG was the ability to easily choose between the ENC28J60 or Wiznet Ethernet chips:

OKG ENC28J60 module

OKG Wiznet W5200 module 
We now have stock of the OKG in kit form in the OpenEnergyMonitor shop. The kits will come with an RFM12B wireless radio (choice of frequency) and a choice of ENC28J60 or Wiznet Ethernet chips.

The OKG with the ENC28J60 is almost hardware identical to the NanodeRF. NanodeRF emonBase sketches will run on the OKG with a couple of little changes.

See the OKG OpenEnergyMonitor wiki page for the changes required to make the OKG work with the RFM12B and OpenEnergyMonitor emonBase examples:

Note: The wiki is still in beta, we will blog more about it once it's been a bit more polished. It's our hope that by using a Media Wiki for our more technical documentation people will be more inclined to contribute corrections and additions and the documentations should be easier to organise and maintain. The OpenEnergyMonitor login should automatically log you in to the wiki. Just make sure your logged into the OpenEnergyMonitor website before opening up the wiki page. Try clearing your browsers cookies if this doesn't work first time.

We are now stocking the OKG in the OpenEnergyMonitor shop:

What about the Raspberry Pi!? 
Don't worry we haven't overlooked the Raspberry Pi! We have successfully had emoncms running on a Pi receiving RFM12B data from a USB JeeLink, we plan to blog more about this soon. We're currently with Martin Harizanov to make a RFM12B to Raspberry Pi GPIO expansion board. The Raspberry Pi will make a very powerful emonbase/home energy server however it's quite a complicated little beast for beginners who have limited linux knowledge. We think there is still a need to also have a simpler Arduino based emonBase option such as the NanodeRF or OKG.

emonTx SMT - Progress Update

Hello Everyone,

Here's another progress update on the emonTx SMT design.

This week I've sent the PCB CAD design files off to get a few prototype PCB's made. It's a very exciting moment, quite a milestone but also nerve wrecking! These prototypes will be assembled by hand and tested extensively. Eventually we plan to get the board ready assembled using factory SMT pick-and-place machines, selling the emonTX SMT as a kit is not an option! We don't plan to discontinue the current through-hole emonTx, we believe there is value in having a simpler through-hole kit version as an option. 

It will be sometime before emonTx SMT's are rolling off the production line, there are many hoops which need to be jumped through before then, but getting the first prototype off the ground is a good step forward! 

As a quick re-cap the main features of the emonTx SMT are:
  • Improved accuracy when monitoring low power levels with a programmable gain op-amp input stage on the CT inputs..see blog post
  • Single power supply - Half-wave rectifier AC-DC circuit to power the unit (under basic operation) and monitor mains voltage (enabling real power readings) from a single AC-AC PSU..see blog post
  • Direct USB programming - No USB to UART or FTDI cable required - built in USB host using the ATmega32u4 building on Arduino Leonardo design.
  • On-board Ethernet and multiple wireless radio options – RFM12B will remain the default option.
  • Pluggable-terminal block - Pulse-counting input, 2 x one-wire temperature inputs and other spare I/O broken out on a pluggable-terminal block.
  • Casing - designed to fit in a nice wall-mountable extruded aluminium case with laser cut fascias and external SMA antenna.

For more info, design notes and my general ramblings see my forum post...

Overload Protection of Mains Electrical Circuits by Robert Wall

A new guide by Robert Wall on overload protection of mains electrical circuits.

Robert Wall writes:

The fundamentals.

All electrical circuits must have some means of protecting against overload. This may be done by including a fuse or circuit breaker in the circuit, or the protection may be designed in to the supply that is feeding the circuit. We will only consider fuses and circuit breakers, as these are most likely the device that will be encountered in a normal environment.

How to determine the ratings.

First, you need to know the current that the load demands, under all conditions. You then need to choose a fuse or circuit breaker that will not blow (or trip) when carrying that current, but with only the smallest safety margin under worst-case conditions. Finally, you must choose everything in between – cable, switches, plug & sockets, etc – so that each will safely carry the current that will guarantee that the fuse will blow, or the circuit breaker will trip, under all conditions (paying particular regard to ambient temperature and the ability of the cable expecially to stay cool, i.e. it must not be buried in insulation).

Read the full guide here: Overload Protection of Mains Electrical Circuits

Plug and play sensor nodes

At the moment to get a system up and running you need to do quite a bit of manual firmware configuration, setting unique node-id's selecting different emontx sketches for different functionality, hard coding the emonglcd to get different screens. To add a second emontx you need to add several lines of code to the emonbase and the basestation needs to know the data structure of the sensor nodes.

What if you could upload a standard sketch to each of these nodes, plug your base station into your router, power up an emontx and see emontx 1 appear in a node list in emoncms. Then as you add more nodes they automatically appear in emoncms, no additional base station configuration needed. In time the software could develop to allow remote configuration of nodes such as emontx config and emonGLCD pages: beginning with page template functions as described in the last blog post: emonglcd template based displays

Continue reading the forum post here.