emonTx V3 AC Adapter Socket Error

Just quick post to highlight a small bug in the first batch of emonTx V3...there had to be something!

The barrel-jack socket on the first batch of emonTx V3's has got a middle pin with a diameter of 2.5mm. Our adapters don't fit since they are 2.1mm..Doh! 

To solve this we're shipping a little adapter unit with each emonTx V3:

The AC-AC adapter is important and very useful on the emonTx V3 since it can power the unit as well as simultaneously providing an AC voltage sample for IRMS and Real Power readings.

It's been a busy day of order fulfilment today, all emonTx V3, emonTH and emonTx Shield SMT pre-orders will be shipped by tomorrow. Sorry for the slight delay, waiting for these adapter units to arrive was the cause for the delay. Hopefully the orders should all arrive at the destinations before Christmas.

Onwards! We're working hard to keep shipping out right up until Christmas. See my last post regarding last postage days for guaranteed delivery before Christmas.

Setup Emoncms on a RaspberryPI with a Hard drive

After the SD Card write issue setback I've been keen to get an alternative that works for running emoncms locally on a raspberrypi. I now have my home monitoring setup running off a raspberrypi with a hard drive (as recommended by Paul Reed, thanks Paul!) and a pihub as in the picture above and I've uploaded an image and details of this option to the documentation here:

I think there are quite a few benefits to running emoncms locally especially as we start to add the ability to control things from emoncms in addition to monitoring. Being able to set a heating profile shouldn’t depend on a working internet connection or a remote server's uptime. Control needs to be as robust and secure as possible.

I also think there is an important benefit of improved privacy: Energy, temperature and other environmental data is often sensitive personal data. Storing data locally is probably the best way to ensure you have full control over the privacy of your data as it never needs to leave the home unless that is you decide to share certain feeds for an open dataset or enable remote access for access while away or on the move.

Another benefit that I make regular use of is to develop new features such as the recent open source building energy modelling work (see application guide), because my data and the application was local I could edit the visualisations and model and see the results on my data straight away.

The emoncms raspberrypi hard drive image comes with quite a few add-on emoncms modules installed out of the box including the Event and MQTT module by Nick Boyle (elyobelyob). I tested the NMA notify my android feature and it worked great.

Here are all the add-on modules installed out of the box:
  • PacketGen, A new module for sending control packets.
  • Event module for sending Prowl, NMA, Curl, Twitter and Email notifications.
  • MQTT module for subscribing to MQTT topics containing data to be logged to emoncms.
  • Notify module sends an email if feeds become inactive
  • Energy module. Create a David MacKay style energy stack
  • OpenBEM module. Open Source Building Energy Model. Investigate the thermal performance of buildings.
  • Report module. Create's electricity use reports including appliance list exercise
There are two images in the zip attached in the documentation, one is called boot.img and the other pi_hdd_stack.img.

The boot.img goes on the SD card and just tells the PI to use the filesystem that is on the external harddrive. The second image pi_hdd_stack.img needs to be written to the hard drive.

With the harddrive connected to the PI through a powered hub and the boot SD card inserted in the PI, the pi should just boot-up and and be ready to use.

Environmental Considerations 

Adding a HDD to a Raspberry Pi setup does increase power consumption however I measured that my 2.5" 1TB USB external hard drive connected to the Pi only added 1.8W to the over all setup (3.9W). This modest increase in power can probably be justified.

Obviously purchasing a HDD has a considerable embodied energy, however there is an excess of low capacity second hand IDE hard drives commonly sold on ebay which are perfect for this application. A 20Gb drive (masses of space for emoncms!) can be picked up for a few £. With regular backup should provide a few years of service. It's great to be able to make use of something which otherwise would have to thrown away or at best recycled for raw materials.

Adding control to emoncms: RFM12b Packet Generator

I've had many conversations with people recently about controlling things like radiator set points, boiler thermostat's, heatpump's and so on.

I visited John cantor of today who is implementing an impressive control and monitoring system for a heatpump central heating system which involves EmonGLCD's with relay's connected to radiator valve actuators in each room. The buttons on the EmonGLCD are used to set the target temperature for the room and information is displayed on the EmonGLCD about heatpump power input and outside temperature.

On the way back I stopped at the Centre for Alternative Technology where Marnoch who is on their student placement there is working on doing something similar but for a radiator system connected to a wood chip boiler.

Both Glyn and Ynyr are keen to get boiler thermostat controllers working in their houses and Ken Boak who designed the NanodeRF phoned up the other day suggesting we start an openenergymonitor sub project and discussion on this topic, Ken's been working on central heating control for quite some time.

Its something I have a growing interest in too as we will be soon installing a heatpump system at home which has a lot of opportunity for control and monitoring to make sure its running efficiently.

So I've been giving some thought to how control features could be added to emoncms in a more integrated way than initial efforts I've made before.

I like the simplicity of the way Jean Claude Wippler designed the jeelib library to use c structures as the method in which data is packaged and sent via the RFM12b radio's and so I thought that maybe a good basis for adding control functionality to emoncms would be to first make it possible for emoncms to easily generate RFM12b data packets in the format that can be easily decoded with a struct definition on the listening control nodes. Emoncms would then broadcast this control packet containing all the state information about the system at a regular interval in the same way as an emontx broadcasts it's readings. Listening nodes can then be programmed to selectively use variables in that control packet depending on what they need to do.

I've put together an initial working concept of a module that can be used to do this and its up on github here:

Here's a screenshot of the main interface:

One of the nice things about it is that as you generate your control packet the module generates example Arduino code to make it easier to start coding a control node, you can just copy the code any drop it into the Arduino IDE and upload to a atmega+rfm12b based node and it should start receiving the control packet right away.

I see this module and control in general being run on a local installation of emoncms running on a raspberrypi + harddrive combination rather than The next step is an image with all this installed and working out of the box.

2013 Christmas Ordering Info - Royal Mail Last Posting Dates for Delivery before Christmas

It's that time of year again! We're busier than ever with the launch of our new hardware modules this week.Thanks to everyone who's helped support the project by ordering through the shop or contributed to the project in some way shape or form. All pre-orders for the new hardware will have been shipped by beginning of next week. 

Since I last posted a similar post this time last year we have shipped over 2500 orders and have moved into a new office

Here are the last posting dates for delivery in time for Christmas according to Royal Mail. We will continue to ship right up until Christmas but to avoid disappointment if you would like your items before Christmas please get your order in soon! 

International Airmail including
Airsure® and International Signed For™

Wednesday 4 December Asia, Far East (including Japan), New Zealand
Thursday 5 December Australia
Friday 6 December Africa, Caribbean, Central & South America, Middle East
Monday 9 December Cyprus, Eastern Europe
Tuesday 10 December Canada, France, Greece, Poland
Friday 13 December USA
Saturday 14 December Western Europe (excluding France, Greece, Poland)

UK Services  
Friday 20 December 1st Class
Monday 23 December Royal Mail Special Delivery Guaranteed™

We will also be shipping between Christmas and new year and as soon as the postal system opens again beginning of Jan. 

We will hopefully get another post up when I have a bit more time over the festive period taking stock of the year; what we've achieved and our aims and hopes for next year. Until then, here's a quick couple of photos from the office today:

emonTH assembled PCB's tested, firmware uploaded and ready to go
Trystan's been testing a setup with Raspberry Pi with RFM12Pi running emoncms on an external hard drive, potentially a great low cost solution for home emoncms microserver with robust logging and local storage...expect blog post and HDD image soon

emonTx V3 cases getting ready to case you're wondering, yes we did give that poor suffering plant in the background some water later today!

Trystan and the the dev bench

Myself and Trystan 'working', I should add that I'm doing 'Movember' , only a few more days to go, please donate, it's for a good cause

Printing postage for order parcels

Order fulfillment 

New hardware is in the shop and starting to ship this week

If you have been following the blog and twitter over the past couple of weeks you will probably have noticed that we've been gearing up for and starting manufacture (part 1 and part 2) of a trio of new hardware units:

1. emonTx V3

The emonTx V3 is our new flagship energy monitoring wireless node. Thanks to community contributions and user feedback we've improved on the previous through-hole emonTx design while trying to keep as much flexibility and user customisation potential as possible.

Full emonTx V3 system with 4 x CT-sensors an AC-AC power adapter: Real Power, Power Factor and VRMS measurements

The unit uses the same Atmel ATmega328 Arduino compatible microprocessor and all free I/O pins have been made accessible on a screw-terminal block for easy expandability.

Apart from the obvious change in enclosure and move to SMT pre-assembled electronics, the most notable hardware improvement is the addition of an AC-DC circuit. This enables the emonTx V3 to be powered from a single AC-AC plug-in adapter while simultaneously providing an AC voltage sample for VRMS and Real Power measurements.

emonTx V3 with ATmega328, RFM12B and on-board 3 x AA batteries

Instead of having a separate firmware sketch example for each function as we had with the emonTx V2 (e.g. Real Power (CT123_voltage), Apparent Power (CT123), Temperature etc.) we have for the emonTx V3 made progress combining all these into one 'auto-adaptable' piece of firmware. We have called it emonTxV3_discrete_sampling, it's  based on EmonLib but no longer relies on the internal bandgap of the Atmel chip for ADC measurements. This will increase ADC accuracy and reduce the need for additional calibration. The main features of the firmware at present are:

  • CT detection – only sampling from required channels
  • Power supply detection & presence of AC-AC adapter – VRMS and Real Power measurements are enabled if powering from AC-AC adapter and power saving mode is enabled of powering from batteries
  • Temperature sensor detection – external DS18B20 temperature sensors are auto detected and temperature readings added to measurements

The name 'discrete sampling' was used to draw distinction between the EmonLib based discrete sampling approach and Continuous Sampling. A Continuous Sampling PLL example is included in the emonTx V3 examples, after some testing we hope to integrate this into the main firmware: the emonTx V3 could switch to a more accurate continuous sampling mode when powered from AC-AC adapter of 5V DC USB.

The emonTx V3 is now available from the OpenEnergyMonitor shop. 

2. emonTH 

Wireless Temperature and Humidity Node

emonTH Temperature and Humidity Node

The emonTH is a new unit to replace the Low Power emonTx Temperature node (which was bit of a cludge!).

emonTH installed in my house, I will be keeping a close eye on temperature and humidity in my house over the winter in North Wales

Working with community groups like the Manchester Carbon Coop who are involved in retro-fitting houses with insulation we realised the need for an easy to deploy, long lasting temperature and humidity monitoring node. Many emonTHs can be deployed thought a house or building to inform a building performance model, heating control system or just for general interest! The emonTH supports DHT22 and DS18B20 sensors as well as simultaneous indoor and outdoor temperature readings using a remote DS18B20 sensor wired into terminal block.

As with the emonTx the data is logged via a web-connected base station to emoncms server for logging, graphing and analysis. As with all our hardware, the emonTH is open-source and Arduino IDE compatible using the ATmega328 with the Arduino serial bootloader.

The emonTH comes pre assembled and has an on board DC-DC converter to increase battery life.

The emonTH is now available from the OpenEnergyMonitor shop. 

3. emonTx Arduino Shield SMT

Arduino compatible energy monitor shield add-on. 

emonTx Shied SMT on Arduino Leonardo

For prototyping and lab experiences an Arduino is a great tool to quickly try something. The emonTx Shield allows energy monitoring functions to be added to a standard Arduino Uno / Leonardo or Yun as well as a Nanode RF. As with all our hardware units we provide as many software examples to help you get up and running. The emonTx Shield SMT uses Surface Mount (SMT) pre-assembled electronics. To keep costs down the through-hole components (headers and connectors) require manual solder assembly.

The emonTx Shield SMT is now available from the OpenEnergyMonitor shop. 

Hardware Manufacture Begins! Part 2: emonTx V3 and emonTx Arduino Shield SMT

This morning I visited Kaizen Technology in Royston near Cambridge. They're manufacturing the emonTx V3 and emonTx Arduino Shield SMT energy monitoring nodes. We choose Kaizen since we have worked with them before to supply us with components and as far as manufacture goes that have very good capabilities. They are able to perform wave soldering to automate the soldering of thru-hole components. This keeps the cost down on a board like the emonTx V3 where there are many thru-hole connectors. 

Since the emonTx Arduino Shield has got thru-hole components which are inserted from the rear of the board this could not be wave soldered. To keep cost down we will be shipping the emonTx Shield with only the SMT components placed and the thru-hole components supplies as a kit. The emonTx V3 will be fully assembled (SMT + thru-hole), apart from the RFu328 to retain user flexibility as some users might want to use an SRF instead of the RFM12B or even use a different MCU in place of ATmega328. 

Mm lots of components

Video showing solder paste being applied and pick-and-place doing it's thing

Checking the placements with the pick-and-place camera

Big re-flow oven, wave soldering machine on the right hand side
Manufacture of the emonTx V3 and emonTx SMT shield should be done by the end of this week and we expect to receive them next week. We hope to start shipping the pre-orders soon after:

Fully assembled emonTx V3, emonTx Shield SMT and emonTH

Complete emonTx V3 energy monitoring node

Hardware Manufacture Begins! Part 1: emonTH Temperature and Humidity Node

It's a busy for weeks for us at the moment. If you've been following us on twitter you will have see in the last week we have had the production PCB's designs for the emonTx V3, emonTx Shield SMT and emonTH (temperature and humidity) back from our manufactures. This coincided with manufacture of all three units starting this week.

Even though the designs have undergone several iterations and have been extensively tested over the past 12 months or so it was still a nail biting moment assembling, powering up and testing the final designs. Luckily it all went smoothly and all functions on all three designs seem to be working exactly as planned..just as well since we have 500 PCS of each!

Building up the final productions designs..scary! Thankfully it all went smoothly

Final designs for emonTx V3, emonTx Arduino Shield SMT and emonTH all working!
Last week I traveled over to Ciseco  in Nottingham to oversee starting manufacture of the emonTH Temperature and Humidity monitoring node. Ciseco have come a long way since I first visited them over a year ago. They've moved new offices, upgraded their pick-and-place machine and picked up a few more pairs of hands along the way. 

Here's a few photos of the manufacture of the emonTH:

Step 1: solder paste being applied

Step 2: Setup the pick-and-place to identify the 'fiducials' on the emonTH...Google it! If you look hard you will see these little shiny round marks on all SMT boards.

Step 3: practice placements

Step 4: let the pick-and-place do it's stuff! I love watching machines at work

Step 5 & 6: Reflow the board in the oven then add the thru-hole bits

Step 7: Test, test and test! Here's show's testing of the RFu328 (ATmega328 MCU + Radio) modules 

Step 8: Done!

Unfortunately we had a few issues during step 7; a some of the RF12B's which were reflowed onto the RFu328's were not working as they should. We think this was down to the large MCU package on the RFM12B absorbing more heat during the reflow process than expected. For the next batch we will look at tweaking the reflow oven's temperature profile. For now we will be hand soldering the RF modules. These sorts of niggles are not uncommon in manufacture, this is why testing is so important. We will be testing the units at several stages before they're shipped to ensure all functions are working. 

emonTx V3 final production design ready to go! 

The emonTx V3 and emonTx Arduino Shield are now in our shop on pre-order. The emonTH should be in the shop in the next few days (just waiting to tie down pricing). We hope to start shipping at the end of this month (November), so hopefully everyone who want's one should get them in time for Christmas. Documentation for all the units is slowly coming together, please bare with us during the next week or so; we hope to have all the documentation and open-source designs ready for we start shipping. Likewise it would be most helpful if you spot any typos (or screaming errors!) in the shop description or wiki documentation please let us know. If you want to take a look the Arduino compatible firmware code for all three units has been pushed to GitHub.

I'm currently on on the train down to London ready to travel to Cambridge tomorrow to meet with another manufacturer and begin manufacture of the emonTx V3 and emonTx Arduino Shield. Stay tuned for another post in the next couple of days! I will also try and tweet some sneak-peak photos of the manufacturing process as it happens tomorrow. 

Documentation highlight: Application guides

To try and bring the work on the application guides more to the fore, I've rearranged the OpenEnergyMonitor Getting Started Guide bringing the application guide list up to the top of the page.

There you will find Robin Emley's documentation on his MK2 PV diverter and Martin Robert PLL PV diverter documentation. I have also made a start on updating the heat-pump application guide and writing a guide on building thermal performance monitoring.

Here's the full application guide list so far:

Home Electricity monitor
Solar PV Monitoring
Diverting surplus PV Power, by Robin Emley
Solar PV power diversion with emonTx using a PLL, emonGLCD and temperature measurement, by Martin Roberts
Building thermal performance monitoring and modelling
Heatpump Monitoring
Solar Hot Water monitoring and control

Documentation highlight: Solar PV power diversion PLL

I would like to highlight the recent addition of some really great documentation written by Robert Wall and Martin Roberts detailing Martin's Solar PV power diversion implementation which uses an emonTx running PLL based firmware,an emonGLCD and temperature measurement.

You can find it here
and its linked through from the main Getting Started Guide.

Improving emoncms performance with Redis plus interesting consequences for SD cards.

As part of recent work to improve the performance of emoncms because of high load's on there is now an emoncms branch that uses redis to store feed and input meta data including last feed time and value fields which where causing significant write load on the server.

Using redis in this way leads to quite a big performance improvement and potentially could lengthen the lifespan of raspberrypi SD card systems significantly.

I've been working on the redis implementation with Ynyr Edwards a good friend and an experienced software developer who recently joined Glyn and I helping us with development and running the shop. Ynyr had been telling me about the usefulness of in memory databases and caching for improving performance for long time. There appeared to be a significant amount of waiting on io going on on and the mysql processlist was full of last time and value updates to the feeds meta data table.

In order to compare the use of mysql versus redis for storing input and feed meta data in emoncms a test was created that was representative of the typical kind of data input seen in emoncms.

The test consisted of a node posting 3 power values, with each value being “logged to a feed” processed into kwh/d data and histogram data. So 9 feeds in total, three of them timestore and 6 mysql based.

The node post rate was set to once a second and the time taken for each request was measured. After single request time's where measured a second test was carried out which involved sending request continuously and measuring the time taken to make 100 sequential requests from which the average requests per second value is determined.

Its important to note that the following results are for sequential requests rather than concurrent requests. Its possible to achieve significantly higher request rates with concurrent requests which spawn many parallel apache processes. Sequantial requests give us a good base line test to work with.

The CPU on the test machine was set to 2.0 GHz x 4 cores

Testing Mysql

The following results show the effect of turing off last input and feed time and value meta data entries in the pure mysql implementation:

Mysql with all metadata switched on:
10x sequential posts, 31ms to 79ms @ 4x 2GHz, average 20 sequential req/s.
With the processor set to 0.8GHz on all 4 cores the request rate was 10 sequential req/s

With a single poster process free-running the CPU is 7.4 us and wait is 11.5 wa

Mysql without input last time value being saved:
10x sequential, posts 26ms to 64ms, average: 25 sequential req/s

Mysql without input or feed last value but still histogram last value:
10x sequential posts: 18ms to 31ms, average: 46-48 req/s

Mysql without input or feed last value or histogram last value:
10x sequential posts: 10ms to 11ms, average 94-95 req/s


Here are the results with the redis implementation that's up on github here:

All meta data enabled 11-14ms, 94-95req/s @ 2.0GHz
CPU us 22us, 0.0wa

CPU Performance:
In the redis test the CPU utilisation was around 22%, the mysql CPU utilisation was around 7.4%. Idle CPU us is around 0.2%. Redis is however handling 4.7x the number of requests, if it where handling the same number of requests the cpu us may be around 22/4.7 ~4.7% us.

We can see a significant reduction in the amount of time spent by the system waiting. With mysql every time a feed was updated, the time and value of the update was written to the mysql feeds table. The first idea was that this waiting was caused by waiting on mysql table locks, testing however with both MYISAM and InnoDB showed similar overall performance even through InnoDB is row locking while MyIsam table level locking. Looking at the disk write rate with vmstat and iotop however showed a really high write rate with mysql so it may be that the waiting was just waiting because the disk was working so hard, see more on this below.

IO Disk write rate:
Here is the output from vmstat with the apache access and redis logs turned off.


procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

2 0 0 4282700 352236 2002508 0 0 0 0 1670 44128 20 7 72 0

1 0 0 4281496 352236 2002976 0 0 0 43 1622 44148 20 7 73 0

1 0 0 4280636 352236 2003440 0 0 0 0 1651 44424 21 7 72 0

1 0 0 4280096 352236 2003952 0 0 0 1793 1678 43999 21 6 72 0

1 0 0 4279316 352236 2004416 0 0 0 49 1658 44106 20 7 73 0


procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

1 0 0 4264420 352244 2041008 0 0 0 4280 1695 14833 7 3 79 11

1 0 0 4264904 352248 2039972 0 0 0 4296 1777 15615 8 3 78 11

1 0 0 4264664 352252 2040312 0 0 0 4477 1713 15072 7 3 79 12

1 0 0 4264440 352264 2040420 0 0 0 4355 1700 14982 7 2 79 11

0 1 0 4267364 352284 2037604 0 0 0 4332 1712 15012 7 2 79 12

Using a larger number of vm readings than the 5 listed above the average redis input was 215kb/s and mysql 4430kb/s, idle being 4kb/s. First its surprising how high the mysql write rate is and then second its surprising how large the reduction in the amount of disk writing done is, about 21x less and that's with 4.7 times the post rate. The reduction in the amount of writing could therefore be as much as 100x. The quanitiy of writes cannot only be explained by the kind of writes that are being done in emoncms, most of it appears to be due to ext4 filesystem journaling (61.16% of write capacity) while mysql is only responsible for a couple of percent.

MySQL iotop
Redis iotop 

"A journaling file system is a file system that keeps track of the changes that will be made in a journal (usually a circular log in a dedicated area of the file system) before committing them to the main file system. In the event of a system crash or power failure, such file systems are quicker to bring back online and less likely to become corrupted" -

Raspberry PI SD card installation
The problem with running emoncms on SD cards was that we where wearing the SD cards out in only a few months. If redis reduces the amount of writing by around 100x then this could mean a significantly longer SD card life span.

If in addition to using redis a filesystem without journaling is used the lifespan could be extended even further, although this does increase the risk of data corruption from power failure, but then if such failures are recoverable with a disk check on startup then that’s much better than a worn out SD card.

To potentially improve things further the IPE Debian operating system could be put on a read only partition as is currently done with the oem gateway and the data could be placed on a write partition which could use the ext2 or fat32 filesystem both of which are non-journaling. 

Redis branch on github
If you'd like to try out the redis branch its up on github here:
You will need redis-server installed and phpredis: