New emoncms features part 1: blue message bar

Spent a bit of time implementing new features for emoncms today, the first one was a much needed but quite small addition: a themed location to output messages such as "user already exists" and so on, see the blue bar in the screen below:



Try it out here: http://vis.openenergymonitor.org/emoncms3/

How it works

If you have a look at the user_controller here github user_controller.php
On line 28 and 29 you will see the following:

$output['content'] = "";
$output['message'] = "";
(ie line 65) $output['message'] = "Your new account has been created";

The returned output from the controllers used to be just the one variable $output now we have two, one for main body content output and the other for message bar output. These variables are passed through index.php to the theme The theme then places the message and content in separately themed locations.

Community Energy Plan maker

Over the last year Glyn Hudson, Bethan Gritten and I have been running a community energy project with our local sustainability group Ecobro in the Penrhyndeudraeth area of North Wales. The aim of the project is to create a comprehensive snapshot of our current energy use as a group that we can then use as a solid foundation for creating a sustainable energy action plan.

As part of the project we have installed 20 openenergymonitor energy monitors and have also developed for the project a community energy auditing and plan making application which is what this blog post is about.

This is how the application works:

1) Fill out a single household energy audit, compare how your energy picture compares to the average UK household energy use and how it relates to the Centre for Alternative Technology Zero Carbon Britain 2030 energy scenario:


2) Aggregate data from each household to create a group view:


3) Explore how group scenario is built up of individual households contribution:


4) Explore future energy scenarios, see the effect of switching to electric cars, heatpumps etc on real household data from your group:


5) In the next version, see effect of energy efficiency measures, insulation, draft-proofing and see the cost and payback details of scenarios including a comparison of different measures by sustainable energy % increase impact per £.

The idea is that through this process we can help bring focus to areas and solutions that can have most impact and also give a tangible reference of progress so that milestones can be celebrated when achieved.

The application is largely inspired by David Mackay's book Sustainable Energy Without the Hot Air (free to read online) as you can see from the use of the energy stacks. The aim is also very much about creating a tool that helps to create sustainable energy plans that add up and in the spirit of openenergymonitor and David Mackay's book the application is open source.


Try it out:

If you would like to help with development that would be great, please get in contact / send us github pull requests. If your doing a community energy plan making project and would like to use this tool, please do and let us know how your using it. 

University campus energy monitoring in France

Hello, my name is Baptiste GaultierWhen I started working on the project of monitoring the energy consumption of Telecom Bretagne (an engineering school in Brittany, France), I was looking for products which can measure, store, and track the energy consumption data of a typical university campus.

After spending some time researching the available solutions, I found a lot of expensive proprietary products and a project you may have heard of, called openenergymonitor. This project provides a very powerful, web-connected energy monitoring system, and the best part is that it is open-source !

I started to work on top of emonTx and emoncms and here are some new features I added to emoncms (the new hardware features will be addressed in a future blog post)  :
  • Map widget : allows you to see a map with all the sensors used for energy monitoring
  • Device manager : provides an easy way to add devices on the map widget
  • Node information widget : allows you to see all information (hostname, IP addresses, device type, power measured...) of a monitoring node

Screenshots

Dashboard with my university campus map



Device manager tab


We have made some progress, but we have still a lot of work to do before our goals are met, most notably:
  • Tagging and renaming the devices
  • Displaying the energy chart when clicking on a node
  • Drag and drop node positioning on a map (device manager)
  • Node filtering (depending on power consumption for instance)
  • How to use the map widget (documentation) 
The code has been released on github (I forked couple of days ago) : https://github.com/bgaultier/emoncms3/tree/devicemanager

The entire team working on the project at Telecom Bretagne is convinced of the great quality of the openenergymonitor project and we hope our work may be useful in any way.

Cheers !

emonGLCD - getting time for the internet

The emonGLCD is an open-hardware wireless graphical LCD display based on the ATmega328 microcontroller and is Arduino compatible.



The emonGLCD works very well as a real-time living room energy monitoring display. For the emonGLCD to calculate and display how much energy has been used on the current day it must know when the day begins and ends. Therefore it must know the current time.

The emonGLCD does not have a hardware RTC but its possible to implement a very accurate clock using the Jeelabs software RTC library and a web-connected Nanode RF. In the OpenEnergyMonitor end-to-end energy monitoring system the emonGLCD is used in conjunction with the emonTx (transmitter) and emonBase/NanodeRF which is used to post data on-line to emoncms.

When data is posted to the emoncms sever with a http request the sever response contains the current time of the sever. We have written a code example to extract this time from the http header reply and pass it via RFM12B wireless to the emonGLCD. The time on the emonGLCD is always correct. Both the emonGLCD and the NanodeRF receive power monitoring data from the emonTx

Demo video (single CT sensor system):



Demo video (solar PV monitor system):



Taking this a step further: when the emonGLCD receives the time from the NanodeRF the emonGLCD transmits back the current room temperature from it's on-board temperature sensor, then the NanodeRF posts this online to emoncms.

The Arduino sketch examples are up on github:

emonBase/NanodeRF:
https://github.com/openenergymonitor/NanodeRF/tree/master/NanodeRF_singleCT_RTCrelay_GLCDtemp

emonGLCD single CT:
https://github.com/openenergymonitor/EmonGLCD/tree/master/emonGLCD_singleCT_AutoTime_TempLog

emonGLCd solar PV: https://github.com/openenergymonitor/EmonGLCD/tree/master/emonGLCD_solarPV_type_1_and_2

Full web-connected home energy monitoring system

The mission of the OpenEnergyMonitor project is to design and build and open-source energy monitoring and eventually control system. We would like this system to be work well for home energy monitoring systems but also be scalable. For more information see our newly updated project vision.

We are now at the stage were we have a fully functional end-to-end web-connected home energy monitoring system. We have recently written a full step-by-step build guide for the system. This build guide covers everything from assembling the electronics, uploading the Arduino firmware to setting the web dashboard. 



The system consists of three wireless hardware units emonTx (transmitter), emonGLCD (display), emonBase (web-connected base station) and a powerful web-application emoncms.

We also have build guide/documentation for setting up a solar PV monitoring system using the same modules. 


Solar PV Monitoring



I have just finished detailed documentation of an OpenEnergyMonitor Solar PV Monitor system see: http://openenergymonitor.org/emon/applications/solarpv

The system monitors both generation and consumption and gives the user a clear indication of when their household electricity demands are being met by their solar PV array (green light on display) or when their not (red light on display). The display also shows how much electricity is currently being exported or imported. 

The system is web connected, posting data to emoncms for logging and historical visualisation.

It has been suggested that installing a solar PV system results in decreased energy consumption and consumption pattern changes due to increased awareness of energy consumption verses generation. To enable these effects to take place the home owner needs generation level put into context with consumption level, in real-time. The OpenEnergyMonitor Solar PV monitoring system does just this, by simultaneously monitoring and displaying both generation and consumption. Most energy monitors currently available either monitor generation or consumption, not both.

The system is based on the emonTx, emonGLCD, emonBase and emoncms. All hardware and software are open-source and have been developed by OpenEnergyMonitor and contributors. 

We would like to be able to offer the system pre-assembled in the future. 

Please use the forums for discussion. 

Here's a bit of background to solar PV monitoring in the UK:

The Feed in Tariff scheme (FIT)currently running in the UK pays the home owner a fixed amount per kWh generated, the home owner also gets paid a much smaller amount per kWh exported. At the moment (Dec 2011) most homes in the UK don't have export or smart meters fitted, utility companies assume, for calculation purposes that the home owner exports half of the power they generate. The cost of importing power from the grid is higher than the amount paid per unit of exported electricity. Therefore it makes financial sense for the home owner to use as much of their generated power as is possible.

A monitoring system with a real-time living room display indicating the amount of power being imported/exported allows the home owner to attempt to match their power consumption to the power being generated. I.e. make tea when the sun is shining! Financial benefits aside it can be very interesting and satisfying to monitor the yield of your solar PV system. 

New emonTx PCB

Introducing a new emonTx PCB design V2.1. No major changes just incremental improvements. Onwards and upwards!

The main changes are the addition of a footprint to enable a DS18B20 digital-temperature sensor to be soldered directly onto the board. The can be used instead of connecting an external DS128B20 temperature sensor using a 3.5mm jack.

The other change is the addition of a voltage select solder jumper for the IRQ pulse counting input which also doubles as a digital I/O port (Arduino Dig3). This solder jumper must be connected to either 3.3V (emonTx internal voltage) or 5V (if emonTx is powered via 5V usb). Routing 5V to the Vcc pin on this pulse IRQ port is useful when interfacing directly with pulse-output utility meters or controlling 5V relays. 3.3V works fine when using a TSL257 optical sensor to detect pulses from a pulse output LED. 


Front
The other changes are purely aesthetic, a nice big Welsh dragon to recognise where Trystan and I are based up in the mountains of North Wales. We are very proud that our PCB's are designed and manufactured in the UK using an environmentally friendly process. 

We have also added an area to note which frequency RFM12B modules has been soldered on, since once soldered on there is no way to telling! RFM12B frequency has become more important with the release of the NanodeRF which comes shipped with 868Mhz modules. OpenEnergyMonitor has in the past used 433Mhz modules; we now use modules of both frequencies. It's important to remember to use the correct length antenna for the frequency (details here) and set the correct frequency of the modules in the sketch. 

Rear

We are on schedule to open an OpenEnergyMonitor online shop in the new year. If you're interested getting your hands on an emonTx then please fill in the interest form and we'll get back to you: emonTx interest form  

open-hardware emonTx PCB designs are up on solderpad

The priority in the new year is to get custom encloses manufactured for the emonTx. Currently we have been drilling our own holes in a generic case for the connectors. If you can help us or have experience in designing and manufacturing electronic enclosures please get in touch. 


Likewise if you have any ideas for improvements or new features for the emonTx then please start a discussion in the hardware section of the forums. We love to hear all ideas. 

Merry Christmas everyone!



How to contribute on OpenEnergyMonitor dev + development plans


If your wondering how best to get involved with OpenEnergyMonitor development here are a couple of thoughts and possible pointers.

First a few insights and ideas from an another open source project we follow: DIYdrones. Thanks to Amin Zayani for the heads up on these: [1] [2] [3]
In Chris Anderson of DIY drones's words, who we think has articulated it well:
“The best way to participate is to do something cool on your own and share it here. All of our code bases and hardware design files are open. If you see an opportunity to improve something, just do it, post it and then tell the community about what you've done. If it's good, people will use it, help improve it, and we'll get a sense of what you can do. That makes it much easier for us to figure out what your skills are and where you'd fit in best.”

If your looking for ideas here are a list of things we would like to do but no one is currently leading on them, Many of these problems are quite large, it would probably good to break them down into smaller sub projects. If you have a feature you would like to implement or would like to take the lead on any of the features below that would be really great, please post with your thoughts.
  • Appliance inference - infer from a single power feed of a house what each appliance uses and so providing a energy use by appliance breakdown.
  • PCB and casing for the 12 input pulse counter (a DIN rail and non din rail option)
  • Plug based energy monitor - for per appliance monitoring  - an alternative method to appliance inference.
  • Larger monitoring PCB's: maybe 12 input CT?
  • 3 phase monitoring sketch for emontx - quite a few people are asking for this so this would be really useful.
  • Develop a series of control modules, that extend the emon system into an open source monitoring and control system that can be used for domestic and industrial situations. Control modules could include:
    • multiple relay board (for dump load control, immertion heaters, pumps, lighting)
    • solenoid valve driver board for central heating control.
    • motor control
    • This emon monitoring and control system could link together by either a wireless bus say rfm12 or a wired bus (maybe rs485). The modules could be designed to fit in both DIN rail style casing or non-DIN rail casing.
  • Merge emonTx firmware examples into an Arduino library.
Short term goals
For a general overview here are the short term goals that are being worked on at the moment and in the very near future:
  • New casing design for emontx and emonbase
  • Transfer emonbase sketches over to EtherCard library
  • Compatibility of main sketches with Arduino 1.0
  • Complete watchdog addition
  • Emoncms development see feature plans and discussion page on forums.
  • Complete and publish energy audit software v1.0
  • Improve full emon system documentation.
This blogpost is repeated as a forum post here, lets discuss this on the forum as its a better place for it.

Thankyou
Trystan


How to move feed data between servers

There is often a need to transfer large feed tables between different servers. Here is one way to do it via ssh that does not require downloading the data to a local machine. This makes it particularly fast especially if your local internet connection is relatively slow.

1) Login via ssh to the source server
ssh [email protected] (source server ip)

2) Export feed table from source instance database
mysqldump -u username -p database feed_1 > feed_1.sql

3) Transfer feed table via ssh to destination instance server
tar -cjvf - feed_1.sql | ssh [email protected] tar -xjf - -C /home/user (destination server ip)

4) Login via ssh to the destination server
ssh [email protected] (destination server ip)

5) Import feed table into destination instance database
mysql -u root -p database < feed_1.sql

It would be great to have some kind of utility within emoncms that makes it possible to move large feed tables between emoncms instances efficiently and easily without having to use terminal and ssh. If you know of an efficient way of doing this, advice would be most welcome.

Open source and software as a service

In addition to developing emoncms as an application that can be downloaded and installed on any server, interest willing, I would like to work towards offering emoncms as a service. So if you just want to build up your hardware and connect to a ready to go, maintained service with support, the option will be available to you.

One of the reasons I wanted to develop emoncms in the first place was that I was frustrated with fact that all available services where proprietary, I wanted to be able to download, install, modify, take part in development because I very much enjoy doing so and wanted to take part in collaborative innovation on important problems as Irving Wladawsky-Berger described so well in his post on the essence of open source which I blogged about last week.

As this article by mygnulinux.com and another article here by William Judd describe, there is a concern that as more an more of our applications move into the cloud and are run by organisations who adopt the proprietary software as a service business model we will loose the ground that we have gained in the open source software movement, there is also a complication to do with licensing.

There are however a great deal of advantages in having applications in the cloud many of which are also very beneficial for energy monitoring, such as: access to your data from any location and any device and possibilities that open up by using shared platforms.

The answer: cloud based software as a service does not have to be proprietary software, it can be open source and this opens up some really exciting opportunities.

So the idea is that the installed web application's code is placed on github or other code sharing location and is a mirror of the installation on the main server/servers. The service can be used as one would normally use a software service. However if you want to improve the code, modify, add something you can. You just need to fork the repository, do something cool, share it with the community and if its adoption is agreed it can be added to the service install. So this allows for taking part in development, for modification and improvement. It is a little more complicated than editing your own instance of a particular piece of software as changes need to be agreed on by those using the service as any changes will effect everyone, but this is not a prohibitive problem as there are systems that have already been worked out for determining what makes it into the final release in many large open source projects.

As mygnulinux.com and William Judd's articles explain, the other important problems in software as a service is: lock in, data ownership and licensing.

The first can be addressed by creating a comprehensive API that makes it possible to move data from one service to another or to a local installation if one desires to do so.

The second can be addressed through good policy by the service providers.

Licensing is important. The GPL up to and including v3 does not ensure openness for software as a service provided over a network as the requirement to release the source code and any modifications to that source code is only triggered when the software is distributed not when that program is run. This essentially as I understand turns the GPL into BSD like licence as there is no requirement for open sourcing when providing the software as a service across a network.

In response to this loophole the Free Software Foundation bought out another licence called the GPL Affero licence (AGPL) which essentially requires anyone providing a service using the software across a network to release the source code, the one additional provision over GPL v3 being:
“if you modify the Program, your modified version must prominently offer all users interacting with it remotely through a computer network (if your version supports such interaction) an opportunity to receive the Corresponding Source of your version by providing access to the Corresponding Source from a network server at no charge, through some standard or customary means of facilitating copying of software.”
In conclusion
It's possible to get the benefits of open source and the benefits of the cloud and software as a service platforms. I think its going to be exciting to explore this with emoncms.
  • The code for the latest version of emoncms is available on github: https://github.com/openenergymonitor/emoncms3
  • Its the exact same version as is deployed here: http://vis.openenergymonitor.org/emoncms3/
  • It is licensed under AGPL
  • A comprehensive API and tools for easy porting of data between instances is a core aim and will be an important part of continued feature development.
  • A template terms of service needs to be worked out and written.
Examples of open source software as a service
SugarCRM - a customer relationship management package (From the articles linked above)
Identi.ca - a micro-blogging service (From the articles linked above)
Thingspeak - internet of things application

Drop a comment below if you know of others that can be added to the list.