RaspberryPI: SD Cards, HDD, Gateway Forwarder

After several Raspberry Pi emonBase SD card failures between myself and Glyn in the last two weeks and the general experience of short SD card lifespan on the forums (SD cards have a limited number of writes), we thought we'd make a concerted effort this week to move the openenergymonitor Raspberry Pi documentation, pi images and pre-installed SD cards over to the more stable solutions:

Raspberry Pi emonBase with RFM12Pi in Pibow Timber Case

1. We read Martin Harizanov's blog post on creating a rock-solid gateway using a read-only filesystem with Jerome Lafréchoux's excellent python oem_gateway to forward data to emoncms.org

This is a reliable solution which is simple to setup and works well if you just want to forward data to a remote emoncms server like emoncms.org. We have created a Raspberry Pi SD card image for this read-only filesystem oem_gateway setup, see documentation page for full details, there's a link to download it there too:


The ready-to-go pre-loaded SD card available in the shop will be pre-loaded with this image from now on. 

2. We also heard from Paul Reed about his setup where the SD card is just used to boot an external hard drive (powered by a USB hub) with the Pi's root partition running a web server and the full version of emoncms on the external hard drive. This solution is great if you want to keep your data locally or have an additional backup. This could also double up as a 24/7 file server running in your home for music streaming, document backup etc.

Glyn's using the read-only gateway for his setup and I'm going to setup the hard drive local backup in addition to forwarding to emoncms.org for mine.

Here's a diagram that illustrates these different options, including the NanodeRF, local and remote storage, backup options and emoncms data storage options (mysql, timestore etc):
If your not sure which way to go, its probably easiest to start with the Raspberry Pi oem_gateway forwarder to emoncms.org, you can re-configure it for local storage and backup via an external harddrive if you want later. If your more comfortable with Arduino code than the Raspberry Pi and linux then the NanodeRF (pre-assembled SMT) may be best for you.

To summarise the above with pros and cons, here are the two main emonBase Raspberry Pi options:

1. Run a read only filesystem on the Pi and forward data straight to a remote emoncms server.
Developed by Jerome Lafréchoux and Martin Harizanov's
http://harizanov.com/2013/08/rock-solid-rfm2pi-gateway-solution/
  • + More robust and easier to setup
  • + Cheaper and lower power than using an external HDD
  • - Does not utilise the full potential of the Pi
  • Run IPE – Industrial Perennial Environment a special blackout-proof flavour of Raspbian which can be locked down after setup to work in read only mode
  • The Oem Gateway Python script by Jérôme Lafréchoux to forward the data received by the RFM12Pi straight to emoncms.org
2. Move the Pi's file system to an external HDD using the SD card only to boot 
This option requires an external hard drive connected to the Pi through a powered USB hub. This does require extra expense and an increase in power consumption, the hard drive we tested here a 1TB USB hard drive adds 1.8W to the power consumption of the Pi (3.9W). However this option does have several advantages:
  • + Ability to run emoncms on the Pi to log data locally and have full control over your setup as well as forwarding data to a remote emoncms server.
  • + You now have the potential to have 24/7 file server running in your house for music streaming document backup etc.
  • - More complicated to setup
  • - Higher cost and power consumption

Measuring building thermal performance - coheating tests

Improving the thermal performance of buildings is an area where some of the largest energy and carbon savings can be made. Building energy use and carbon emissions can be reduced by as much as 60-80% through better insulation, draught proofing (improved thermal performance) and heating efficiency and controls. But how do you go about working out the performance of your house and what measures are best to undertake to reach this level of performance improvement? Improving building fabric is expensive, how do you work out which measures will be most cost-effective? and how do you make sure that your house actually achieves the target performance? how do you measure and check it?

These are the questions I’m currently grappling with for my own house and the openenergymonitor lab, these are also the questions we’re trying to develop improved processes for answering with our work with Carbon Coop.

Here’s a graphic from the Centre for Alternative Technology’s Zero Carbon Report illustrating the kind of performance improvements that are possible:


The most common way to investigate domestic building performance is by using simple building energy models such as SAP:
http://openenergymonitor.blogspot.co.uk/2013/06/building-energy-modelling-carbon-coop.html

The accuracy of a model is always dependent on its input data and assumptions, work that has been done on comparing modelled energy consumption as calculated by SAP vs actual energy consumption show that there is often a discrepancy and in some cases the discrepancy is so large (even a 100% or more) as to undermine decisions made based on model outputs.

https://www.bsria.co.uk/download/asset/agm-2011-co-heating-tests-sarah-birchall.pdf
To rely therefore on modelled performance only can be misleading.

It is possible however to measure the total building thermal performance by measuring how much energy it takes to heat a building up to a given temperature above the outside temperature. This procedure is known as a coheating test and was pioneered by the Center for the Built Environment at Leeds Met University. (There's also a good info sheet on coheating tests here by Peter Warm www.peterwarm.co.uk/?dl_id=6 )

The standard co-heating test involves heating building when unoccupied to an elevated internal temperature of 25C over a period of 1-3 weeks with electric heating and monitoring the electrical heat input, internal and external temperatures. Keeping the building unoccupied reduces unknown variables and so increases the accuracy of the measurement but it alongside the elevated temperature makes widespread testing of this kind difficult.

The question that several people have been asking and I’m aware of several groups working on this (@Housahedron and Richard Jack, lboro) is; can a method be developed for undertaking an ongoing co-heating equivalent test that can be undertaken while the building is being used, that could even measure thermal performance over time as measures are undertaken. The analogy is a car MPG meter but for your house.

Over the last few weeks I’ve been working on an approach that I think is showing promising results, it involves applying a dynamic model to realtime monitored heating input and external temperature data to model indoor temperature. The modelled internal temperature is then compared to the actual indoor temperature.


The model parameters that give a good match tell you the heat loss factor of your building and also it’s thermal mass. The heat loss factor is your MPG equivalent for household fabric thermal performance.

The model type is a multi stage resistor-capacitor model, using the resistor-capacitor analogy for thermal modelling is a standard approach used in many thermal models, there's an interesting page on it here: http://lpsa.swarthmore.edu/Systems/Thermal/SysThermalModel.html


This is all open source and the code so far is up on github here: https://github.com/emoncms/openbem,
There's also a forum thread here with a bit more information on the model and tests http://openenergymonitor.org/emon/node/2783

Faire Maus! A 'fair' USB optical mouse

A few weeks ago I wrote a blog post about ethical and sustainable electronics.

In the post I mentioned a German project called Nager IT who have designed an optical USB mouse using low carbon materials and have gone to great lengths to source ethically produced components with a transparent supply chain and exploitation free manufacture.

We have decided to support the Nager IT project by becoming a UK reseller of their 'Faire Maus' or fair mouse. This mouse is as good as it gets at the moment in terms of ethical and sustainable electronics.

The mouse is a well-made three-button USB optical scroll wheel mouse made using low carbon materials, ethically produced components with a transparent supply chain and exploitation free manufacture.

The mouse is now available for purchase on a non profit basis (for us) in our shop: http://shop.openenergymonitor.com/fair-usb-optical-mouse/





The mice are assembled by hand in Germany and feel very well made, the texture of the wood plastic enclosure feels good in the hand and the sleek and rubbery USB cable runs smoothy over the table.



Every time I use the mouse it it reminds me that ethical and sustainable electronics is a goal which is worth pursuing.

Purchasing a fair mouse can be thought of like purchasing fair trade coffee; it might cost a little bit extra but you know that your doing good.

Checkout the great graphic Nager IT have produced illustrating their transparent supply chain behind this mouse.




Posting data from Adafruit Tweet-A-Watt to emoncms

Tweet-a-watt is a project from Ladyada / Adafruit to modify a Kill-A-Watt appliance power monitor to transmit the power data via Xbee to a computer (could be a Raspberry Pi) and post the value to twitter. Chris Whiting has modified the python code to post the data to emoncms







Chris Whiting writes: 

A Tweet-A-Watt is plugged into an outlet with a device plugged into it. A xbee is wired to the Kill-A-Watt electronics, which reads an analog voltage representing the power and current being used by the device plugged into the Kill-A-Watt. The xbee sends the data to a receiving xbee which is plugged into a computer using an USB FTDI TTL-232 cable.

I have a script called "wattcher.py" running on a Raspberry Pi which was originally developed by the folks at Adafruit. This script reads the data sent to the receiving xbee through the serial port. The script takes the raw data and processes the data to calculate the power being used. I modified the script to send the collected data to a web socket. The receiving web socket is the oem_gateway.py script that's running on my web server where Emoncms is currently installed.

Chris has kindly put his cod on GitHub for everyone to try: https://github.com/casestudies/tweetawatt



emonTx V3 Progress Update

Since my last post introducing the emonTx V3  prototype design progress has been steady. The design has undergone several iterations and has now reached a stage of maturity.

The main features of the emonTx V3 have stayed the same:
  • ATmega328 Arduino IDE compatible microcontroller
  • RFM12B or Ciseco SRF 433/868/915Mhz RF wireless compatiable with RFM12Pi Raspberry Pi emoncms basestation
  • 3 x Standard (23kW max) CT channels
  • 1 x High sensitivity (4.5kW max) CT channel 
  • Integrated AC-DC power supply to enable powering the unit from a single 9V AC adapter while also sampling the AC voltage to calculate Real Power and AC Vrms readings 
  • Low power design with option to power from 3 x AA batteries for Apparent Power (current ony) measurement
  • Enclosed in wall mountable extruded aluminium enclosure
  • Terminal block connection for optical pulse sensor and DS18B20 temperature sensors
  • Pre-assembled SMT electronics
emonTx V3 with 3 x AA battery's and 1 x CT (Apparent Power setup)
Fully assembled with antenna in wall-mount enclosure

I am currently in the process of obtaining assembly quotes from manufactures as well as performing lots of testing.

emonTx V3.1 PCB Design 
emonTx V3.1 Schematic

The AC-DC circuit that was initially designed with the aid of simulation then bench tested is performing as expected.

Blue = output from 9V AC adapter Yellow = input to voltage regulator when unit is drawing 7.7mA  @ 3.3V, sudden dip is caused by RFM12B firing up to transmit four integer data packets (approx 24mA for 2.7ms)
If all goes well we're expecting to get the emonTx V3 into production in the next few of months.


emonGLCD 433 / 868 Mhz RF Scanner & Signal Strength Meter

Martin Roberts has developed a fantastic bit of firmware for the emonGLCD to enable it to be used as an RF scanner and signal strength meter on the 433 / 868 Mhz RF band.

 This makes the emonGLCD running this firmware a useful tool for debugging RF transmission and checking out signal strength. To run the firmware a small hardware modification is required to the emonGLCD to access the analogue signal strength output signal from the RFM12B.

Martin's sketch has been added to the emonGLCD github repo.

Full details including build-guide and discussion can be found on the original forum thread.






Notify on inactive feeds

Here's a simple feature that sends an email whenever emoncms feeds become inactive for more than two hours, designed to stop those instances where you login to emoncms after a month and realise that something went wrong and it hasn't been updating for a while.

The notify control page can be accessed via the Extras menu. It can be disabled or enabled and an email address that is different from you main account email can be used if needed.

A cron job runs every 2 hours on the server to check if any feeds have become inactive for more than 2 hours but less than 4 hours. This ensures only one email is sent and historical inactivity is ignored.

Notify is now running on emoncms.org, you may want to check that your email address is correct before turning it on.

To install notify on your raspberrypi or own server see installation details here:
http://github.com/emoncms/notify



PCB Manufacturer Meeting

A couple of months ago I met up with Stacey Driver who is the Business Development Manager of PCB manufacturer Stickleback Manufacturing Ltd. We have been working with Stickleback for several years now, over that time they have grown in size and moved to a new factory. It's great to support (in the small way we can) a UK based manufacturer.

Here's a copy of Stickleback's recent newsletter where our meeting is mentioned alongside details of their new water treatment plant they have recently installed. It's great to hear that they are continuing to invest in their manufacturing processes and reducing their environmental impact. This links in well with my previous blog post on Ethical and Sustainable Electronics 


Stickleback Chat - Official news of Stickleback PCBs

Image

Water Recycling Treatment Plant

As part of our continuing energy efficiency scheme we are thrilled to announce that we have installed a water recycling plant for our factory in Rochester, Kent. The introduction of the unit will significantly reduce the volume of waste water produced as a direct result of our processing.
Acids and alkalines used in our manufacturing processes are neutralised under controlled conditions. The ph. of the solution is carefully controlled allowing the water to be reused. This reduces the volume of water drawn by our company and removes the possibility of any polluting effects.
More and more companies are investing in waste water treatment. For Stickleback, we see it as part of our corporate responsibility to preserve and protect the natural resources we have wherever possible. We can’t wait until it is all put together and up and running.
Image

Open Energy Monitor and Google Campus

Stacey Driver had a fantastic meeting with our long-standing, valued customer, Glyn Hudson, at the Google Campus.
Glyn is a highly ambitious and motivated young entrepreneur, who is proud to be manufacturing a British product. The Open source hardware product, manufactured by Open Energy monitoring, allows us to relate our use of energy to our energy systems and embrace the challenge of sustainable energy.
We’ve worked with Glyn from the prototype stage through to production and his quantities continue to grow as his products are sent worldwide.
We are proud to support growing business’; start-ups and SME’s and have developed smart prototyping systems to help minimise the required outlay at these costly development stages.
Contact Stacey today on [email protected] or 01634 272416 if you think we could become part of your businesses fantastic journey.
We would be delighted to discuss where Stickleback may be able to help!



Ethical and Sustainable Electronics

Like many of us I love technology and enjoy using the latest gadgets, but this often sits uncomfortably on my conscience since I also care deeply about the environment and try and do my best to reduce my energy consumption and carbon footprint where possible.

The choices we make as consumers do make a difference, the stuff we buy accounts for about 40% of our current energy consumption, this is for the average person, it's probably a fair bit higher for 'techies'! The energy used to make the stuff we buy is more than the energy we use for personal transport and more than our energy demands for heating and cooling [1 – P.94]. Hi-tech electronic items have a large embodied energy carbon footprint and often contain 'conflict materials' such as gold (contacts) , tantalum (capacitors), tungsten etc. Manufacturing a PC requires 11 times it's own weight in raw materials to produce compared to 1-2 times for a car [1 - P.94]

Fair Trade Graphic from Nager IT

In many areas of purchasing there is an ethical and lower carbon choice available, like buying local vegetables and fair trade coffee and chocolate etc. However when purchasing electronic items I have always assumed there was not this option, that no product is more ethical or lower carbon than another (they are all just as bad!).

I would like to highlight two projects that I have discovered lately that are working together to try and improve the state of electronics manufacture. I should state that I have no connection with these projects other than reading their websites.

The first is a German project called Nager-IT who have produced a "Faire Maus" or Fair Mouse.

They have designed the USB mouse using low carbon materials and have gone to great lengths to source ethically produced components and have pushed for transparency throughout the supply chain (something which is almost non existent in electronic production at the moment). Checkout this graphic they have produced illustrating their transparent supply chain:



 The mouse they have produced can be purchased through their website.

The second project is called FairPhone, they are a social enterprise in the Netherlands with the aim of " designing, creating and producing our first smartphone and taking the next crucial step in uncovering the story behind the sourcing, production, distribution and recycling of electronics." They are developing and aiming to start stating in Autumn this year a 'fairer' android powered smartphone.

Graphic from http://www.fairphone.com website

It's uplifting to hear of small independent companies taking on the big players in the consumer electronics industry by offering a 'fairer' alternative. We as consumers should do our best to show that there is demand for more ethically produced electronic projects.

I have also discovered that GreenPeace produce an independent ranking of the large electronic companies based on how 'green' they are.

Graphic from GreenPeace


[1] – Sustainable Energy Without the Hot Air – David MacKay [2009]

How this relates to OpenEnergyMonitor 

Steps we have taken so far

  • Choose RHOS and components which are free of conflict materials
  • Choose low embodied energy materials for enclosures which can be easily recycled
  • Use standard parts which can be easily re programmed and used for other purposes in the future
  • Use recyclable packaging (paper bags which are naturally antistatic) for our shipping 
  • Choose low power equipment, LED lighting, recycled paper for our office and switch things off at the end of the day! 
  • Choose surface freight (boat) rather than planes to ship stock (mainly power adapters) from China, this takes 3 months rather than 3 days but worth it for the 32 times less energy consumed [1 - P.92]. 
  • Choose a PCB manufacture who are based in the UK who uses lead free techniques, complies to the highest environmental industry standard and is actively investing in techniques and equipment to reduce wastage and environmental impact (e.g water treatment and recycling) 
  • Strive to optimise electrical consumption in our hardware to be as low was possible 

Steps we would like to take in the future 

  • Produce an end-to-end supply chain map of our manufacture (possibly using http://sourcemap.com)
  • Try and calculate the embodied energy that goes into producing our designs and constantly try and find ways we can reduce this
  • Sell through local distributors to reduce air based shipping and investigate possibility of local manufacture
  • Use recycled plastic / other low embodied energy martial in the moulding of the housing for our next version of the emonGLCD and temperature monitoring node 
  • Investigate how we can build recyclability into the design, checkout the video below from the Great Recovery Project:


Preparing emoncms.org feeds for conversion to timestore

One of the central principles behind the Timestore data storage approach is that datapoints are stored at a fixed interval. Because you know that there is a datapoint every 10s you dont need to store the timestamp for each datapoint, you only need to store the timestamp for the first datapoint. The timestamp for every other datapoint can be worked out i.e:

timestamp = start + position * interval.

Storing time series data in this way makes it really easy and very fast to query. The tests so far have shown timestore to be several magnitudes faster while also using significantly less disk space.

If your logging to emoncms.org there is now an interface that can be used to select the interval rate that you would like your feeds converted too, its linked from the feeds page.

On the feed/convert page:

The new interval column states the average interval rate of the existing feed and is calculated simply as the end time minus the start time divided by the number of datapoints in the feed. This interval rate can be skewed if the monitor dropped off for a period, so its worth double checking that it is correct.

You may wish to change your interval rate, if your logging temperature data at 5s intervals and 60s is enough to see the changes in temperature you want to see then select 60s as this reduces the disk use of the feed considerably.

To set the interval rate you wish your feed to be converted to, click on the pencil button to bring up in-line editing:

Click on the drop down menu under convert to and select from the list the interval you wish to use. Click on the tick button to complete.

Repeat the above steps for every feed you wish to convert and then once your done click on Add feeds to conversion queue button that is at the bottom of the conversion page.

As mentioned in the last post, its going to take up to a month for the server to convert through all the feeds so you wont see anything straight away. I will update with conversion progress.

I've put all the code for the above on github under an emoncmsorg branch of emoncms, the conversion scripts can also be found in the repo usefulscripts