Blog
OpenEnergyMonitor

Part 2/3: Firmware Continuous Test & Build

This post is part of a series


Following on from the last blog post on using PlatformIO to compile and upload firmware locally, we’re now going to take things a step further and do the same but in The Cloud!

Groan…I know I just used the clichéd ‘C’ word, however there are many advantages to compiling and testing the code in the cloud. At least I didn’t mention ‘IoT’…whoops, just doing my bit for SEO!

In this instance when we say ‘compile in the cloud’ I mean use GitHub, Travis IO and PlatfromIO to compile the firmware and if the branch is tagged with a ‘Git Release’ auto-generate a compiled binary and upload it back to GitHub release page.

The motivation behind this automated-build and testing is working towards creating a robust infrastructure to push OTA updates to ESP8266 connected nodes (EmonESP dev) inspired by this blog post by Daniel Eichhorn (@squix78).

Read on →

Part 1/3: PlatformIO open-source embedded development ecosystem

This post is part of a series:


Part 1/3: PlatformIO open-source embedded development ecosystem

Getting an Arduino based project (or other embedded platform) to compile and upload can be a pain. Making sure all the libraries are installed in the correct locations and of the correct versions can be tricky and time-consuming.

I’m sure many developers will agree that the tools we use for embedded development are generally not as good as those used for web application development.

The Arduino team have done a good job with their IDE to try and make the embedded development tool-chain setup as easy as possible. However, I still find library management a cause of frustration. Especially since I move between computers and OSes frequently.

Recently I have been using PlatformIO and am rather impressed with the ease of setup, speed of compilation, uploading (auto port detection), and most importantly an excellent library manager.

PlatformIO is an open-source ecosystem for IoT development.

Cross-platform build system, IDE integration and continuous testing. Arduino, Espressif, ARM and mbed compatible.

PlatformIO IDE

This post got quite long, therefore, It’s been split into three posts:

  1. PlatfomIO overview & compiling + uploading locally and on a Raspberry Pi
  2. Continuous testing and auto release binary generation using PlatformIO & TravisCI
  3. Continuous Deployment (OTA to ESP8266)

Here are some things that have impressed me about PlatformIO (pio) after using it for a couple of weeks:

  • Fully open-source with active and friendly dev community.
  • Easy to install - Pure python based installed using pip
    • Tool-chains are auto installed on first compile / upload e.g.
      • If trying to upload to Arduino avrdude will automatically be installed
      • If trying to compile an ESP8266 project the ESP toolchain will be installed
    • Cross-platform (Linux, Windows and mac all work the same), this will make support much easier
  • Supports all the main embedded platforms
    • atmelavr (Arduino), espressif (ESP8266), ststm32 (ARM etc.)
  • Command line and IDE
    • At work I use pio via Atom IDE on Ubuntu
    • At home, I use a Chromebook with Caret IDE and interact with pio via command-line
    • When developing directly on an emonPi / RaspberryPi pio command-line works great - Yes, that’s right pio works on a RaspberryPi to compile and upload code directly!
  • Excellent library manager
    • Libraries can be searched and reviewed using command-line or web 2.0 manager
    • Required libraries can be specified in the platformio.ini file and if not present install is prompted upon compilation
    • Specific library version (as defined in library.json (emonLib example)) or git commit SHA
  • Fast compilation & pain-free uploading
    • PlatformIO’s compiler is the fastest in the business
    • USB / Serial ports auto detected
    • Since the emonPi has been added as a custom board PlatformIO can even upload direct to emonPi via RaspberryPi GPIO and trigger GPIO AVR reset, not more hacked avrdude.
  • Easy integration with continuous testing & deployment
    • Integration with TravisCI to enable compilation, code testing and binary (.hex / .bin) release generation in the cloud.
    • emonPi and emonTx git repos are now enabled for continous-testing and auto binary generation on release tag…this is a step change in embedded development.

Example: Compiling emonTx / emonPi firmware

PlatformIO has some excellent quick start docs to cover setting up your first ‘blinky’ project, therefore I will not duplicate here. In this example I focus on setting up pio and compiling emonTx / emonPi firmware. This example assumes a totally bare metal machine with nothing installed except python (2.7 recommended). Pio works great on a Raspberry Pi.

This guide used command-line steps on Linux, if using windows or if you want to use the platformio IDE see PlatformIO Getting Started Page.

The PlatformIO IDE or Atom IDE is excellent and very easy to use (self explanatory). This guide uses pio via command line as this gives a more hands-on experience to how pio is working.

1. Install PlatformIO

The easiest way if running Linux is to install use the install script, this installed pio via python pip and installs pip if not present. See PlatformIO installation docs:

$ sudo python -c "$(curl -fsSL https://raw.githubusercontent.com/platformio/platformio/master/scripts/get-platformio.py)"

2. Clone emonTx / emonPi repo

We’ll use the emonTx (V3 discrete sampling) as an example here but the steps are exactly the same for emonPi.

emonTx V3

$ git clone https://github.com/openenergymonitor/emontx3`
$ cd emontx3/firmware

emonPi

$ git clone https://github.com/openenergymonitor/emonpi`
cd emonpi/firmware

3. Compile with PlatformIO

$ platformio run

or shorthand for the lazy

$ pio run

That’s it! That’s all that’s needed to setup pio from scratch and compile emonTx firmware :-D

The first time platformIO is run it will ask to install the required libraries (at the specific version) and avr toolchain. The required libraries are defined in platformio.ini in the emonTx discrete sampling firmware folder:

e.g.

lib_deps = 
  DallasTemperature @3.7.7
  EmonLib
  [email protected]

Libs version can be defined by github commit ID, version number (git tag) or latest version (no specific version specified.)

The latest version of the libraries are automatically cloned from their native git repos into the .piolibdeps folder in the project dir.

3. Upload using PlatformIO

3a. Via USB to UART programmer to emonTx

$ sudo pio run -t upload

Pio will attempt to auto detect the USB programmer and upload :-)

Note: you will need sudo unless you have correctly configured udev rules.

There is even a built in serial monitor

$ pio device monitor

3b. Using ttyAMA0 GPIO to upload direct to emonPi

This is where things get very exciting, pio can compile the code directly on a Raspberry Pi. The latest version of pio (due to be released in the next few weeks) will have built in support for the emonPi auto-reset on upload using GPIO4

The emonPi board is defined in platformio.ini in the emonpi/firmware folder:

board = emonpi
upload_port = /dev/ttyAMA0

All that is needed to compile and upload directly on the emonPi is:

cd ~/emonpi/firmware
$ sudo pio run -t upload

Thanks a lot to Ivan Kravets for helping integrate emonPi into the PlatfomIO ecosystem. View commit changes


Stay tuned for part 2/3 of the PlatformIO post series which will cover setting up firmware continuous testing using Travis CI and auto compiled binary generation on release tag directly on github. This is currently live on the emonPi & emonTx repos if you want a sneak preview. See travis.yaml file and Travis build status icon on the main emonpi repo Readme.md.



Electrical Supply Line Fault

Last week my home emonPi energy monitor came in handy to help diagnose an electrical supply fault.

A neighbour alerted me that her lights were flickering and internet router was dropping offline. At the time I was doing some DIY and had noticed the power drill I was using change speed erratically. I had presumed it was my drill about the give up the ghost.

I checked the Emoncms graph on my emonPi energy monitor and noticed that VRMS was well out of spec at about 150V (we usually get 230V*) and occasionally jumping up to almost 300V! The electrical company was alerted and the fault was traced back to a frayed conductor on our street. I helped the electrical contractors view the real-time VRMS value to confirm the fault was fixed.

VRMS Fault

Read on →

Substantial update to emonPi / emonBase: emonSD-03-May16 pre-built image release

This latest update to emonSD the pre-built SD card image for the emonPi/emonBase significantly enhances the out-the-box functionality of the emonPi by integrating applications such as nodeRED, openHAB and improved MQTT support.

The emonPi in its default configuration works great using Emoncms for data logging and visualisation. However there may be times when more flexibility and the ability to interface with other hardware or services is desirable.

The emonPi with this latest software update is perfectly positioned to become the heart of a smart home automation system. The ‘always-on’ requirement of energy monitoring and the processing power of the Raspberry Pi makes it a fantastic customisable home-hub. See Related Blog Posts at the bottom of this post for inspiration as to what can be achieved.


For the past few years we have made available a pre-built SD card image for the RaspberryPi. Since launching the emonPi last year the pre-built image has been developed extensively. We have now given it a name: emonSD.

The latest update to emonSD (emonSD-03May16) now incudes the following all setup and pre-configured:

Read on →

emonPi / emonBase Raspberry Pi 3

All emonPi’s / emonBase’s shipped to date have included a Raspberry Pi 2. In the next week or so we will be upgrading to include a RaspberryPi 3. This is a nice upgrade: approximately double the processor speed and on-board WiFi with zero cost increase.

Since the RaspberryPi 3 has got on-board WiFi a USB WiFi dongle is not required; even more costing cost saving for our customers, thanks to hard work of the Raspberry Pi foundation and a bit of Moore’s law.

emonPi Raspberry Pi 3 Acrylic Fascia

Read on →


New Forum, Blog and User Guide Website

We have been busy for the past couple of months building some new websites to improve OpenEnergyMonitor.

Introducing a shiny new:

  1. Community Forum
  2. User Guide website
  3. Blog

Later this week on Thursday 5th May 2016 the old forums will be frozen and we will be moving to the new community forums

Read on →

Part 3: Aggregated supply and demand across multiple households

One of the main problems with looking at demand at the single household level is that demand at this level is very spiky, there are often long periods of low use around 100W and then suddenly a kettle or electric car charger turns on and demand increases by 20-30x in an instant. Solar generation can be very spiky too and so there is room for matching but it can be difficult to get the timing right.

One solution to increasing supply & demand matching at the household level is to have in home battery storage and diversion to hot water stores – but it may not be the best solution for all cases.

Aggregating supply and demand across households provides an example of how the degree of matching can be increased significantly by what could be described as sequential demand. Not everyone boils a kettle at the same time or a cloud passing over one households solar pv system may not be passing over another’s.

In order to explore the extent of increased matching possible through aggregation of supply and demand I have been working with Dominic McCann from Carbon Coop on what could be called a ‘virtual microgrid’ concept, we have a provisional front page for this here - please note that its still in early development:

image

Read on →

Part 2: Exploring carbon intensity and renewable energy matching

image

This post describes an approach we have been exploring that uses the MyEnergy emoncms app and work on virtual smart grids with Dominic McCann from Carbon Coop. There’s a good blog and video on what we’ve been working on over on the Carbon Coop blog.

The last post described two approaches to grid carbon intensity that consider the UK grid as a whole. This approach explores what the overall household carbon intensity might be when on-site renewable energy is considered such as home solar but also when renewable energy is bought over the grid, this could be from a green electricity tariff.

If imported electricity is supplied from a 100% renewable energy supplier in the UK: a large portion of the supply will likely be wind energy (54% of good energy’s ‘fuel’ mix comes from wind) and the UK has the best wind resource in Europe. Incorporating grid wind data and exploring matching with wind and solar, means we can really start to test scenarios such as ZeroCarbonBritain in the present.

There is real-time data available on UK wind supply and its possible to scale the total UK wind output in order to estimate a household ‘share of UK wind’ in real-time.

Read on →

Part 1: Attempting to measure and understand the CO2 intensity of grid electricity

Zero carbon energy scenarios such as ZeroCarbonBritain feature significant electrification of energy demands previously provided by fossil fuels such as heating and transport. With electrification it becomes possible to supply these energy demands with renewably generated electricity from primarily wind, solar and backed up by storage technologies from short term stores such as batteries to long term stores such as power to gas.

We do not of course yet have a zero carbon grid. However there has been significant growth of renewable energy and a decline in coal generation in recent years.

The CO2 emissions of demand side solutions such as heat pumps and electric vehicles are significantly affected by the co2 intensity of the supply but in many cases installing and using these technologies make sense at the UK’s present supply mix and can make more or less sense depending on how you calculate co2 intensity and where you assume the additional electricity required comes from.

Read on →