Idea for using redis in-memory database to improve emoncms performance
Reducing the number of MYSQL queries required is usually a sure way to improve the performance of the application. Back in March of this year I re-factored the input processing implementation removing a lot of un-needed repeated queries as described at the bottom of this page: http://emoncms.org/site/docs/developinputproc
The results where really good, see the pic of time spent serving queries here
I've been thinking about how to improve this further, mysql queries are used a lot for getting the last value of a feed or input, its used for all the +,-,x,/ input processes, the Power to kWh/d, histogram processes, in-fact most of the processes. When a new datapoint is added to a feed data table emoncms also updates the last time and value into the feeds table every time. This is then used by the input processors and also to provide the live view on the feeds page.
None of these input or feed last updated time/value reads and write queries need to be persistent beyond the very short term that an in-memory database would be fine for. Using an in-memory database like redis should be much faster than going to the hard disk or SD card and so hopefully implementing this will lead to a good performance improvement. It also has benefits for longevity of the raspberrypi SD card as it reduces SD card writes. That's the idea anyway, here's a bit of basic testing of using redis, the next step is to try to implement this in emoncms.
Install redis and redis php client ubuntu
sudo apt-get install redis-server
Install PHP redis client https://github.com/nrk/predis. To install using PEAR which we've already been using for installation of the serial dio library used by the raspberrypi module, call the following:
sudo apt-get install php-pear php5-dev (if you dont already have pear installed)
pear channel-discover pear.nrk.io
pear install nrk/Predis
Trying redis out:
Trying redis out:
< ?php
require 'Predis/Autoloader.php';
Predis\Autoloader::register();
$redis = new Predis\Client();
// 1) Set redis in-memory key-value pair to hold feed meta and last value data
// ONCE SET COMMENT OUT THIS LINE TO SEE THAT feed_1 DATA IS PERSISTENT
// BETWEEN CALLS TO THIS SCRIPT - AS LONG AS MEMORY IS NOT RESET.
$redis->set('feed_1',json_encode(array('name'=>"Solar Power",'time'=>time(),'value'=>1800)));
// 2) Fetch the feed_1 entry from in-memory db
echo $redis->get('feed_1');
Redis is also used by Jean Claude Wippler in HouseMon which is cool:
https://github.com/jcw/housemon
http://jeelabs.org/tag/housemon/
I'd like to learn more about how HouseMon works, take a little time to get a HouseMon install up and running and familiarise myself with the code and architecture as it looks really nice!
To engage in discussion regarding this post, please post on our Community Forum.
https://github.com/jcw/housemon
http://jeelabs.org/tag/housemon/
I'd like to learn more about how HouseMon works, take a little time to get a HouseMon install up and running and familiarise myself with the code and architecture as it looks really nice!