A new backup system for emoncms.org

A couple of weeks after getting all that setup BigV announced a new feature: archive storage which they say is ideal for backups. Archive storage works in much the same way as connecting a second external drive to a computer. Archive storage on bigv is guaranteed to be on a separate storage pool from the main vm disc which is good news as it wasn't guaranteed that the two separate server method used seperate storage pools (If I understand correctly)

Another advantage to the new archive storage is that its much cheaper to run, costing £2/month for 50GB rather than £16/month for another vm and 30GB extra space.

A simple php script is used to perform the backup, it makes use of the direct file access stuff I recently learnt about (http://openenergymonitor.blogspot.co.uk/2013/07/more-direct-file-storage-research.html) to do incremental backup file copy using php file access commands making it potentially very fast, it can backup files at full file copy speeds:

It accesses the mysql data directly, copying the content of the mysql feed data file i.e:  /var/lib/mysql/emoncms/feed_1.MYD to the backup drive.

On another slightly related note the data in feed_1.MYD is stored simply as:

1 byte for null flag

4 bytes for the timestamp

4 bytes for the float data value
repeated...

and because the readings are inserted one after the other in ascending time we can actually use the mysql feed data files directly with the direct file get_feed_data method to get 10-20x query speed improvement for generating visualisations: https://github.com/emoncms/experimental/blob/master/storage/directfiles/get_feed_data.php

We could even get rid of the mysql feed table index's to save disk space although that will probably slow down mysql updates (I need to look into it). This could be a short term measure before a full timestore emoncms implementation is complete which provides many other benefits.