Data portability challenges
There are some challenges with implementing data portability for energy data. 5s energy feed data can grow to be quite large, I have several feeds that are approaching 100Mb and a total account size of about 700Mb and thats only just approaching one year.
Moving large quantities of data uses server resources including: cpu, memory and bandwidth. Emoncms.org is hosted on a BigV virtual machine, where you get 1GiB Ram, 25GiB Disk space, 200GiB transfer for £10 a month, which is a fantastic deal for those specs.
As exporting or importing large feeds takes time its important that it does not hog resources on the server making the server unusable for other users, any import or export code needs to use a small percentage of the server resources and ideally the amount of resources used could be adjustable. We also need to keep an eye on the data transfer limits.
In the next post I will post up the initial working feed export code.
To engage in discussion regarding this post, please post on our Community Forum.