Emoncms development update: Modules
I've been a little quiet over the last few weeks, emoncms blogposts and progress updates have been far and few between but that does not mean there has been no development progress happening, to the contrary there has been a lot of significant development work been going on.
I outlined at the start of the month in the october emoncms bug and dev list the idea of re-factoring/reorganising emoncms along more modular lines, pointing to some earlier thoughts on it. Since then I made the leap and started the re-factoring process, Ilde gave me a lot of help with splitting up the language files for translations, and good ideas for menu system implementation, Baptiste Gaultier gave me some impetus to look at modularising the dashboard widget code and now we are almost there.
The old structure had a folder for Controllers, Models and Views, with all the different controllers, model and view scripts for a feature mixed in together in these folders and also some files such as setup.php which had code in it for all the different features. As the application was growing this was getting a bit unwieldy, harder to understand/read the code, add new features and keep track of development.
This is how the old structure looked like:
Controllers
feed_controller.php (part of feed module * )
input_controller.php (part of input module * )
...
Models
feed_model.php (part of feed module *)
input_model.php (part of input module *)
...
Views
feed_list_view.php (part of feed module *)
input_list_view.php (part of input module *)
By changing the structure:
Modules
feed
feed_controller.php
feed_model.php
feed_list_view.php (includes no feed message)
feed_schema.php
input
input_controller.php (merged with enum.php)
input_model.php
input_list_view.php (includes no input message)
input_schema.php
and following the following design principles:
I think this change gives us the following benefits:
https://github.com/emoncms/emoncms
Given the potential escalating number of repositories as new modules are added I though it would be best to create a dedicated github organisation for emoncms hence the new location. If you go to: http://github.com/emoncms you will see quite a few repositories:
The emoncms repository is the main emoncms bundle which is build up from the emoncms_framework and the other core modules such as input, feed, dashboard and so on. Then there are repositories for the framework and modules independently of the main bundle including modules that are not in the main bundle such as the raspberry pi module which I will come back to in the next post.
The idea being that the main bundle could be built automatically from the module repositories and the framework and that the commit history could be on a module basis.
Adding a module
To add a particular non core feature such as the raspberrypi interface module to your emoncms installation its now just a matter of downloading the raspberrypi repository and placing it in the modules folder of your emoncms installation, much the same way as you add libraries to an arduino project.
Things to complete
As I mentioned at the start most of the refactoring work has been done but there are still some parts that are not complete such as the notify, admin and statistics module and there are some missing message boxes such as when no inputs, feeds or dashboards exist that need to be re-added.
The feed module implementation is also a little different in that the there is no longer a feed_relation table that used to link a user to their feeds, this is now handled in the feeds table which simplifies things but means a simple database migration process needs to take place if your upgrading from an old emoncms version, I will write a short script to do this soon and document the migration process.
Another important change is the API used to post data from basestations to emoncms what used to be api/post is now input/post as this action is handled by the input module. You can if you want sidestep input processing completely and just use the feed module which now has a full api. The api calls for both the feed and input module are documented in helper pages linked to from the top of the input and feed pages.
That's it for now, in the next few post I will discuss the raspberrypi module and also how to migrate from the old emoncms version.
To engage in discussion regarding this post, please post on our Community Forum.
I outlined at the start of the month in the october emoncms bug and dev list the idea of re-factoring/reorganising emoncms along more modular lines, pointing to some earlier thoughts on it. Since then I made the leap and started the re-factoring process, Ilde gave me a lot of help with splitting up the language files for translations, and good ideas for menu system implementation, Baptiste Gaultier gave me some impetus to look at modularising the dashboard widget code and now we are almost there.
The old structure had a folder for Controllers, Models and Views, with all the different controllers, model and view scripts for a feature mixed in together in these folders and also some files such as setup.php which had code in it for all the different features. As the application was growing this was getting a bit unwieldy, harder to understand/read the code, add new features and keep track of development.
This is how the old structure looked like:
Controllers
feed_controller.php (part of feed module * )
input_controller.php (part of input module * )
...
Models
feed_model.php (part of feed module *)
input_model.php (part of input module *)
...
Views
feed_list_view.php (part of feed module *)
input_list_view.php (part of input module *)
setup.php (mixed code for input, feeds, etc) - if you add a new module you had to add db schema code in here.
notify.php (part of notify module but in the root emoncms directory)
notify.php (part of notify module but in the root emoncms directory)
* Module or feature specific files are all in different locations
By changing the structure:
Modules
feed
feed_controller.php
feed_model.php
feed_list_view.php (includes no feed message)
feed_schema.php
input
input_controller.php (merged with enum.php)
input_model.php
input_list_view.php (includes no input message)
input_schema.php
and following the following design principles:
- A module add's functionality in a self-contained way - not requiring (or minimizing as much as possible) modification of any other modules or the core framework.
- A module can depend on another module. ie dashboards depend on feeds but feeds does not need to depend on dashboards.
I think this change gives us the following benefits:
- Clearer application structure - easy to find module specific code
- Adding a new module is really easy, you dont have to put the controller in the controller folder, the view in the views folder, model in the models folder, database setup in setup.php, all you have to do is download the module folder and place in the modules folder.
- We can then use git to develop particular modules independently of the rest. A developer looking at commit logs therefore only needs to concern themselves with that particular module rather than the application as a whole.
- Its easier to test modules and the framework independently of each other.
- More developer autonomy, project self organisation. You can add a module to the optional modules list without having to coordinate closely with the main emoncms bundle.
The new modular emoncms version can be found here:
Given the potential escalating number of repositories as new modules are added I though it would be best to create a dedicated github organisation for emoncms hence the new location. If you go to: http://github.com/emoncms you will see quite a few repositories:
The emoncms repository is the main emoncms bundle which is build up from the emoncms_framework and the other core modules such as input, feed, dashboard and so on. Then there are repositories for the framework and modules independently of the main bundle including modules that are not in the main bundle such as the raspberry pi module which I will come back to in the next post.
The idea being that the main bundle could be built automatically from the module repositories and the framework and that the commit history could be on a module basis.
Adding a module
To add a particular non core feature such as the raspberrypi interface module to your emoncms installation its now just a matter of downloading the raspberrypi repository and placing it in the modules folder of your emoncms installation, much the same way as you add libraries to an arduino project.
Things to complete
As I mentioned at the start most of the refactoring work has been done but there are still some parts that are not complete such as the notify, admin and statistics module and there are some missing message boxes such as when no inputs, feeds or dashboards exist that need to be re-added.
The feed module implementation is also a little different in that the there is no longer a feed_relation table that used to link a user to their feeds, this is now handled in the feeds table which simplifies things but means a simple database migration process needs to take place if your upgrading from an old emoncms version, I will write a short script to do this soon and document the migration process.
Another important change is the API used to post data from basestations to emoncms what used to be api/post is now input/post as this action is handled by the input module. You can if you want sidestep input processing completely and just use the feed module which now has a full api. The api calls for both the feed and input module are documented in helper pages linked to from the top of the input and feed pages.
That's it for now, in the next few post I will discuss the raspberrypi module and also how to migrate from the old emoncms version.