Online Data Management
The PermaSense data management system is based on GSN, a stream processing framework from EPFL. Our GSN architecture is based on two concatenated server instances: The "private" server imports raw data and serves as the master database only accessible to administrators. The "public" server includes metadata mapping and value conversion to SI values and offers a scalable and open frontend to all users. Using these two servers we allow for full roll backs and traceability of all data being processed.
The system is further augmented by supervision systems built from standard IT components (Zabbix, Cacti, SmokePing).
The visualization of long-term sensor data requires value aggregation for reducing the amount of data points to be shown to a user. Performing the necessary aggregation operators on state-of-the-art relational database management systems requires numerous slow table scan operations occurring in delays on the order of tens of seconds that are not tolerable for interactive browsing.
Instead of calculating aggregates on-the-fly, we have proposed a new caching architecture to precompute data aggregates at different levels of granularity resulting in efficient data retrieval during interactive sessions. This caching architecture is augmented by an in-web browser visualization framework (SensorViz) and is currently used to display large data sites from the field sites on the web. These results have been presented at IPSN 2011.
