rasdaman newsletter 01/2017


December 2016 - Last month INSPIRE officially adopted the OGC Web Coverage Service (WCS) as INSPIRE Coverage Download Service. Jordi Escriu, Facilitator of the corresponding Thematic Cluster 3, calls "WCS the natural way to deliver and exploit coverage data".

The OGC Web Coverage Service (WCS) standards suite defines services on coverages, a concept including spatio-temporal sensor, images (timeseries), simulation, and statistics data. In practice, coverages often represent "Big Data" as orthoimagery, elevation data, weather forecasts, and point clouds. The WCS Core establishes basic functionality all implementations must provide: access to whole coverages or parts thereof via spatial/temporal subsetting. Extensions add optional functionality, such as data format encodings (like GeoTIFF, JPEG2000, NetCDF), service model extensions (such as scaling and reprojection), and protocol bindings (GET, POST, SOAP, and in future REST).

Representatives of industry, administration, and research are positive about the standard model WCS and its reference implementation rasdaman: "The RASDAMAN product is currently the world leading environment in this domain and the standard working horse for OGC standardisation on these innovative data access interfaces", Guenter Landgraf from European Space Agency (ESA) said.

Already during the workshop “Implementation and potential of INSPIRE coverage data and WCS” at the INSPIRE Conference in Barcelona last Fall attendees were impressed by the options and possibilities WCS is offering. In his presentation INSPIRE delegate, Peter Baumann, and Alex Dumitru from Jacobs University showed the ease of using WCS in a hands-on-session doing live ingest and retrieval using rasdaman. Presenter Julia Wagemann from the European Centre for Medium-Range Weather Forecasts (ECMWF) underlined: “Experience from ECMWF on coverages and WCS is a good example of the benefits and the potential of such kind of data and services. Worth to look at and take into account by other organizations and communities." Editor of the INSPIRE WCS specification is James Passmore from British Geological Suervey (BGS) who has gained ample experience in coverage services as participant of the EarthServer project. Peter Baumann, head of rasdaman, was involved as national German delegate to INSPIRE; based on a long-standing experience in building flexible, heavy-lifting services he is editor of "Big Datacube" standards in OGC and ISO.

Read more: JRC EC Europe: Results from the Workshop on Implementation and Potential of INSPIRE Coverage Data and WCS

rasdaman joining EARSC

January 2017 - Starting 2017, rasdaman is full member of the European Association of Remote Sensing Companies (EARSC), a not-for-profit association promoting Earth Observation (EO) technology especially from European companies. Monica Miguel-Lago expressed that EARSC "are delighted to accept your company as new EARSC full Member".

With these activities EARCS fosters development of the European EO geo-information services industry. EARSC is open to full and observer members with an interest in using or providing services delivering information based on observations of the Earth and its environment from any platform; satellite, aircraft or UAV.

"We see high value in joining EARSC and the hitech portfolio aggregated there, it's like meeting with friends", says Peter Baumann, rasdaman CEO. As a key technology provider in Big Earth Data services, rasdaman enables researchers to browse, access, and analyze massive multi-dimensional data sets from a wide range of source. rasdaman is deeply involved in several Earth observation projects, such as the EarthServer initiative, where monitoring the Earth and analyzing EO timeseries data are at the heart. The rasdaman technology allows real-time analytics on 3D and 4D Petabyte cubes for spatio-temporal sensor, image, simulation, and statistics data as well as their intercontinental fusion. This power of data handling can be wrapped into any kind of clients, ranging from direct visual interaction through maps and virtual globes (like OpenLayers and NASA/ESA WebWorldWind) over Web GIS (like QGIS) to high-end analytisc (such as python).

EarthServer: Experts Attest Significant Progress

Brussels, January 2017 - "The EarthServer-2 project has successfully completed its second milestone and has made significant progress towards the eventual aim of producing a comprehensive big data analytics platform for Earth Science", the European Commission currently attested on the occasion of project´s midterm review performed in Brussels.

Among the main innovation activities, rasdaman, the platform technology of all services, has been extended with new functionalities, such as support for GRIB data and NetCDF ingestion. A new metadata engine, FEMME, has been developed and, over the next period, will be fully integrated with xWCPS, which leads to combined array data and metadata retrieval. Search facilities on both XML-encoded metadata, freetext, and OGC coverages will be an important step as well. Work on mixed queries enabling new user interfaces has been done already, and will be contined. Services for marine, climate, earth observation, and planetary data have been created using the NASA/ESA WebWorldWind visualisation API. Live demonstrations of pilots during the review meeting showed that significant progress has been made in producing high-quality interfaces, as well as in ingesting valuable data resources and integrating new providers, such as ANI.

One of the important results achieved is a dramatic reduction of the amount of data downloaded. While the extraction of data with traditional methods requires users to download all data for the time series, up to Terabytes, with rasdaman users can access data with surgical accuracy, thereby reducing data transfer to Kilobytes. Additionally, access is simplified substantially; for example, users only need to click on the globe for retrieving a timeseries values for this point.

Further, the project has achieved important impact on international big data standardization process. For example, INSPIRE has chosen rasdaman as reference implementation for the INSPIRE WCS Download Service. The technical capability gained during the project, enabled the participating SMEs to substantially strengthen their market footprint.

In the next period, the Petabyte frontier will be attacked, further enhancements will be implemented on service as well as technology level, and "Big Datacube" standardization will be progressed further.

Contact: Dr. Peter Baumann

OGC Confirms rasdaman Compliance with Datacube Extension Standards

Mayland/USA, January 2017 - OGC officially confirms that rasdaman has achieved compliance certification on a number of important extensions to the OGC "Big Datacube" standard, Web Coverage Service (WCS).

OGC operates a compliance program which allows implementations of standards to be tested free of cost test through a neutral, verified, suite maintained by OGC (see www.opengeospatial compliance). The purpose is to increase system interoperability while reducing technology risks. Buyers gain confidence that a compliant product will work with another compliant product based on the same OGC standard, regardless of which company developed the product. The rasdaman array analytics engine has passed the WCS test suite for Core and GET/KVP extension already in 2013 and today is official reference implementation in OGC and also INSPIRE. While rasdaman always has been a first implementer of the OGC Big Data standards - and currently offers the most comprehensive support for the WCS suite - now there is official confirmation for the extensions WCS Post, WCS Processing, WCS Scaling, WCS Interpolation, WCS Range Subsetting. By combining the complementary know-how of both partners Big Picture aims at establishing large-scale automated analysis offering individual insights and support for agricultural production in Germany.

These compliance tests have been developed by Fuzhou University in China. "We are very happy about Fuzhou University undertaking the work of establishing such a test suite", says Peter Baumann, WCS suite editor and rasdaman Principal Architect. "This valuable service to the community allows tool providers like us to get an independent compliance confirmation, which of course is worth more than any self assessment".

Work on demonstrating compliance is ongoing: testing rasdaman against the WCS CRS extension is already in an advanced state, and the WCS-T extension is planned to get addressed next.

rasdaman newsletter 10/2016

rasdaman 9.3 Released

Open-source rasdaman v9.3.0 is now available for download, see rasdaman.org. Important feature additions include:

  • Enhanced NetCDF support
  • GRIB format support
  • Advanced JSON-style format-parameters support in encode/decode
  • WCSTImport supports a general coverage recipe that can be used with complex formats like GRIB or NetCDF
  • WCPS uses a brand new and improved parser and implementation by default
  • CRS reprojection now available in both WCS and WCPS
  • automated download, compilation, and deployment through a brand new installer script

Deployment can be done through one of these options:

  • From source - just run the installer script
  • Packages - RPM (CentOS 7) and DEB (Ubuntu Trusty / Debian 8) are available; more distros will come soon
  • Virtual machines - readily configured with a running rasdaman instance

Full details can be found at rasdaman version 9.3 .

Ready for INSPIRE - Reference Implementation rasdaman in Action at Intergeo

Oktober 2016. This year the annual INTERGEO fair was held at Hamburg/Germany from Oktober 11 through 13. The rasdaman company was represented witha fair booth, a congress keynote talk on INSPIRE WCS and a workshop demonstrating solutions for dealing with it. Highlights included realtime intercontinental weather/satellite image fusion and ultrafast on-demand cloud processing.

Jacobs University demonstrated in a one-hour-session the radically simplified handling of spatio-temporal sensory, images, weather data, and more based on Web requests following the open OGC standards WCS, WCS-T, and WCPS. Critically important, all components of the WCS suite are seamlessly harmonized.

In the first part of the presentation, implementation and potential of INSPIRE coverage data and WCS was outlined: Member State institutions have to make their coverage data conformant to the INSPIRE directive before 2020, and coverages occur across many, if not most of the Annex II and III themes. According to INSPIRE experts the list is long indeed: Orthoimagery, Elevation, Geology, Atmospheric conditions and Meteorological geographical features, Oceanographic geographical features, Soil, Land cover, Natural risk zones, Energy resources, and more.

Concretely, in the scope of INSPIRE raster data shall be implemented and made available as coverages according the concepts stemming from OGC coverages, ISO 19123 and other related standards currently in force. To this end, INSPIRE is establishing a Web Coverage Service (WCS) based on the OGC WCS standard adopted in 2012. This INSPIRE WCS has been published in a draft version, and comments from the member states have been collected.

In the second part of the presentation a live demo showed how straightforward coverage services can be. In the talk and at the rasdaman booth visitors experienced live ingestion of NetCDF files and their retrieval as TIFF files. It took just a simple change in one configuration file to switch from mosaicking to timeseries generation in the database. The open-ended capabilities of the OGC geo analytics language, Web Coverage ProcessingService (WCPS), was impressively shown. In the Amazon cloud, a Terabyte of data was analyzed in about 100 milliseconds. Heavy rainfall risk areaswere determined in realtime through automatically orchestrated queries combining ECMWF in the UK and NCI in Australia.

Contact: Dr. Peter Baumann, rasdaman GmbH.

INSPIRE Conference 2016

September 2016. Europe, on its quest for a harmonized Spatial Data Infrastructure, has held its annual INSPIRE conference in Barcelona from 26 to 30 September. Highlight was the presentation of INSPIRE WCS, crafted under the lead of British Geological Survey. INSPIRE Editor Jordi Escriu concluded: "If you have raster data you should use WCS" after experiencing live the ease of ingest and download through INSPIRE WCS Reference Implementation, rasdaman.

On Monday 26, "Choosing the Right INSPIRE Download Service Technologies" initiated discussion. Next, large-scale practice proofs of WCS were presented in the session "Improving access to big data through OGC standard interfaces - a hands on WC(P)S session", based on 150+ Terabyte services and intercontinental datacube fusion. Over the next days, a lot of conference attendees took the opportunity to visit the INSPIRE booth and, next toit, the EarthServer booth.

On Thursday 29, "Integrating INSPIRE into a single seamless information space" highlighted the integration power of the WCS suite. Finally, on Friday 30th "Implementation and potential of INSPIRE coverage data and WCS" summarized and deepened the topic. Live demos based on rasdaman which hasbeen chosen as INSPIRE WCS Reference Implementation.

WCS has proven itself through several years as an OGC standard for regular and irregular grids, such as imagery, elevation data, climate data, andmany more spatio-temporal data. The modular suite of standards enables data access, extraction, download - up to versatile analytics with OGC WebCoverage Processing Service (WCPS). Based on the unifying concept of coverages, data homogenization and service integration is achieved. The OGC coverage data and service model currently is on the way of becoming the lingua franca in Big Earth Data being adopted by INSPIRE as well as ISO. Both the leading open-source and proprietary tools support WCS, with rasdaman being the only fully comprehensive implementation of the WCS suite, in particular with its award-winning multi-dimensional parallel, distributed array engine.

Big Data Help Harvesting More

The German Ministry of Food and Agriculture is funding project "BigPicture - Diagnosis in the field: Big-Data-based determination of causes for satellite-image-derived and site-specific variations" carried out jointly by rasdaman GmbH, Bremen, and Spatial Business Integration GmbH, Darmstadt. Goal is helping farmers to optimize returns and to protect the environment through innovative Big Data technology.

A ceremony marked the official kickoff and underlined importance of this project. "For German agriculture it is important to play in the worldwide ivy league when it comes to hitech services. Projects like BigPicture play an important role due to their outstanding innovation and practical relevance", says Parliament State Secretary of the Federal Ministry for Food and Agriculture, Peter Bleser, while handing out the funding certificate on October 10.

With this handover of the certificates to rasdaman GmbH and Spatial Business Integration (SBI) the Federal Ministry of Food and Agriculture officially announced its financial support of the BigPicture research project through its innovation grants program. Overall volume of this 2.5 years project is above one million Euro. Technically, BigPicture aims at interpreting satellite captured variations in farm fields through Big Data technologies so that targeted measures, such as fertilizer placement, application of pesticides, or choice of species to grow, can be derived. Manifold information will be combined in such analyses, including satellite imagery, weather data, as well as the farmers' experience. This way, the process chain from satellite-based symptoms capturing in the field overdiagnosis up to therapy recommendation will be automated completely.

By combining the complementary know-how of both partners Big Picture aimsat establishing large-scale automated analysis offering individual insights and support for agricultural production in Germany.

Bremen-based hitech company, rasdaman GmbH, is world leader with its multi-award-winning agile Big Data technology for massive spatio-temporal datacubes. Just this summer US magazine CIOReview has chosen high-speed server rasdaman into their list of top Big Data solutions worldwide. Scaling seamlessly from laptops over clouds to intercontinental federations, rasdaman is in operation at leading data centers. The European Space Agency (ESA) operates a rasdaman database with 145 Terabyte, and the European Centre for Medium-Range Weather Forecast (ECMWF) wants to unleash its world's largest climate data archive encompassing 90 Petabyte. International "Big Datacube" standards used rasdaman as a blueprint, such as ISO Array SQL and the Open Geospatial Consortium (OGC) "CBig Geo Data" suite, WCS.

Spatial Business Integration GmbH (SBI) focuses on providing services on a worldwide scale based on satellite and geospatial data for its customers in the agricultural sector, as well as the agro-chemical and finance industries. "We care about the world, which is why we provide information which can protect food, water and nature", stated General Manager Dr. Born recently at a conference attended by delegates from the agricultural, agro-chemical and space industries. Providing intelligent information products derived from geospatial data assists SIB's clients inrealizing their business strategies, ensuring efficient business processes, saving time and money, and managing limited resources such as arable land, nutrients and water. Spatial Business Integration's success is anchored in more than 20 years of experience and expertise in the fields of satellite remote sensing, geospatial technologies, plant sciences, water and agriculture.

The BigPicture project is supported by funds of the Federal Ministry of Food and Agriculture (BMEL) based on a decision of the Parliament of the Federal Republic of Germany via the Federal Office for Agriculture and Food (BLE) under the innovation support programme.


rasdaman newsletter 08/2016

US CIOReview: rasdaman Among Most Promising Big Data Solutions

Published from Fremont, California, CIOReview is a print magazine that explores and understands the plethora of ways adopted by firms to execute the smooth functioning of their businesses. A distinguished panel comprising of CEOs, CIOs, IT VPs including CIOReview editorial board finalized the "100 Most Promising BigData Solution Providers 2016" in the U.S. and shortlisted the best vendors and consultants.

In their July 20 press release where Rasdaman has been ranked one of the 100 Most Promising Big Data Solution Providers by US magazine CIOReview.

"It's a delightful experience to announce Rasdaman as one of the 100 Most Promising BigData Solution Providers 2016," said Jeevan George, Managing Editor of CIOReview. "Rasdaman is recognized for providing highly innovative products that helps to store and query massive, multi-dimensional array of data into the data base, which brings new dimension to the industries in the Big Data arena."

More info: CIOReview rasdaman assessment, CIOReview digital magazine

INSPIRE Conference: Coverages Trending

Europe, on its quest for a harmonized Spatial Data Infrastructure, is about to congregate for the annual INSPIRE conference, this time in vibrant Barcelona in mid-September. Highlight will be the presentation of the INSPIRE WCS, crafted over many months under the lead of British Geological Survey.

On Monday September 26, "Choosing the Right INSPIRE Download Service Technologies" will initiate discussion. Next, large-scale practice proofs of WCS by PML, ESA, ECMWF, and others will be presented in the session "Improving access to big data through OGC standard interfaces - a hands on WC(P)S session", based on 150+ Terabyte services and intercontinental datacube fusion. Over the next days, you may want to visit the INSPIRE booth and, next to it, the EarthServer booth. On Thursday 29, "Integrating INSPIRE into a single seamless information space" will highlight the integration power of the WCS suite. Finally, on Friday 30th "Implementation and potential of INSPIRE coverage data and WCS" will summarize and deepen the topic. Live demos will be based on rasdaman which has been chosen as INSPIRE WCS Reference Implementation.

WCS has proven itself through several years as an OGC standard for regular and irregular grids, such as imagery, elevation data, climate data, andmany more spatio-temporal data. The modular suite of standards enables data access, extraction, download - up to versatile analytics with OGC WebCoverage Processing Service (WCPS). Based on the unifying concept of coverages, data homogenization and service integration is achieved. The OGC coverage data and service model currently is on the way of becoming the lingua franca in Big Earth Data being adopted by INSPIRE as well as ISO. Both the leading open-source and proprietary tools support WCS, with rasdaman being the only fully comprehensive implementation of the WCS suite, in particular with its award-winning multi-dimensional parallel, distributed array engine.

rasdaman newsletter 07/2016

Open Source Parallel rasdaman at VLDB 2016

At the VLDB BOSS workshop this fall it is planned to release rasdaman community BOOST!, the open-source engine that allows parallelizing single queries across clouds, clusters, and data centers.

Help to get this accepted, vote for rasdaman on the workshop page:

... and the timing is really tight: only 3 days left! Thanks for your click to support rasdaman! Please feel encouraged to distribute immediately over all your channels.

rasdaman newsletter 04/2016

EGU 2016 Splinter meeting: EarthServer - the Global Datacube Federation

Meet EarthServer at the splinter meeting "EarthServer: the Global Datacube Federation" at EGU 2016, Tue, 19 Apr, 12:15–15:00 / Room 2.83.

One of today's most influential initiatives in Big Geo Data is EarthServer which is paving the way for flexible, scalable datacube services based on innovative NewSQL technology. Researchers from Europe, the US and recently Australia have teamed up to rigorously materialize the datacube paradigm for Earth Observation, ocean, meteorological, and planetary science.

The platform technology, rasdaman, enables ad-hoc datacube analytics and fusion. Integrated metadata / datacube retrieval is under preparation. Relying on the OGC Big Geo Data standards, WCS and WCPS, "any question, anytime, on any size" becomes possible from open-source tools like OpenLayers, ncWMS, QGIS and proprietary tools like ESRI ArcGIS. NASA WorldWind adds high-end visualization. EarthServer databases have transceded 100 TB and are currently heading towards the Petabyte frontier. Single queries have been split automatically across more than 1,000 cloud nodes.

In this meeting, EarthServer will be presented live: its vision, technology, existing services, and its impact on Big Datacube standards. Questions will be answered like "why is it advantageous for our data center join the federation?", "how can we do it, and how much effort is needed?".

PS: Snacks and refreshments will be provided free of cost.

your rasdaman team

See rasdaman in Action - live at EGU 2016!

The EGU General Assembly 2016 provides an essential opportunity to find out more about the technical innovations and web services developed in the EarthServer project, which uses rasdaman as its platform core. Learn more about spatio-temporal analysis on Big Earth Datacubes in different geo domains. The EarthServer initiative establishes a series of novel high performance datacube services. At the EGU conference the partners will demonstrate these innovations through a series of live demos, poster sessions, and presentations as follows:

  • Monday, 18 April 18
    • rasdaman: poster EGU2016-17171 | Big Geo Data Services: From More Bytes to More Barrels | Baumann | 17:30-19:00 | Hall A
    • Jacobs University: SC43 Standards based Services for Big Spatio-Temporal Data | Baumann | 17:30–20:00 | Room 2.44
    • Tuesday, 19 April
    • EarthServer Splinter meeting of EGU2016 | SMP13 | EarthServer: the Global Datacube Federation I 12:15–15:00 / Room 2.83
    • Jacobs University/rasdaman: poster EGU2016-16985 | Science SQL as a Building Block for Flexible, Standards-based Data Infrastructures | Baumann| 17:30-19:00 | Hall A
    • Jacobs University: PICO EGU2016-3996 | Planetserver | Remote Sensing Data Analytics for Planetary Science with PlanetServer/EarthServer Pio Rossi, Baumann | 17:30–19:00 | Hall X4
    • ECMWF: Poster EGU2016-16899 | EarthServer: Opportunities and Challenges of serving ECMWF´s peta-sized Archive through OGC Web Services | Wagemann/Siemen/Lamy-Thepaut | 17:30 - 19:00 | Hall A
    • Wednesday, 20 April
    • Jacobs University: poster EGU2016-16753 | Datacubes as a Service Paradigm | Baumann | 15:30-17:00 | Hall A
    • Friday. 21 April
    • Jacobs University: oral EGU2016-17298 | The EarthServer Federation: State, Role, and Contribution to GEOSS | Merticariu, Baumann | 11:45- 12:00 | room 2.61

rasdaman newsletter 03/2016

rasdaman 9.2 with more performance and functionality

We are proud to announce availability of rasdaman 9.2.1, which represents yet another important increase in performance, functionality, and OGC standards support. The team has worked hard, committing 216 patches in total since v9.1, see the full details.

Major changes:

  • Scalability:
  • Improved storage support, up to exascale datasets on all filesystems
  • Improved performance and reliability for the rasnet protocol; rasnet is now the default communication protocol.
  • Automated balancing of worker processes
  • Ingestion:
  • WCS-T import user interface improvements: WCS extract recipe, ability to resume and retry imports, improved filtering of input files, improved status messages
  • WCS-T import stable recipe API allowing developers to create their own custom recipes for their specific usecases
  • Various WCS-T enhancements
  • Improved transactional handling in the SQLite/Filestorage backend
  • OGC Web Services:
  • Embedded server for OGC Web Services platform - now it can also run as a standalone executable without the need of an external server container like Tomcat
  • Support for PNG as an export format in WCS
  • User Interface:
  • Revamped interface for the WCS Client + various enhancements
  • Miscellaneous:
  • runtime configuration of logging
  • Greatly simplified installation procedure with rasdaman installer, RPM and DEB packages
  • Over 200 fixes for issues reported

There was a lot of work on improving the installation experience of rasdaman, and v9.2.1 now offers a range of download options:

  • From source: we have a brand rasdaman installer script that greatly automates installation.
  • Packages: RPM (CentOS 7) and DEB (Ubuntu Trusty) are available; more distros will be supported soon. If you are updating RPMs, note that the repository has changed; further details on the RPM installation page.
  • Virtual machines: a preconfectioned virtual machine, ready for use.

We hope you will enjoy the improvements and new features this version brings; as always, all feedback is greatly appreciated!

your rasdaman team

rasdaman Donates to "Menschen für Menschen" ("Humans for Humans")

Dezember 2015. Our company has been quite successful in 2015 - innovation contributions in the rasdaman product as well as in international research projects, as well as substantial license sales.

Time to share - and so a donation has been made by the company (that means: us) to a project supporting fellow humans living in less favoured areas. Concretely, "Menschen für Menschen" is a multi-decade project building up infrastructure with and for people in Africa.

We all have worked hard towards our success, and our joint donation now helps others. Our little contribution to the global village. Merry Christmas and a Happy New Year!

your rasdaman team

rasdaman newsletter 01/2016

How Fast is rasdaman?

January 2016. As the rasdaman array engine is known worldwide, more and more people ask about a comparison with related technology. Now scientific tests have proven: rasdaman is the fastest Big Array Data technology by far.

The setup was immense: at the 2015 Annual Conference of the American Geosciences Union, more than 20,000 visitors were present. Researchers from George Mason University and University of Wisconsin / Madison presented a performance comparison of rasdaman with Apache Hive and Apache Spark, to renowned Big Data technologies. Test case was evaluation of MODIS satellite imagery. The outcome reported showed that rasdaman outperforms both competitors by far.

"Our team is very happy about these findings" says rasdaman Principal Architect, Peter Baumann. "We still have many ideas for speeding up rasdaman even further, in particular on massive datacubes, so we are excited to excite again in future".

Simultaneously, a general Array Database benchmark was presented at an expert workshop on database benchmarking held in India this December. Here rasdaman was compared against SciDB and SciQL, to other Array Databases. The various tests, which are available as open source code on github for reproduction by the community, revealed that rasdaman often is faster by several order of magnitudes. However, some of the tests could not be scaled to the intended "Big Data" situations as SciQL and SciDB could not ingest large datacubes. Notably, known rasdaman databases exceed 130 TB.

Elephant Meeting on Big Data Services

November 2015. While we try to get to grips with the ever-increasing “Big Data” deluge we recognize that adequate Web services are a key prerequisite for ubiquitous, flexible, and fast data access. In a massive concertation effort several large European initiatives have teamed up now to address the service challenge. From 12 through 13 November 2015 an inaugural EUDAT Workshop on services for Big Data held successfully at the Supercomputing Center in Barcelona, Spain. Representatives of three decisive Big Data projects - EUDAT, EarthServer, and EPOS - have come together to discuss innovative alternatives for value-adding services.

To consolidate activities around these specific themes the workshop was divided in several tracks focusing on the topics of Big Data semantics, federated Data Mining, and multi-dimensional Array Databases for large time series. Discussions started by capturing best practices and discussing the current state of development and activities in the respective areas. Questions like: How can data processing be orchestrated optimally or how can scientific workflows make use of EUDAT services were discussed intensively in different working groups.

Peter Wittenburg, scientific coordinator of the EUDAT Data Infrastructure, convened a critical variety of expertise from Europe and the US. Especially the topic of multidimensional arrays was focused by the experts because of playing a major role in scientific and engineering data. In a summary Mark van de Sanden, EUDAT Workpackage Leader, and Peter Baumann, workshop facilitator of the EUDAT Array Database track, pointed out possible roles of EUDAT in the future:

  • IaaS provider: providing a cloud infrastructure to run Array Databases
  • SaaS provider: providing an Array Database as an domain-independent, horizontal service
  • Providing tools for easy data movement between EUDAT DCI domain and User domain
  • Providing domain services (e.g., geo, astro, life sciences) based on a common horizontal platform of array services, thereby leveraging cross-community effects

Peter Baumann resumed his experiences of running large-scale infrastructures in his presentation: "Of course multidimensional arrays do not stand alone, they are intertwined with other data types, but typically they constitute the "Big Data" part. Therefore, it makes sense to integrate arrays into common data management platforms." The flexibility of querying data, achieving data independency, scalability and standards conformance are critical advantages of Array Database technologies. Among the challenges spotted were integration of heterogeneous data types, including arrays, into a single common information space for users. Array intensive domains like the Earth-, Space-, and Life Sciences were considered as possible candidates of future EUDAT services.

OGC Requests Comments on Coverage Implementation Schema

December 2015. The Open Geospatial Consortium (OGC) requests comments on the OGC Coverage Implementation Schema (abbreviated as "CIS"). This candidate OGC Standard complements the OGC Abstract Topic 6 Standard (which is identical to ISO 19123) with a concrete coverage model whose implementations are conformance testable and, hence, are interoperable. ISO has expressed interest to adopt CIS 1.1 as ISO 19123-2.

The Coverage Implementation Schema version 1.1 is the backwards compatible successor of the "GML 3.2.1 Application Schema - Coverages" version 1.0 (abbreviated as "GMLCOV"). The CIS candidate standard has been renamed to reflect the breadth of available data encodings beyond GML and to reduce market confusion. While incorporating all features of GMLCOV 1.0, CIS 1.1 adds important new modeling capabilities, such as coverage partitioning, interpolation, geometry/value pairs, SensorML support, and point clouds. Further, a general model for all combinations of non-regular grids is supported, which includes GML 3.3 coverage types as special cases.

The OGC Abstract Topic 6 Standard defines an abstract coverage model. Coverages represent homogeneous collections of values located in space/time, such as spatio-temporal sensor, image, simulation, and statistics data. Common examples include 1-D timeseries, 2-D imagery, 3-D image time series or spatial models, as well as 4-D climate and ocean data. Generally, coverages encompass multi-dimensional regular and irregular grids, point clouds, and meshes.

The OGC Coverage Implementation Schema complements OGC Abstract Topic 6 with a concrete coverage structure definition of a subset of coverage models in OGC Abstract Topic 6 that can be conformance tested, regardless of the implementations’ data format encoding. CIS enables a wide range of interoperable implementations.

Coverages can be encoded in suitable formats (such as GML, JSON, GeoTIFF or CF-netCDF) and can be partitioned, e.g., for a time-interleaved representation. Coverages are independent of service definitions and therefore can be accessed through a variety of OGC services types, such as the Web Coverage Service (WCS) Standard. The coverage structure can serve a wide range of coverage application domains, thereby contributing to harmonization and interoperability between and across these domains.

Download the candidate OGC Coverage Implementation Schema and provide feedback by 16 January 2016.

rasdaman newsletter 11/2015

Simplified Big Data Handling: New OGC Standard

2015-oct-30: Voting has closed on the OGC WCS-T specification, resulting in the effective adoption of the standard as part of the OGC WCS suite. WCS-T enables Web-based insertion, deletion, and modification of spatio-temporal coverages offered through a WCS-based service.

The OGC Web Coverage Service (WCS) Interface Standard is an open standardwidely implemented in geospatial applications around the world, both open-source and proprietary. The standard's main function up until now has been to support retrieval of geospatial "coverages" data such as spatio-temporal sensor, image, timeseries, simulation, and statistics data. With the new Transaction extension, the standard will support not only retrieval but also transactions such as insert, update, and delete of a coverage on the server.

"The high-performance datacube services running rasdaman showed us the need for enhancing operator support for the maintainers", explains Peter Baumann, Professor of Computer Science at Jacobs University and editor of the standard. "Incremental creation and maintenance of 2-D, 3-D, and 4-D datasets, possibly with irregular grids along space or time, can be challenging and error-prone. With WCS-T, a simple Web request performs all actions necessary, and ensures consistency of the server offering at all times." Simultaneously to writing the specification, WCS-T has been implemented in rasdaman. He explains, "With rasdaman, we have the ideal test vehicle for evaluating new service concepts." Naturally, therefore rasdaman becomes reference implementation lateron. In the intercontinental EarthServer initiative [0], WCS-T is already regularly used to establish datacubesgrowing beyond one Petabyte each.

But there is more to it than just convenience. Effectively, WCS-T closes a gap when it comes to Earth data mash-ups and federations: Coverage subsets can be extracted from one server using any suitable service interface, such as WFS, WCS, WMS, WPS, or SOS and can be fed into another service node via WCS-T.

The team around Baumann already has extensive experience in crafting widely accepted "Big Data" standards. They have developed the whole WCS 2.0 suite which today is supported by MapServer, GeoServer, ESRI, and many others. Recently, OGC WCS has entered the adoption process of both ISO and INSPIRE. Further, the group has initiated ISO work on domain-independent SQL support for multi-dimensional arrays, the forthcoming SQL/MDA standard. WCS data offerings today exceed hundreds of Terabytes, and Array SQL queries have been parallelized over more than 1,000 cloud nodes.

A draft version of the WCS-T standard is available at the OGC Coverages info hub [1], the authoritative standard document will be made available soon on the official OGC WCS download page [2].

[0] earthserver.eu

[1] opengeospatial.org/CoveragesDWG

[2] www.opengeospatial.org/standards/wcs

EarthServer TV Documentary is in 2nd place of the International Science Film Festival

2015-oct-23: The 1h TV documentation Big Earth Data - the digitized planet produced by Jacobs University Bremen and rasdaman GmbH as part of the EarthServer project in cooperation with the german broadcast ZDF/ARTE is in 2nd place of the "International Scientific Film Festival in Szolnok, Hungary. The documentary which has been funded by the European Commission and broadcasted in February 2015 is concerned with the chances and challenges dealing with Big Earth Data.

The contest received 146 films from 32 countries for consideration this year. National pre-juries of local science education and media experts in all participating festival countries have concluded their selection process and the official selection of the Science Film Festival is comprised of 52 films from 17 countries in 2015. Last year the Science Film Festivalreached 600 000 visitors in 13 countries. It is the largest event of itskind worldwide in terms of audience reach. The contest provides forum for such distinct films which can be connected to science, includes scientific results or activities.

It was the 14th year that the International Scientific Film Festival witha wide spectrum of topics has started its way. Since 2001, every other year a film forum emerged on which the creators of film and science had the opportunity to display their work that had a scientific theme. During the last decade scientific educational films, documentaries and portraits about science, short and long features, experimentals with a scientific topic, documentaries dealing with protection of nature and environment, and films dealing with the relationship of science and fake science took place. The aim is to spread the wide spectrum of films to the audience, to university and college students in and outside the country who major in film studies, and to pass more and more of these films to one of the tv channels.

Read more: Filmfreeway.com

Big Data Cube Standard: SQL Standard Draft Established

2015-oct-17: With the NoSQL movement, many voices have been heard that the classical database language, SQL, does not support Big Data adequately. The latest extension of SQL with query support for massive multi-dimensional arrays demonstrates the power of declarative languages in a new realm, Big Data Cubes. Implementations like the rasdaman array analytics engine underpin this with a proof of high performance and scalability.

In practice, arrays appear in manifold ways in science and engineering, but also in social media and business analytics. Examples include satellite imagery, weather forecast data, microscopy data, cosmological simulations, connection information in social networks, and business datacubes. Frequently, such multi-dimensional arrays constitute the "Big Data" part inapplications.

"Due to the lack of array support in SQL, industry and data centers had to resort to self-programmed workarounds which typically lead to 'silo solutions' which are limited in functionality, hard to maintain, and not scalable", observes Peter Baumann, Professor of Computer Science at Jacobs University and inventor of Array Databases. Together with Jim Melton (Oracle) and Dimitar Misev (Jacobs University) he is editor of the SQL/MDA specification. Baumann and his team, including Misev, have implemented rasdaman ("raster data manager"), a highly flexible and scalable Array Database system which meantime is in manifold practical use.

While this may sound like a field for the techies among us, it effectively heralds a disruptive change in how we deal with Big Data. Due to technical limitations, data today get subdivided into "the data" and metadata, both being maintained and served using entirely different paradigms. For example, metadata may sit in queryable SQL databases while data reside infiles offered for download. "Common opinion is that metadata are small, smart, and searchable whereas data are big, clumsy, and only amenable to simple download", explains Baumann. "With SQL/MDA, this age-old subdivision can be overcome, and we can freely mix queries on data and metadata - effectively, we do not need this subdivision any longer." The resulting single, coherent information space will be instrumental for a new era of user-friendly services on large-scale, heterogeneous data assets, and ultimately for our understanding of "Big Data.

At its Tokyo meeting this October, the ISO SQL working group has decided to suggest the meanwhile completed specification for "CD ballot", that is: for comments by the nations participating in ISO. Final adoption as an ISO standard is expected for 2017.

ESA Conference "Big Data from Space" 2016

2015-oct-31: The Second Conference on Big Data from Space (BiDS '16) will be held at the Auditorium of Santa Cruz de Tenerife (Spain) on 15-17 March 2016. The conference is jointly organised by the European Space Agency (ESA), the Joint Research Centre (JRC) of the European Commission, and the European Union Satellite Centre (SatCen), in collaboration with the Instituto de Astrof=C3=ADsica de Canarias (IAC).

The objective is to bring together researchers, engineers and users working in the Big Data from Space sector. The focus of the Big Data from Space Conference is on Data Science, addressing the whole data lifecycle (from data acquisition by space-borne and ground-based sensors to data management, analysis and exploitation). Various space-related domains are concerned, like Earth Observation, Space Science, Solar System Objects, Space Situational Awareness, Secure Societies, etc.

The conference addresses various themes and offers several options for contributions: oral presentations in standard or invited sessions, poster sessions, demo sessions, elevator pitches, etc.

Conference details are available at: congrexprojects.com

Please feel invited to submit, by November 07, two page abstracts via theelectronic submission system, as indicated on the conference web site.

Notification of Acceptance will be provided by December the 20th. Additional information on the Final Program, hotel accommodation, transports, etc. will be provided on the Conference Web site.

rasdaman newsletter 08/2015

Release of rasdaman 9.1 Brings More Performance, Functionality

Version 9.1 of the pioneer Array Database, rasdaman, is now available fordownload. It represents yet another important increase in performance, functionality, and OGC standards support.

After relentless community efforts over the last months, version 9.1 of rasdaman is now available from the open-source portal www.rasdaman.org. Dimitar Misev, a core developer of rasdaman, announced the news on the rasdaman list indicating a series of highly pertinent enhancements.

One set of new features concerns functionality and performance, it includes: External tool support is available for R and python; queries automatically respect null / nodata values; based on user experience, new functionality has been added which specifically eases and accelerates massive timeseries analysis; a new client/server protocol boosting performance and scalability; type management has been simplified by integrating it with the query language.

Further, support for OGC standards has been broadened: WCS-T, the forthcoming OGC standard for Web-based Big Data maintenance and mashups, is fully implemented; support for WMS 1.3 has been added; the WCS Web client offers comprehensive support for WCS Core and all Extensions; and WCPS is prepared for the forthcoming version 2.0 of the standard which is currentlyunder preparation.

Notably, the rasdaman company keeps on contributing code. In addition to manifold detail code enhancements, with version 9.1 null/nodata support has been released from the commercial to the open-source version, a feature which has been repeatedly been asked for by the data science community.These contributions have been made possible through EU funded projects PublicaMundi and EarthServer-2.

At the EarthServer review in Fall 2014, independent reviewers attested, based on "proven evidence", that rasdaman will "significantly transform the way that scientists [...] will be able to access and use data in a way that hitherto was not possible".

EarthServer Project Goes into the Second Round

Pushing the boundaries of Big Earth Data services, the intercontinental Earthserver initiative enables researchers to browse, access, and analyze massive multi-dimensional data sets from a wide range of sources. Big Earth Data at your fingertips - this is the vision of EarthServer for unleashing the potential of Big Data through a disruptive paradigm shift in technology:

  • from isolated silos of data with disparate functionality towards a single, uniform information space
  • from a difficult, artificial differentiation between data and metadata access to unified retrieval
  • from zillions of files towards few whatever-size datacubes
  • from limited functionality to the freedom of asking anything, anytime, any server in a peer network of data centers worldwide.

In phase 1, EarthServer has established open ad-hoc analytics on massive Earth Science data, based on and extending the leading Array Database technology, rasdaman. According to EU Commission and inde pendent reviewers,rasdaman will "significantly transform the way that scientists in different areas of Earth Science will be able to access and use data in a way that hitherto was not possible" as demonstrated by portals with over 230 TB of spatio-temporal data. EarthServer "with no doubt has been shaping the Big Earth Data landscape through the standardization activities within OGC, ISO and beyond".

Now phase 2 of EarthServer has started, with an even more ambitious goal:data centers will provide at least 1 Petabyte of 3-D and 4-D datacubes. Technology advance will allow real-time scaling of such Petabyte cubes, and intercontinental fusion. This power of data handling will be wrapped into direct visual interaction based on multi-dimensional visualization techniques, in particular: NASA World Wind. Following the motto "a cube says more than a million images" EarthServer has set out to redefine the BigData service landscape even more.

This way, critical support will be given to Copernicus and the Sentinel satellite data: a single 3D x/y/t datacube will be constructed for each satellite instrument so that millions of images form a single, simple data space, irrespective of its size resulting. Likewise, each climate dataset will form a single 4D datacube. Access to these cubes is through a clean-slate standards-based query language on n-D grids, OGC WCPS. This yieldsthe agility that any query can be sent at any time, without admin intervention on server side. Multiple cubes can be combined based on parallel, distributed processing. Altogether, the WCPS language allows navigation, extraction, aggregation, and fusion of any-size space/time data cubes using simple, yet powerful query operators. The consortium consists of Jacobs University (Germany, coordinator), , rasdaman GmbH (Germany, SME), , Plymouth Marine Laboratory (UK), European Center for Medium-Range Weather Forecasts (UK),MEEO s.r.l. (Italy, SME), and CITE S.A. (Greece, SME). Additionally, twohigh-profile international organizations participate: NASA (US) and National Computational Infrastructure (Australia).

US Government Looking at Datacube Processing Standards

In face of the Big Geo Data challenges the OGC standards for spatio-temporal keep advancing. Recent emphasis was on the semantic-rich server-side processing of coverages by bridging the Web Coverage Service (WCS) 2.0 and the Web Processing Service (WPS) 2.0. The standards observing group of US ?Federal Geographic Data Committee (FGDC) sees coverage processing as a mandatory standard in future.

Server-side filtering and processing of spatio-temporal coverages is a core trend emerged in 2014. The Web Coverage Service (WCS) standard suite represents a modular framework for versatile access to coverages; a special extension, WCS Processing, allows to send datacube queries to a server using the OGC Web Coverage Processing Service (WCPS) language. Up to now,the WCPS way of processing was separate from the OGC Web Processing Service (WPS). The new "WPS Application Profile - Coverages" (short: WPS-Cov,OGC document 09-045r1) has been adopted unanimously as OGC Discussion Paper by the WPS Standards Working Group. It specifies how to invoke WCS and WCPS functionality using the WPS protocol. A simple parameter mapping enables WCS and WCPS requests over WPS as carrier while remaining fully conformant with all these specifications individually.

In an earlier project funded by the European Space Agency (ESA) such mechanisms have been investigated successfully: a commercial client, ChartLink by Envitia, communicated with an open-source server, rasdaman by JacobsUniversity. "WPS-Cov adds coverage semantics to WPS; on the other hand, WCS is complemented with asynchronous processing, something tremendously useful for complex coverage analytics" explains Peter Baumann, editor of this specification as well as the WCS standards suite.

Hence, with its Web services portfolio OGC offers a tailorable, modular spectrum of standards ranging from simple access with WCS over agile analytics with WCPS to arbitrary algorithms with WPS. The recent specificationcontributed by Jacobs University closes the last gap, arriving at a seamless spatio-temporal Big Geo Data suite.

This sustained activity on flexible, value-adding coverage services has also attracted attention of the US Federal Geographic Data Committee Standards. In its October update published by Julie Binder Maitre the WCS suite components receive particular attention. FGDC reports that in its Fall 2014 meeting the Geospatial-Intelligence Standards Working Group (GWG) voted to "add the WCS Processing Extension as an emerging standard, with the intention to move it to mandated in a future baseline". Peter Baumann assesses, "we expect that after this announcement and 170 Terabyte datacubes queried through WCPS in ESA installations, uptake of OGC WCPS will increase significantly in the near future".

Big Data in the Geosciences

A workshop to be held at the 2015 IEEE International Big Data Conference.

Computing is fundamentally transforming science. We have entered the so-called Fourth Paradigm in which observational data is threatening to overwhelm scientists. There are now numerous examples of the five "V's" of BigData (Volume, Velocity, Variety, Veracity, and Value) impacting the geosciences. The rapid improvements in scientific instruments have led to a massive growth in data production. As data volumes often prohibit transferof such Big Data, it is becoming increasingly important to develop sustainable infrastructure, clever data management and reduction analytics, and community, around Geoscience Big Data. The increasing emphasis on sharing and collaborating has led to a growing focus on discovery and variety.

In addition, scientific data, and the processes used to analyze that data, can differ substantially from those encountered with Big Data in other fields. Moreover, educating the next generation of geoscientists is itself becoming a pressing issue.

This workshop focuses on highlighting the state-of-the-art research surrounding Big Data in the geosciences. The Workshop Chairs and Program Committee, with assistance from the Open Geospatial Consortium, seek to engagethe scientific community in understanding their big data challenges and problems. Finally, we aim to identify gaps in current technologies as well as gaps in the education of new data scientists. This workshop will provide a venue for participants to showcase innovative tools and services, build new collaborations, and hear from application scientists and educators about the challenges they face.

The Program Chairs of the workshop are Tom Narock (Marymount University, USA), Marshall Ma (Rensselaer Polytechnic Institute, USA), and Peter Baumann (Jacobs University Bremen, Germany).

rasdaman newsletter 08/2014

OGC announces Big Data Domain Working Group

13 August 2014 - At the recent OGC meetings in Geneva, members of the OGC formed an OGC Big Data Domain Working Group. This OGC domain group provides an open forum for discussions and standards recommendations for Big Data interoperability, access, and analytics related to geospatial information. To this end, the open forum will encourage collaborative development among participants representing many organizations and communities, and will ensure appropriate liaisons to other relevant working groups inside and outside OGC.

The group will consolidate findings on a public wiki to inform both the OGC membership and the greater public and allow for feedback during the editing phase and after. A report will be submitted to the OGC membership for publication as an OGC Best Practice paper.

Because location-based and geospatial data applications are major contributors to the Big Data deluge, the OGC is positioned to provide guidance on the use of OGC standards in managing such data. Further, with the advent of increased machine-machine communication, interoperability is gaining even more importance. OGC, therefore, is establishing a position addressing Big Data issues, including – but not limited to – science, implementation, market value, and societal effects.

The initial membership of the BigData WG will consist of the following members and individuals with extensive education and experience in Big Data issues:

  • Peter Baumann, Jacobs University (co-chair)
  • John Herring, Oracle (co-chair)
  • Juergen Seib, Deutscher Wetterdienst
  • Stan Tillman, Intergraph
  • Marie-Francoise Voidrot, Meteo France
  • Jeff de la Beaujardiere, NOAA
  • Bruce Gritton, US Navy MetOc
  • Chuck Heazel, WISC (co-chair)
  • Mike McCann, MBARI
  • Pedro Goncalves, Terradue
  • Don Sullivan, NASA
  • Ed Parsons, Google
  • Robert Gibb, Landcare Research New Zealand
  • Jean Brodeur, Geoconnections, NRCAN
  • Jinsongdi Yu, Fuzhou University
  • Arnaud Cauchy, Airbus Defence & Space

These members encourage others inside and outside the OGC to participate in discussions and in preparation of a final report. The OGC® is an international consortium of more than 475 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. Visit the OGC website at: http://www.opengeospatial.org/

rasdaman newsletter 06/2014

Big Science Data Coming to SQL Databases

Bremen, June 17 - Multi-dimensional sensor, image, simulation output, and statistics data make up for most of the "Big Data" in science and engineering. Last week, it has been decided to extend the SQL database language with massive multi-dimensional arrays. Initiated by Prof. Peter Baumann from Jacobs University, work has commenced on the forthcoming standard named SQL/MDA.

SQL has been tremendously successful in running any-size databases in business and administration. However, the "Big Data" in science are structured differently. Instead of simple tables, they often consist of multi-dimensional "data cubes". In Geo sciences, for example, this encompasses 1-dimensional sensor data, 2-D satellite imagery, 3-D x/y/t image timeseries as well as x/y/z geophysical voxel data, and 4-D x/y/z/t weather data. In Life sciences, there is laserscan microscopy and brain scans. And it can grow as large as simulations of the whole universe when it comes to astrophysics.

But SQL is not able to find, filter, and process such arrays, and consequently today arrays largely are maintained outside databases. Recognizing this shortcoming, Peter Baumann, Professor of Computer Science at Jacobs University Bremen, and his group have long been researching on ways to extend SQL appropriately. The rasdaman system which the group has established effectively has coined a new technology, Array Databases. In a recent technology demonstration, more than 1,000 computers have collaborated in a cloud to jointly compute the result of one single database query. This "distributed query processing" means a massive speedup, and research challenges on multi-Petabyte data cubes can be answered that hitherto were unsolvable.

Meantime, international data centers use this tool to allow scientists gain unanticipated insights into their spatio-temporal data cubes, and rasdaman installations can be found at NASA, ESA, British Geological Survey, Plymouth Marine Laboratory, Deutscher Wetterdienst, and many more.

At last week’s meeting in Beijing all national bodies participating in the SQL working group of ISO unanimously have agreed on the importance of arrays in SQL. Following thorough assessment of all options available, the group has accepted the proposal of Baumann for further elaboration. The new standard will be named ISO 9075 SQL/MDA, for "Multi-Dimensional Array"”.

rasdaman newsletter 05/2014

FOSS4G-Europe 2014 - Early Bird Registration extended to May 30 - Final Program released

It's been a busy few weeks as things start to come together for FOSS4G Europe 2014 – but now, admittedly with some delay, we have managed the challenge of compiling an exciting program, which you can find under: http://www.foss4g-e.org/program-glance

To give everyone time to have a look at the schedule before booking, we’ve extended the Early Bird registration deadline to May 30, 2014. In the meantime, many thanks to all who submitted abstracts – it has been a challenge to select the best of all excellent submissions passed in the last weeks. Altogether, the conference received about 78 abstracts in the general and 16 for the academic track. Every abstract has been carefully evaluated by the program committee to obtain a fair ranking. On the whole there has been a very satisfying presence of topics relating to the relevant Open-Source-tools like GeoServer, MapServer, Zoo, rasdaman, Grass, and several more. As well OGC-standards like WMS, WMS, WCS, WCPS, WPS, and SOS have the attention of the community. All in all the bandwidth of proposals shows a good representation of the general trends in the geospatial world. In addition the Local Organizing Committee has secured facilities to run additional workshops and a code sprint during the conference.

A lot of registrations form Europe and beyond have poured in already, so make sure you are part of the crowd in this visionary European geo happening. Sign in now! http://www.foss4g-e.org/

Rasdaman Company releasing Array Booster into Open-Source

Bremen, May 10, 2014 - The donator of the massively scalable open-source rasdaman Array Analytics engine, rasdaman GmbH, today announces the next substantial contribution: The performance-boosting direct file storage will be made available in the open-source rasdaman community. Of course, backward compatibility will be maintained: the new rasdaman version will smoothly run on existing databases.

"We wholeheartedly believe in open source, and we continue contributing to open-source rasdaman community all the time. We are glad that now we can provide this single major breakthrough in addition", says Peter Baumann, architect of rasdaman and CEO of rasdaman GmbH. "Several customers have expressed high interest in this feature. We love databases, but file-based analytics opens up scenarios where users decide not to run databases."

Until now, rasdaman stores its data in a PostgreSQL database. While this is operational in many places on this planet, a direct file-based storage can offer two distinct advantages. First, going directly to file is massively faster; preliminary tests show an improvement over PostGreSQL by almost a factor of two. Of course, PostgreSQL delivers good extra service, but if this is not needed (like often for satellite image archives) the new rasdaman storage will play out its full performance advantage.

A second advantage is simplicity: no PostgreSQL installation is needed, no database demon needs to run, so zero administration costs arise here.

One more time, this confirms the worldwide leading position of rasdaman in the rapidly expanding field of Agile Array Analytics. While research projects appear at the horizon, rasdaman remains the only fully implemented Array Database, operationally used on dozens of Terabytes.

Jacobs University maintains rasdaman community, supported by a worldwide community of contributors and the rasdaman company, which additionally offers the proprietary performance optimized version, rasdaman enterprise. As of this writing, rasdaman community is in its final steps for adoption by OSGeo Foundation. Further, it is the official OGC WCS Core Reference Implementation. In 2013, rasdaman has been awarded the prestigious Geospatial World Forum Innovation Award.

rasdaman info letter FOSS4G-Europe 03/2014

FOSS4G-Europe Welcoming Contributions

Independent software developers, scientists, industry experts, and agency representatives will come to meet at the FOSS4-Europe conference held in Bremen, Germany from July 15 to 17. Contributions can now be submitted on the conference website http://www.foss4g-e.org/ until April 15.

Under this year’s motto “Independent Innovation for INSPIRE, Big Data and citizen participation” the conference series FOSS4G-Europe (Free and Open Source Software for Geospatial - Europe) is uniting the GIS community from a broad spectrum of fields like geodesy, geo information, land management, and remote sensing. Industry and public authorities are expressly invited as they increasingly realize value and economic potential of free and open source products.

Several additional events make the FOSS4G-Europe in Bremen a unique event. A public viewing of the soccer World Championship final will kick off a week packed with events. The week will open with a workshop “Big Geo Data & INSPIRE” especially targeted at public authorities. INSPIRE Annex II and III will be discussed in relation to the pertaining standards of the Open Geospatial Consortium http://www.opengeospatial.org/. Canadian Jeff McKenna, President of the OSGEO Foundation, will open the event with his keynote talk. As a charter member McKenna has got involved with spreading and usage of open-source software for geo information systems for many years. In a plenary, the winners of the international academic NASA World Wind Challenge Europe programming contest will be announced and honored at the conference.

Being in the tradition of the worldwide FOSS4G conference series, this event will bring together open-source experts and enthusiasts from all over Europe and beyond creating a unique, visionary atmosphere.

Altogether, this unique technology demonstration paves the way for services merging data across data centers, nations, continents - ultimately, across the planet.

Conference Dates

General presentations and workshops:

  • Abstract submission: 2014-apr-15
  • Review notification: 2014-apr-30
  • Conference: 2014-jul-13

Scientific Track:

  • Abstract submission: 2014-apr-15
  • Full paper submission: 2014-apr-22
  • Review notification: 2014-may-15
  • Camera ready version: 2014-jun-15
  • Conference: 2014-jul-13
Early bird registration deadline: 2014-may-22

rasdaman newsletter 01/2014

First-Ever Direct Data Mix between US NASA and European ESA

rasdaman connects NASA Ames Research Center and ESA

Geneva, January 17, 2014. Commonly, Big Data are seen as "too big to transport" - however, often the real hindrance comes from security policies data centers have to implement to safeguard themselves. High-ranking international officials in Switzerland now have witnessed secure simultaneous access to NASA and ESA archives based on the rasdaman Big Data engine, effectively bridging the Atlantic in real time.

International delegates gathered in Geneva at the 10th Plenary of the Group on Earth Observations, GEO, to discuss future steps to create societal value from innovative hi-tech tools and data. On this occasion, a first-ever direct and real-time mixing of NASA and ESA data was demonstrated. The Web client on a laptop in Geneva accessed the NASA server to fetch a color image taken by a NASA drone, Ikhana, while it was supporting local firefighters combatting large bushfires in California. From ESA, an elevation model giving height information about the Californian hills was retrieved, originally obtained from a radar satellite. Both were combined in the browser to a 3-D landscape that could be zoomed into and viewed from all sides, similar to a virtual globe like GoogleEarth.

The client made use of the OGC Web Coverage Processing Service (WCPS) standard, a language that allows clients to task servers with processing and filtering on massive spatio-temporal data assets, such as satellite image or climate data sets. Access to the servers on both sides of the Atlantic was accomplished despite both machines are heavily secured by their operators. Not that security was compromised – the resulting work from the GEOSS Architecture Implementation Pilot (AIP) 6 showed that security-interoperable access is possible via so-called Access Management Federation setup. It establishes a well-defined level of trust between parties and supports Single-Sign-On: A user logging into at one organization of the federation will be trusted by all other members of the federation to perform managed access - and nothing else.

An international collaboration made this transatlantic gathering possible. Jacobs University and rasdaman GmbH provided the Big Raster Analytics engine, rasdaman, which NASA Ames Research Center and ESA are running on their operational servers. Secure Dimensions provided the integrated, overarching security tools. MEEO performed data ingest at ESA. Partners met during their collaboration in the OGC standards development and international research collaborations.

Altogether, this unique technology demonstration paves the way for services merging data across data centers, nations, continents - ultimately, across the planet.

OGC Adopts Extensions to Big Geo Data Standard

January 20, 2014. OGC now has adopted a series of functionality extensions to its Big Geo Data standard, Web Coverage Service (WCS). Web services implementing the WCS interface standard offer simple, yet powerful access to spatio-temporal sensor, image, simulation, and statistics data. The standards adopted now focus on data formatting and server-side processing of large-scale information to enable versatile, fast access to both detail and summary information.

The concept of coverages unifies regular and irregular grids, point clouds, and meshes. A coverage can be described as the digital representation of some spatio-temporal phenomenon, be it observed or generated. While ISO 19123 establishes an abstract model of coverages, the OGC GMLCOV standard refines this to a concrete, interoperable specification. The corresponding compliance tests offered by OGC can test compliance of implementations down to single pixel level.

Coverages can be served out via many OGC-based interfaces, such as WMS, WFS, WCS, SOS, and WPS. The broadest functionality set is offered by WCS, ranging from simple subsetting to complex analytics. The latest WCS extensions add in various functionality facets. For the encoding of grids in the GeoTIFF format, the WCS GeoTiff Coverage Encoding Extension [OGC 12-101] has been established. One or more "channels" or "bands" can be extracted conveniently from a multi-band or hyperspectral coverage through the functionality defined in the WCS Range Subsetting Extension [OGC 12-040]. Retrieval of grid coverages in reduced resolution is possible through the WCS Scaling Extension [OGC 12-039]. The WCS Interpolation Extension [OGC 12-049] serves to control interpolation in a server on all operations where interpolation occurs, such as CRS transforms and grid scaling.

A particular answer of OGC to the Big Data challenge is the Web Coverage Processing Service (WCPS) standard. This coverage query language allows clients to send queries in a simple language for evaluation on a server, in clusters, clouds, or federations of data centers. The WCS Processing Extension [OGC 08-059r4] connects the abstract WCPS language into WCS, allowing to send queries as WCS-style requests using any standardized protocol.

While WCS Core and Extensions are domain neutral and generically spatio-temporal, WCS Application Profiles bundle functionality for particular purposes and applications. The WCS Earth Observation Profile [OGC 10-140r1] adopted now establishes such an Application Profile for large-scale satellite imagery.

Altogether, WCS constitutes today's most powerful and flexible service environment for Big Geo Data appearing as coverages. The broad support by industry, academia, and government agencies during the vote underlines the broad worldwide uptake of WCS. A significant number of WCS implementations exists already, and many large vendors are preparing their own products based on WCS. A mapping of the INSPIRE Download Services to WCS is in preparation.

Comprehensive information about OGC's Big Coverage Data standards is provided at http://external.opengeospatial.org/twiki_public/CoveragesDWG/WebHome

rasdaman going Space

Darmstadt, January 16, 2014. The European Space Agency (ESA) has now confirmed selection of a Big Data engine, developed at Jacobs University Bremen in collaboration with a small hi-tech company, to go on board of a satellite. Goal is to demonstrate how satellites can process data on board and participate in planet-wide analytics networks.

Satellite imagery is indispensable for rapid response to disasters such as floods and wildfires, but also for everyday's governmental tasks, like land use monitoring: who is farming what and how much? However, data from a satellite take a long journey: when the satellite is flying over a ground station, it attempts to download as much of the imagery taken as possible. That is passed on to some data center which undertakes a lot of processing, puts the resulting products into its archive, and eventually publishes them. Not always is it easy for users to find the needle, that is: those data they are interested in, from the resulting haystack of data.

Researchers at Jacobs University want to change this now, allowing regular users to get data directly from the source. To this end, this image source, the satellite, will be instrumented to allow flexible processing and filtering of data on board. This is accomplished by the rasdaman system ("raster data manager"), an innovative analytics engine specialized on science and engineering data which is already in operation internationally, among others at NASA Ames and ESA. "A core challenge will be to micro-miniaturize the rasdaman engine", explains Prof. Peter Baumann, architect of rasdaman. Together with his team at Jacobs University and a research spin-off, rasdaman GmbH, he is preparing the system to use less power while fulfilling more tasks. This effort is not just a one-time endeavour: insights gained can help to save substantial amounts of electrical energy in computers on the planet in future.

In ESA's OPS-SAT project, a so-called CubeSat will be launched into an earth orbit. A CubeSat is a small, low-cost satellite which can be quickly and flexibly assembled for different configurations and tasks. CubeSats have been invented by US universities to overcome the high costs and time it takes to launch satellites, effectively hindering them from doing space research. While traditional remote sensing satellites cost hundreds of millions of Euros and have development times of a decade and more, CubeSats can be flexibly adapted to new experiments and launched with moderate costs. Today, the final specification of the satellite has been released for review by the Jacobs software engineers. Launch of OPS-SAT is foreseen for early 2016.

The OPS-SAT model hosts a Linux PC, camera, and GPS so that images and their position on earth can be recorded, processed, and transmitted to earth. Accessing the satellite will be done through open standards well known by geo service providers on Earth, so that no special programs are needed; any regular Web browser is sufficient. Therefore, the researchers see this as a contribution towards democratizing data access: on principle will allow anybody to access such a service via Internet - neither expensive receiving stations, nor huge data centers will be needed.

OGC navigator - Jacobs University supports OGC certification

Bremen. On the occasion of Intergeo 2013 Jacobs University introduces a new service adressed to software vendors: mentoring and conformance guidance for OGC certification.

This competent support implies extensive cost- and time savings. Varieties of geo software make the decision what product should be provided difficult: OGC, ISO, INSPIRE - many requirements software must fulfill to convince customers. Especially conformance with relevant standards will be checked critically. Conversely it´s difficult for suppliers standing out from the crowd and profiling themselves because everyone claims standard conformance. For this reason OGC has established standardized conformance testings - a seal of quality which is an important criteria providing software.

Jacobs University has coined relevant OGC standards for many years. Under the direction of Prof. Dr. Peter Baumann, the research team of the department of Computer Science has developed more than a dozen OGC standard specifications, i.e. the unified coverage model, WCS suite and all conformance testings for coverage services. Particularly with regard to INSPIRE Annex II und III where coverages play a central role, this worldwide unique expertise is of eminent importance for products claiming standard conformance. Jacobs University provides this expertise to general public. Seminars and advisory services will be offered to data and service providers.

“Principally each company is abled to carry out conformance testings for itself because these testings are freely accessible”, Prof. Dr. Peter Baumann, head of Jacobs University reseach team, explains. “However for staff it´s essential familiarizing themselves with underlying facts and practicing the test engine autonomously. This procedure will save time and money. We are cutting corners.”

For vendors this procedure implicates cost savings and accelerates time-to-market. Furthermore researchers offer more intensive testings on demand as ruled by OGC, which means an increase of product quality. Though Jacobs University guaranties strict confidentiality: no product leaves its place of development - only web access to product installation is necessary for testing. All results will only be communicated to the awarding authority. All testings are carried through under full cost-control. Every step of certification can be booked individually as well as advisory service.

Further information: http://jacobs-university.de/ogc and email p.baumann@jacobs-university.de .

rasdaman newsletter 11/2013

FOSS4G Europe Conference at Jacobs University Bremen

Bremen, November 2013. Europe's largest ever event on free geospatial and location-based software will be held at Jacobs University in Bremen, Germany from July 15 through 17, 2014. Also at this conference, the winners of the NASA Worldwind Europe Challenge will present their innovative apps to the public.

More than 500 delegates are expected; notably, the FOSS4G-Europe conference does not only attract software developers and open-source service companies, but also industry and governmental agencies which increasingly recognize the value and cost savings potential of open-source software. "We are proud and excited to have the honour of organizing the first pan-European FOSS4G", says Conference Chair Prof. Dr. Peter Baumann, head of the Large-Scale Scientific Information Systems (L-SIS) Research Group at Jacobs. Adds Organizing Committee Chair and L-SIS group member Susanne Ebrecht, "This event actually will contribute to bridging gaps between open-source and commercial software worlds". The main financial support is coming from the transatlantic Big Geo Data initiative, EarthServer, funded by the European Commission. Additionally, sponsors are being invited from industry, government, and other organizations.

FOSS4G (Free and Open Source for Geospatial) is a conference series held under the auspices of OSGeo. This is a worldwide acting, independent, nonprofit legal entity established to support the collaborative development of open source geospatial software, and promote its widespread use. Its international Board of Directors elected periodically by the OSGeo Charter Members, one of which is Prof. Dr. Peter Baumann. The OSGeo Board now has agreed to hold the FOSS4G-Europe Conference on the campus of Jacobs University in Bremen coming summer. Following an outstandingly successful FOSS4G-CEE, for Central and Eastern Europe, in Bucharest this summer, it was decided to widen scope to a pan-European event next year. Expecting over 500 participants this will be the largest event of its kind in history, only surpassed by the global FOSS4G conference. As part of this activity network, there are continental FOSS4G events held in Africa, Asia, and Latinamerica.

The NASA Worldwind Europe Challenge is an annual programming competition organized by NASA, Patrick Hogan, and Prof. Maria Brovelli, University Como, Italy. The challenge is to develop solutions that serve the spatial data needs of the European Community and respond to the INSPIRE Directive. Solutions sought will use NASA’s World Wind, an open-source virtual globe like Google Earth. Data used should relate to INSPIRE, the Infrastructure for Spatial Information in Europe. Examples of applications include a wildfire management tool, a tool for urban management, tools for weather display and climate research, and others.

The term free and open-source software denotes computer programs which, together with their blueprint (called the program's source code), can be distributed without ownership limitations and can be modified and improved by anybody. This prevents from software vendor dependencies and typically leverages resources from developer communities whose passion it is to continuously improve freely available software. As universities frequently are engaged in open-source projects, it is not uncommon that such projects actually define the state of the art. Prominent examples include the Apache Web server and the rasdaman array database system.

URL: http://www.foss4g-e.org/

OGC Releases Compliance Tests for WCS 2

Bremen, November 2013. The Open Geospatial Consortium (OGC) announced the new compliance test for the OGC Web Coverage Service (WCS) Interface Standard 2.0.

The OGC Compliance Program provides an online free testing facilityto check for proper implementation of the standards, and a process for certification of products that are compliant with OGC standards.

Once an organization's product has successfully passed a test, the organization can acquire a license to use the “Certified OGC Compliant” mark on web pages and other marketing material. The product is also promoted on the OGC Compliant Products Web page. The OGC invites organizations to start the process of getting their products certified. The Certified OGC Compliant mark increases potential buyers' confidence in a product, making the product easier to market and sell.

The candidate for Reference Implementation is rasdaman, available in open-source at http://www.rasdaman.org.

Release of the WCS test suite is in parallel with the release of further test suites for GML, WFS, SPS, and SOS.

Contact: Luis Bermudez, Director of Compliance, lbermudez@…

EarthServer Climate Data Service integrated at the European Space Agency

Bremen, November 2013. The EarthServer Climate Data Service has been successfully integrated at the European Space Agency: a powerful hardware infrastructure consisting of 40CPUs and 120GB RAM hosts the subsystems to handle EO products and model data.

Atmosphere and Climate products collected within the EarthServer project http://www.earthserver.eu as well as EO products provided by the European Space Agency, are presently accessible through the Multi-sensor Evolution Analysis (MEA) platform. The EarthServer platform runs rasdaman as its enabling analytics engine. The detailed list of EO products and model data is available at https://mea.eo.esa.int/data_availability.html.

Single Sign On (SSO) technology is used to authenticate and login in to the MEA system.

MEA provides to users the following functionalities:

• Real time data access and visualization (time series display)
• Data comparison (e.g. same variable from different data sources)
• On-the-fly data analysis (extraction of statistical parameters)

Honors for Jacobs Master Thesis on rasdaman

Bremen, November 2013. Jacobs alumna Mihaela Korte has been honored for her master thesis with the ‘Studienpreis für Mittelstandsforschung ‘ by local wiwib e.V.

Jacobs alumna Mihaela Korte has been honored for her master thesis with the ‘Studienpreis für Mittelstandforschung’. The prize is awarded biannually by local association wiwib e.V (Wirtschaft-Wissenschaft-Bremen)* in cooperation with the universities of the metropolitan region Bremen-Oldenburg.

Mihaela Korte from Moldova graduated from Jacobs University with an outstanding BA and MA in Computer Science. She wrote her master thesis on „Integrating user-defined functions into the querying process of an array database“ whilst also working at rasdaman GmbH with a focus on big data technology.

The 24-year-old was honored for the development of a new software technology which makes it possible to analyze large amounts of science and engineering data simultaneously with different programs. These programs are working collaboratively while fulfilling different tasks. Commonly, such huge data volumes are referred to as Big Data.

“While experts are still discussing which approach is the best for the analysis of data, Korte’s work as laid the foundations for different technologies to supplement each other. Her work has opened new perspectives for research and practical applications, “ comments Peter Baumann, Professor of Computer Science and Korte’s supervisor at Jacobs University. “Areas of application are manifold, ranging from climate analysis to disaster protection”. Mihaela Korte’s programs are meanwhile distributed by the company as part of its raster data manager software product, rasdaman. The Jacobs alumna has since moved to the US where she is working for a global software company.

*wiwib e.V. is a local association for the support of the Faculty of Business Studies and Economics at Universität Bremen. In cooperation with Bremen businesses and universities from the metropolitan region Bremen-Oldenburg it regularly honors outstanding Bachelor and Master theses.

rasdaman newsletter 08/2013

rasdaman 8.5 released

Bremen, August 2013. Jacobs University and rasdaman GmbH are proud to announce release of open-source rasdaman community 8.5.

This version includes numerous improvements and new features. Most important, performance has been increased again, obtaining a speedup of 2 - 3 orders of magnitude in many cases. Additionally, several more OGC WCS extensions have been implemented. See http://www.rasdaman.org/wiki/Download.

Following roll-out of 8.5, rasdaman 9.0 is planned for release this fall. It will contain substantial innovations, such as cloud and cluster parallelization, leading to unprecedented scalability. Another feature many data centers were eagerly awaiting is "in-situ storage": the rasdaman engine can answer queries not only on the database, but also on any pre-existing data archive structure. The rasdaman 9.0 version will be presented to the public this September at INTERGEO 2013 and several more national and international events.

OGC Progressing with Function-Rich Big Data Standards

August 20, 2013. Yesterday voting on the new OGC standard for Coordinate System handling of spatio-temporal data sets has finished with all YES votes. This extends the OGC Web Coverage Service (WCS) suite, together with several further specifications submitted earlier which are disclosed for public commenting. Together, this makes WCS Swiss Army Knife for big and small coverage data, such as sensor data, raster images, timeseries, point clouds, and meshes.

Among the supporters of the new "OGC WCS Extension - CRS" with OGC document number 11-053 are US NOAA and USGS, German AdV, French BRGM and METEO-FRANCE, Australian CSIRO, the European Union Stellite Centre, Oracle, and EADS. "This new standard closes an important gap, as for the first time it allows to treat spatial and temporal coordinates alike" explains Peter Baumann from Jacobs University, the editor of the specification, and details "this will tremendously ease access to and analysis of satellite image timeseries and weather data, for example".

Prior to voting on the adoption of a standard, as with WCS CRS, OGC mandates a public Request for Comment period where the candidate standard is made available to the public and comments are sought. The document editors duly consider all comments, and the revised version goes into voting.

In case of WCS, several further specifications are in this pipeline: WCS Range Subsetting, which allows to extract channels from hyperspectral images or multi-variable climate data sets; WCS Processing, which establishes a bridget between the simple data access protocol of WCS and the advanced filtering and processing query language of the Web Coverage Processing Service (WCPS); WCS Scaling, which allows clients to get downscaled data sets with reduced resolution, such as for Web browser display and thumbnails; WCS Interpolation, which gives clients fine-grain control over any interpolation the server performs during its processing.

A RESTful protocol binding for WCS is in progress by the OGC WCS Standards Working Group and will likely be rolled out soon as well.

"The team has worked hard over the last few years, and now we are reaping the benefits - WCS is the most powerful, comprehensive Big Earth Data service suite available" says Peter Baumann. "Not the least because we have harmonized it across OGC, so that coverages can be exchanged freely between WFS, WCS, WCPS, WPS, and SOS."


rasdaman inventor now OSGeo Charter Member

Bremen, August 2013. Peter Baumann, Professor of Computer Science, has been elected Charter Member of OSGeo.

The Open Source Geospatial Foundation, or OSGeo, is a not-for-profit organization whose mission is to support the collaborative development of open source geospatial software, and promote its widespread use. OSGeo also serves as an outreach and advocacy organization for the open source geospatial community, and provides a common forum and shared infrastructure for improving cross-project collaboration.

"It is an exciting opportunity to support the open source idea in all its variety", says Peter Baumann and explains, "Not only do the Charter Members elect the Board of Directors, but they also actively contribute in discussion." Baumann is well known in OSGeo circles with the Big Data Analytics engine, rasdaman, which is jointly developed by Jacobs University and rasdaman GmbH, a research spinoff. Research around rasdaman has pioneered the area of Array Databases, which is seen as a key Big Data technology by major database vendors. Further, Baumann is editor of international standards in the field and chair of several standards working groups.

More information:

rasdaman newsletter 07/2013

Space-Time Coordinates for Big Data: OGC Voting on Adoption

June 2013. The Open Geospatial Consortium (OGC) has started voting on adoption of the OGC Web Coverage Service (WCS) Coordinate Reference System (CRS) Extension Standard.This is the final step following a successful 60-day public Request for Comment period. Right after voting started, several institutions and companies have already voted in favor of WCS CRS. Voting ends on August 19.

WCS defines an open standard interface to easily access multi-dimensional, spatio-temporal coverages, such as sensor data, satellite imagery, image time series, simulation data, point clouds, and meshes.

The OGC WCS CRS Extension defines an extension to the WCS Core that enables requests to specify that coverages be returned in one of a number of different Coordinate Reference Systems. While the WCS Core defines access only in the coverage’s Native CRS (i.e., the one in which the coverage is stored on the server), the WCS CRS Extension defines how to request and obtain a coverage in a CRS different from the Native CRS. This flexibility is important, for example, in exchanging data between different environments.

The OGC WCS CRS Extension is free and can be downloaded from the OGC Request for Comment page at http://www.opengeospatial.org/

Always Take the Weather With You

June 28, 2013. At a meeting hosted by NASA, foundation was laid for Big Data support in the field of Meteorology, Oceanography, and Aviation. Rasdaman CEO Peter Baumann, being editor of the Web Coverage Service (WCS) standard, was invited to moderate the meeting from a standardization perspective. The rasdaman system will serve as an experimental platform for verifying concepts during development.

There are many reasons why pilots need to know the weather along their flight route, such as to avoid thunderstorms and minimize fuel consumption. However, this information is not always easy to obtain routinely. Similarly, international exchange of information between meteorological institutions is not a standard routine today.

In a sizzling hot New Orleans, an initiative of NASA, UK MetOffice, the French National Geographic Institute, and rasdaman GmbH / Jacobs University met this week at NASA Stennis to establish the core concepts of the so-called "WCS Application Profile - MetOcean" candidate OGC standard. Goal is to establish simple, yet powerful ways for accessing that specific part from the huge reservoir of weather forecast data which is of interest to a particular user. One example is obtaining the weather along an aircraft's flight path in space and time.

The underlying data range from two-dimensional to five-dimensional. Most systems today cannot serve such data, therefore the rasdaman Big Data Analytics engine was selected for trying out and varying concepts prior to their standardization. This will maximize practical usability of the forthcoming standard.

Peter Baumann, CEO of rasdaman and Professor of Computer Science at Jacobs University, was invited to moderate the meeting from an OGC standards perspective. He is chair of the raster-relevant working groups in OGC and editor of the Web Coverage Service (WCS) standards suite which was unanimously chosen as the trial platform for developing the specification.

Big Data from Space… directly to the Users

Frascati, 12 June 2013. How can we make the large amounts of Earth observation data accessible to a wider user community? To explore this idea, some 250 science, industry and policy-making representatives and national delegates from Europe, the US, Australia, China, and Africa met at ESA’s ESRIN space centre in Frascati, Italy for ESA’s first ‘Big Data from Space’ event.

Data from Earth observing satellites and other remote sensing technologies are growing in volume and diversity at an exceptionally fast rate. This poses challenges and opportunities for their quick access, stewardship and applications.

Peter Baumann was invited to present Jacobs University’s research on flexible, scalable search and processing on huge data archives, Overall, more than 50 presentations during the three-day conference stimulated discussion between the different communities involved in the business of providing and manipulating very large-scale data and complex analyses of satellite and in situ Earth observations.

Diversity of space data, the combined use of data from different space missions and the fusion of satellite data with non-space data lead to new types of user applications. This affects the way such data are collected, processed, delivered and preserved. There is an increasing need to serve a wider user community looking to retrieve simple, understandable and immediate information from remotely sensed data that have undergone complex processes and analyses.

Given the overwhelming success of the event, ESA is now considering whether this can become a recurring event.

rasdaman newsletter 04/2013

Spatio-Temporal Coordinates for Big Data: OGC Seeking Comments

April 2013. The Open Geospatial Consortium (OGC) announced a call for public comment on the OGC Web Coverage Service (WCS) Coordinate Reference System (CRS) Extension Standard.

WCS defines an open standard interface to easily access multi-dimensional, spatio-temporal coverages, such as sensor data, satellite imagery, image time series, simulation data, point clouds, and meshes.

The OGC WCS CRS Extension defines an extension to the WCS Core that enables requests to specify that coverages be returned in one of a number of different Coordinate Reference Systems. While the WCS Core defines access only in the coverage’s Native CRS (i.e., the one in which the coverage is stored on the server), the WCS CRS Extension defines how to request and obtain a coverage in a CRS different from the Native CRS. This flexibility is important, for example, in exchanging data between different environments.

The OGC WCS CRS Extension is free and can be downloaded from the OGC Request for Comment page at https://opengeospatial.org/standards/requests/100.

The OGC WCS Standards Working Group will consider all comments when preparing a final draft of the candidate standard. The comment period ends 5 May 2013. The OGC is an international consortium of more than 480 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. OGC Standards empower technology developers to make geospatial information and services accessible and useful with any application that needs to be geospatially enabled. Visit the OGC website at https://opengeospatial.org/contact.

Big Earth Data: The Programming Contest

April 2013. In the context of the ESA BigDataFromSpace Conference [1], the EarthServer initiative [2] is sponsoring a programming contest to establish innovative and surprising applications on multi-dimensional Earth data. The contest is supported by Jacobs University, rasdaman, ESA, and the European Commission [3].

Students worldwide are invited to build apps/clients for Big Earth Data. A variety of large datasets , ranging from 2-D through 4-D, is provided for your perusal.

No need for downloading, though - the rasdaman Array Database System gives versatile, fast query access on massive multi-dimensional raster data based on the OGC Web Coverage Processing Service (WCPS) open query standard. Clients compose query strings (typically from graphical user interaction), send them to the server for evaluation, and visualize the results coming back. To ease client development, a versatile Web development toolkit is provided, offering input widgets like sliders and gauges as well as output widges for diagramming, image display, and more.

Any common client platform technology (Android, iOS, HTML5, X3D, ...) is acceptable as long as there is open-source support for it.

Winners will surprise and impress the expert jury with a tremendously ( cool | geeky | useful | novel )+ client/app. The first three winners receive a money award and have the opportunity of presenting their development to a worldwide audience of experts in the "BigDataFromSpace" conference plenary.

A team of geeks is curiously awaiting for your ideas, and is prepared to assist where needed.

rasdaman 8.4. released

April 2013. Version 8.4 of the rasdaman Array Analytics Engine [1] has been released. Concurrently, this version is made available for the open-source variant, rasdaman community [2], and the proprietary enhancement, rasdaman enterprise [3].

The international team was breathtakingly busy adding features and closing tickets. Among the new features is a classification operator suitable for elevation coloring, but also for content-based image analysis. Maybe most important, the whole release process has been scrutinized and the trac wiki has been opened to the public for writing. For the community this means they have more overview and improved ways of impacting direction of development.

"By introducing semantic versioning we now can provide stable releases better while, at the same time, allowing developers to collaborate on new features" says Dr Beccati, senior software developer from Ferrara / Italy who has introduced these new features into the release management process.

Meantime development continues. This summer, rasdaman 9.0 will be released bringing along substantially enhanced functionality and novel parallelization features.

The rasdaman technology has coined the technology of Array Databases [4] and still today is the most advanced and operational system. It is reference implementation of the OGC Web Coverage Service (WCS) Core and Web Coverage Processing Service (WCPS). In a unique way rasdaman combines performance, scalability, flexibility, and functional power.

Rasdaman community is available for download as source code, installable RPM packages, and as preconfectioned Virtual Machine. For ways to obtain rasdaman enterprise, see the company website [3].

rasdaman newsletter 08/2012

The OGC adopts a Unified Coverages Encoding Standard

August 2012. The Open Geospatial Consortium (OGC®) has adopted OGC GML Coverages (GMLCOV) 1.0.1 as an official OGC standard. OGC "coverages" are a unifying, standardized concept for the digital representation of phenomena that vary over space and possibly time. Examples of such geospatial coverages include Earth images, referenced and non-referenced rasters, curvilinear grids, and point clouds.

A corrigendum version has now been issued for the OGC GML 3.2.1 Application Schema - Coverages, nicknamed GMLCOV. Based on OGC Abstract Topic 6 (which is identical to ISO 19123), GMLCOV provides a unified method for representing OGC coverages in GML and any other suitable data format, such as GeoTIFF and NetCDF, and in a variety of dimensions, including 2-D x/y, 3-D x/y/t and x/y/z, and 4D x/y/z/t dimensions. As GMLCOV is independent of a particular service definition, it allows coverage data to be exchanged through different types of services that implement these OGC standards. As such, GMLCOV integrates Geography Markup Language (GML), the OGC Sensor Web Enablement (SWE) Common Standard, the OGC Web Coverage Service (WCS) Interface Standard, and others.

The new version, GMLCOV version 1.0.1, provides several enhancements, such as versatile encoding options that include binary formats with or without GML header. To better support this, modularization of the specification has been streamlined.

The OGC coverage model is widely used. For example, the European Commission has supported the development and unification of OGC coverages because these coverages are an important element in the standards framework for INSPIRE (the European legal framework for a common Spatial Data Infrastructure). Also, the widely used GMLJP2 standard, which augments JPEG2000 imagery data with GML-encoded geospatial metadata, relies on the GMLCOV standard.

GMLCOV is free and can be downloaded from at https://portal.opengeospatial.org/files/48553. The OGC is an international consortium of more than 460 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. OGC Standards empower technology developers to make geospatial information and services accessible and useful with any application that needs to be geospatially enabled. Visit the OGC website at http://www.opengeospatial.org/contact.

DREAMing Service Innovation Forward

While "Big Data" increasingly is on the radar of all of us, another big wave is approaching us from behind: the quest for substantially more flexible, powerful, and user-friendly access to geo information. The DREAM (Decision Support and Real Time Earth Observation Data Mangement) project, which is funded by the European Space Agency (ESA), has set out to establish novel solutions or data centers and their users. Now DREAM has finished a concise requirements study paving the way to innovative services.

August 2012. The DREAM consortium, represented by rasdaman GmbH, has proposed several standard drafts extending WCS with essential functionality like image scaling, band selection, and reprojection. These drafts are currently being discussed within OGC, with a planned adoption aiming at the end of 2012.

The DREAM project addresses two core issues of geo service provision: on the one hand, integrated management of high-volume, heterogeneous satellite data; on the other hand, user-driven, quick, and flexible filtering and processing of data stored in such large-scale archives gets tackled. Operational installations will be provided for EUSC (European Union Satellite Centre) near Madrid, EMSA (European Maritime Safety Agency) in Lisbon, the ESA data center in Frascati, and NLR (National Aerospace Agency) in the Netherlands. In the first project period, which has ended now, requirements have been elicited stemming from the divergent missions and day-to-day requirements of these data centers.

Following an ESA requirement, DREAM puts strong emphasis on the use of open, interoperable standards. Not only does this mandate the use of standards-conformant tools; in addition, DREAM actively contributes to advancing standardization by providing new specification components. Focus here is on the main OGC standard set supporting multi-dimensional raster data, the Web Coverage Service (WCS) suite. The GML Application Schema for Coverages (GMLCOV) establishes a common conceptual model of coverages (including raster data), while the WCS Core defines basic service components. WCS extensions add further functionality facets which a vendor may choose to implement. In DREAM, particular emphasis is on the data extraction and processing facilities, such as the Web Coverage Processing Service (WCPS) raster query language.

The rasdaman raster analytics engine, available for free on at http://www.rasdaman.org, provides the platform for developing the new OGC standards specifications, such as support for data formats like LIDAR and GML JPEG2000. Right now, rasdaman is already the OGC candidate reference implementation for WCS. Further, rasdaman has been included in the OSGeo Live set of particularly recommended open-source geo projects and as the only multi-dimensional raster database supporting PostgreSQL and PostGIS.

Latin American Industry Looking At Rasdaman and OGC Standards

July 2012. The markets of Latin America grow continuously. A main reason leading to this increasing economic importance of South American countries lies in their role as world-wide suppliers of oil, gas, and mineral resources. At the NRDM (Natural Resources Data Management) conference in Rio de Janeiro, representatives of industry and research discussed about challenges in exploration data management. In his presentation, Dr. Peter Baumann, managing director of rasdaman GmbH, convincingly argued the potential for cost savings by using open standards. His presentation titled “Geo images, sensors and Web Statistics: meet the Standards“ is available on request.

Multinational players like Shell, Chevron, Petrobras and Dell presented their view on data interoperability and integration. Especially exploration industry showed high interest in the multi-dimensional capabilities and the scalability of the Big Raster Analytics Server, rasdaman. The international professional audience attested rasdaman GmbH unanimously its leadership in geo technologies.

rasdaman newsletter 05/2012

Raster Engine rasdaman 8.3.1 released

The new rasdaman version 8.3.1 has passed the conformance test suite of the OGC WCS Core standard and is now a candidate reference implementation. As usual, rasdaman is available in an open-source and a proprietary variant. The open-source rasdaman community can be downloaded from www.rasdaman.org; additionally, it can be tested and installed via OSGeo Live (live.osgeo.org).

New features of the optimized version 8.3.1 include GDAL-based import through the rasgeo component which supports practically all relevant image formats. Further improvements address new platforms (OS-X), increased hyperspectral support, a web client toolkit (raswct) for rapid client development, integration with 52 North WPS, and general usability as well as numerous fixes. Landcare Research New Zealand have contributed, as well as NOAA people, St. Petersburg Polytechnic University, Ferrara University in Italy, Jacobs University, and rasdaman GmbH. This underlines how much of a community project rasdaman meantime has become. Currently the project continues with extending the OGC Web Coverage Service (WCS) with further specification modules, including their implementation in rasdaman along with conformance tests. This way, rasdaman has become an indispensable tool for the advancement of geo service standards.

German Innovation Award 2012 for Big Data Analytics

The research spin-off, rasdaman GmbH, has been awarded the INNOVATIONSPREIS-IT (Innovation Award IT) 2012 in the category “BEST OF 2012 Open Source”. The national jury thus acknowledged the world technology lead of the high-performance raster analytics engine, rasdaman (“raster data manager”), selected from over 2,500 applications.

The rasdaman technology enables fast and flexible access to multi-dimensional sensor, image, and statistics data of unlimited volume; examples include 1D time series, 2D satellite images, 3D x/y/t image timeseries and x/y/z geophysics data, as well as 4D climate data. Being domain independent, rasdaman equally well supports application domains in Earth, Space, and Life sciences, to name but a few. Among the distinguishing features is the capability of answering ad-hoc questions to perform filtering the desired data from large-scale archives and to process the extracted data into the shape users need them. This includes fusion of data coming from different locations and having different resolution, extent, and structure. The enabling idea is to combine standard database query languages with image processing languages, resulting in a novel, powerful means to access massive archives over the Web. Meantime this principle has found its way into international standards – the Open Geospatial Consortium (OGC) has crafted its raster query language, WCPS, along the rasdaman concepts. While rasdaman has been developed over 17 years into a stable, mature technology, international efforts have emerged with similar goals, most remarkably the US SciDB initiative. Under the patronage of German national commissioner for information technology, Cornelia Rogall-Grothe, and IBM Germany the INNOVATIONSPREIS-IT is in its ninth year of scouting IT innovations.

OSGeo recognizes Effort in Big Geospatial Analytics

Being selected by the Open Source Geospatial Foundation (OSGeo) to become part of their “OSGeo Live” demonstration DVD dignifies a project as being particularly topical, valuable, high-quality, and innovative. The rasdaman geo analytics engine has been added to OSGeo Live 5.5 as the only geo raster server capable of handling unlimited volumes of multi-dimensional “Big Data”.

OSGeo Live is a self-contained bootable DVD, USB thumb drive or Virtual Machine based on Xubuntu that allows to try a wide variety of open source geospatial software without install¬ing anything. It is composed entirely of free software, allowing it to be freely distributed, duplicated and passed around. OSGeo Live can be downloaded from live.osgeo.org and several mirror sites. “This rewarding acknowledgement of rasdaman in the geospatial Big Data world underlines the importance of going the next technology step, beyond mere map serving; geo analytics is indispensable in today’s flow of geospatial information”, comments Peter Baumann, principal architect of rasdaman and well-known editor of OGC raster service standards. In parallel, rasdaman has been given “incubation” status in OSGeo’s open-source geospatial software branding program. Being acknowledged by OSGeo means that not only the software itself, but the project overall has been thoroughly examined in all aspects. Criteria considered include geo relevance, code quality, documentation, strictly open-source code provenance, community, and project management. Every project gets assigned a voluntary mentor; for rasdaman, Bruce Bannerman from the Australian Bureau of Meteorology has kindly accepted this task.

OGC Adopts Enhanced Spatio-Temporal Coverage Service

The Open Geospatial Consortium (OGC) Web Coverage Service (WCS) Standard establishes an open, interoperable service interface for accessing, filtering, and processing of multi-dimensional spatio-temporal coverages, such as sensor time series, satellite imagery and image time series, and climate data. Following the complete overhaul of WCS in 2010, leading to version 2.0, now an enhanced corrigendum version 2.0.1 has been produced.

En detail, WCS actually comprises a modular suite of standards based on the common WCS Core 2.0.1 becoming available now. The GML Application Schema for Coverages defines OGC’s common coverage model, allowing coverages to freely be exchanged between services implementing different standards such as GML, SWE, WCS, and WPS. Further service extensions include the Web Coverage Processing Service (WCPS) standard which establishes a multi-dimensional raster query language. The Earth Observation Application Profile (EO-WCS) is a candidate standard which specializes WCS for the satellite imagery domain. “The new version satisfies stakeholder requests received and offers significantly improved functionality, flexibility, and usability”, says Peter Baumann, chair of the WCS Standards Working Group and editor of the specifications. Within the next few days, OGC will make new specification available on the OGC website where it can be found at www.opengeospatial.org/standards/wcs. The OGC is an international consortium of more than 435 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. OGC Standards empower technology developers to make geospatial information and services accessible and useful with any application that needs to be geospatially enabled. Visit the OGC website at www.opengeospatial.org.