rasdaman newsletter 09/2019

SQL Now With Datacubes

The most recent enhancement to the SQL query language has been published by the ISO standardization body: Multi-Dimensional Arrays (MDA). MDA allows for powerful, efficient queries on massive "datacubes" as they appear in spatio-temporal satellite image timeseries, human brain imaging, astrophysical simulations, as well as business and financial data, to name but a few. SQL/MDA is seen as a game changer in Big Data Analytics.

Since decades, SQL is lingua franca for the management of large and complex data. One reason for this sustained success of SQL is the continued innovation, driven by the ISO SQL working group. "We continuously incorporate new ideas if we see that they are adding substantial value and are elaborated with sufficient rigour" explains Keith Hare, SQL working group convenor and a leading SQL expert.

When the rasdaman team approached SC32 WG3 for adding declarative array support in 2014, WG3 members recognized the value proposition. "Array support closes a gap in large-scale data services; it is a groundbreaking enhancement to the SQL database language", underlines Hare. After own investigation ISO chose the rasdaman query language due to its substantial advantages, such as formal semantics, powerful and practice proven, and fitting seamlessly into the relational model. Over several years SQL/MDA developed in a tight collaboration of rasdaman and the SQL team. Released in summer 2019, it allows for powerful, efficient queries on massive "datacubes" as they appear in science, engineering, business, and other fields. The full designation for this part of SQL is: ISO/IEC 9075-15:2019 Information technology database languages -- SQL -- Part 15: Multi-dimensional arrays (SQL/MDA).

Historically, accessing image metadata is fast and relationally queryable whereas accessing data (such as image pixels) is slow, not queryable, and can only be downloaded as format-encoded files. With SQL/MDA, queries can address both data and metadata simultaneously. "This integration overcomes the age-old distinction between data and metadata, leading to a new quality of data services. This is an exciting perspective," explains Baumann who adds "not least, providing analysis-ready data boosts Machine Learning to new scalability."

rasdaman Ranked Top Open-Source Project

September 2019 - Independent software experts from OpenHub attest rasdaman outstanding quality in comparison with more than half a million further open-source projects. They characterize open-source rasdaman community as mature with a large, active development team, high software quality, and overall high value in commercial terms.

Such performance indicators are increasingly important as, according to OpenHub, today 55% of companies leverage OSS for their production infrastructure. OpenHub, run by Black Duck Software, Inc., USA is a public directory of free and open source software (FOSS) providing quality indicators based on standard software engineering metrics. It continuously monitors a breathtaking 676,791 open-source projects with altogether over 21,000,000,000 lines of open source code. Generally, OpenHub is considered a reasonable measure for open-source project quality.

About rasdaman OpenHub says on the related project page, Openhub.net/rasdaman:

  • large, active development team: This project has a large team with 85 contributors, in the top 10% of all project teams on Open Hub.
  • well established, mature codebase, stable year-over-year commits: The first lines of source code were added to rasdaman in 2009. OpenHub believes that rojects with recent activity, and a code base more than five years old are likely solving vital problems and delivering consistent value, and may be organized to reward sustained effort by an engaged team of contributors. Notably, OpenHub sees open-source rasdaman only since rasdaman GmbH donated the code in 2009, at which time the technology had already seen over 12 years of development. Hence, all OpenHub figures need to be seen as increments over this initial value.
  • estimated value: over 8 million US$, generated through an estimated 150 person years of effort.
  • no vulnerabilities are reported.

In addition to the numerous innovation awards acknowledging the novelty of this software (such as the latest TechConnect? Boston 2019 Innovation Award), the OpenHub valuation represents a clear signal to technology users that rasdaman is not only safe and stable, but actually provides high value for Big Datacube operators.

More information: rasdaman.org

Datacube Analytics through OGC Standards

This chapter of a book released by ESA describes datacube analytics based on the OGC coverage (ie, datacube) standards: Fostering Cross-Disciplinary Earth Science Through Datacube Analytics.

rasdaman: DatSci Data Science Innovation of the Year Finalist

Dublin, September 2019 - The rasdaman datacube engine is finalist in the 2019 DatSci & AI Awards. This prestigious European award recognizes rasdaman for its innovation, its advancement of data science in industry and business, and as such also its outstanding business impact.

Data Science has arrived at the heart of industry and society as a key enabling combination of skills and technologies. Edward McDonnell, Director of CeADAR, The National Centre for Applied Data Analytics & AI says, "The relentless pace and expansion of the analytics industry is reflected in the addition of new award categories covering AI, Industry 4.0, health and wellbeing. In fact, there are very few industries today that are not experimenting with or implementing analytics and AI solutions". Linda Davis, founder of the DatSci awards, adds that the European Commission released a report in 2017 that the amount of data workers in Europe will grow to 10.43 million by 2020.

The European DatSci & AI Awards are driving data science forward by recognizing the best scientists, companies, and technologies in the field every year. "The DatSci & AI Awards is on a mission to raise the profile of data science in Europe, especially with the exponential growth expected in the next two years," says Linda. "The awards in 2019 are the biggest to date, and the event itself will highlight the tremendous impact Europe’s data scientists are making across all fields." The Awards recognise excellence in Data Science and Artificial Intelligence across industry, education and social responsibility in Europe in 10 Award Categories. This year, DatSci proudly cooperates with BDVA and CeADAR. The Big Data Value Association (BDVA) is the private counterpart to the EU Commission to implement the Big Data Value PPP program. CeADAR is the Irish National Centre for Applied Data Analytics & AI. The 4th annual Awards Ceremony will take place on the 5th September 2019. The full-day event and Awards, hosted at Croke Park in Dublin, Ireland will connect and celebrate the best of the data science and artificial intelligence ecosystem across Europe. One winner will be awarded for each category and will be announced at the Awards on 5th September 2019.

The independent DatSci judges have honoured several aspects when selecting rasdaman: its contribution to analysis-ready Earth data by establishing datacubes; the scalable analytics capabilities of the rasdaman engine; and its strong impact on datacube standardization. "The team is happy and proud about this recognition, which stands in line with further international awards, like TechConnect in the US recently", says Peter Baumann who is rasdaman architect, inventor of queryable datacubes, and initiator and editor of the datacube standards in OGC and ISO together with his team, with strong impacts into INSPIRE and other communities. "A highlight of our ongoing work 2019 certainly was the publication of the SQL datacube standard by ISO. We have been overwhelmed by the feedback of the ISO SQL experts when we proposed this extension back in 2014, and the joint effort now has led to a game changer in Data Science and AI", says Baumann.

More information: DatSci.com

rasdaman special newsletter 08/2019

Risikoerfassung von Krankheiten auf dem Feld per App (in German)

Erstmals haben Wissenschaftler mit BigData-Analysen Satellitenbilder und Wetterdaten kombiniert, um Frostschäden in Winterkulturen eindeutig zu identifizieren und Handlungsempfehlungen geben zu können. So lassen sich schnell mehrere hunderttausend Felder in ganz Deutschland untersuchen und per App Tipps für jeden einzelnen Schlag geben.

Pflanzenschutz, Düngung, Beregnung – ja oder nein, wieviel davon und wo genau? Das Verbundprojekt BigPicture hat hierzu Antworten gefunden. Satellitenbilder zeigen, wo auf dem Feld die Pflanzen gut oder schlecht wachsen. So sehen Landwirte mobil und unmittelbar, wo Handlungsbedarf besteht und können auf Teilflächen gezielt eingreifen.

„Wir wollen nicht nur wissen, dass die Pflanzen in einem Bereich des Feldes besser wachsen, sondern auch warum. Nur dann können wir Empfehlungen geben, welche Behandlung für die Teilfläche richtig ist,“ sagt Projektleiterin Dr. Katrin Kohler von der Spatial Business Integration GmbH in Darmstadt, die mit der Bremer rasdaman GmbH die Kompetenzen aus Satelliten- und BigData-Technologien vereint.

Ganz Deutschland in einer halben Stunde analysiert: Viele Faktoren wirken auf das Pflanzenwachstum und müssen bei der Diagnose berücksichtigt werden: Bodeneigenschaften, gefolgt von Nährstoff- und Wasserversorgung sowie Krankheiten. Dementsprechend müssen die Eigenschaften jeder Teilfläche und die lokalen Wachstumsbedingungen anhand unterschiedlicher Daten untersucht werden. „So stellen wir fest, mit welcher Wahrscheinlichkeit eine Krankheit auf einer Teilfläche ausbrechen könnte“, erklärt Kohler.

In einem Fall kombinierte das Projektteam Satelliten- mit Wetterdaten, um Frostschäden oder Gelbrostrisiken in Winterkulturen eindeutig identifizieren zu können. So steigt das Risiko eines Gelbrostbefalls in Winterweizen mit zu- nehmender Dichte und feuchtem Mikroklima im Bestand. Dies kommt oft in Tallagen eines Feldes vor. Deshalb führten die Wissenschaftler entsprechende Informationen aus den Satellitenbildern und einem Digitalen Höhenmodell für die Risikoanalyse zusammen. „Eine solche Analyse, ausgeführt über hunderttausende von Feldern in ganz Deutschland, dauert nicht viel mehr als 30 Minuten,“ erklärt Prof. Peter Baumann von der Jacobs Universität Bremen, der die Daten mit Hilfe der hochskalierbaren und leistungsfähigen Datenwürfeltechnologie zusammenführt.

Den Landwirten sollen demnächst konkrete Handlungsempfehlungen für ihre Teilflächen zukommen. Dafür haben die Forscher bereits die App "Solorrow" entwickelt, die Landwirte kostenlos im App-Store herunterladen können. Diese stellt Landwirten bereits jetzt Satellitenbildkarten zur Verfügung, auf denen Teilflächen im Feld anhand ihres mehrjährigen Wachstumspotenzials unterschieden werden und eine angepasste Düngung unterstützen. In Zukunft soll diese um kombinierte Daten ergänzt werden. Das Bundeslandwirtschaftsministerium fördert das Verbundprojekt "BigPicture" noch bis August 2019 in seinem Programm zur Innovationsförderung über die Bundesanstalt für Landwirtschaft und Ernährung als Projektträger.

Lesen Sie mehr auf der Website der Bundesanstalt für Landwirtschaft und Ernährung.

rasdaman newsletter 07/2019

Amazon and rasdaman Teaming Up in Cloud Research

Seattle, summer 2019 - Amazon has assigned financial and technical support to rasdaman for advanced Earth datacube services in the Amazon cloud via its AWS Cloud Credits program. Particular emphasis will be on securing services and intercontinental federation aspects.

At TechConnect Boston where rasdaman received this year's innovation award, rasdaman CEO Peter Baumann had a strategic meeting with a team of AWS engineers and managers. They were impressed by rasdaman's power and versatility and particularly the federation capabilities, and further talks were initiated. AWS considers rasdaman an outstanding state-of-the-art application for the Earth and is interested in advancing AWS-based Earth datacube services.

The federation capabilities are utilized, among others, in this year's OGC Testbed 15 where rasdaman contributes its unique location-transparent federation for interoperability experiments between USA and Europe. Being highly parallelized (without a single point of failure) rasdaman acts as a distributed federation, within clouds and between data centers; workload-adaptive partitioning with several strategies is offered as a tuning parameter. Alternatively, existing archives can be accessed by just registering instead of copying, so without data duplication. Federations of >2.5 PB have been built this way. Rasdaman understands >100 formats. Single queries have been dispatched over more than 1,000 Amazon cloud nodes.

The AWS Cloud Credits for Research Program is aimed at new projects that seek to build a cloud-hosted service, software, or tools and/or migrate a research process or open data to the cloud. Amazon's Earth data assets the rasdaman view provided by rasdaman will be made available for direct analytics and visualization in a particularly user-friendly and standards-based manner.

More information: Dr. Peter Baumann

PARSEC SME Accelerator Call

Brussels, July 2019 - EARSC and the PARSEC consortium are happy to announce that foreseeably in Fall the Open Call for the PARSEC accelerator programme will start.

PARSEC is a business accelerator supporting the creation of new Earth Observation-based products and services. The accelerator targets SMEs, start-ups, entrepreneurs and researchers who want to have an impact on the emerging Food, Energy and Environment sectors. The holistic acceleration programme will distribute a total of €2.5 Million equity-free funding to 100 beneficiaries, alongside other key resources (coaching, matchmaking, promotion) needed to develop and launch 15 new products and services into the market.

Businesses selected will get access to a state-of-the-art datacube infrastructure based on rasdaman, the world-leading datacube analytics engine, which is a proven business accelerator. GeoMatrix will provide value-adding services to boost ramp-up of the SME's Earth Observation based services. A huge amount of data will be made readily available, ranging from DIASs to the data centers federating via rasdaman.

PARSEC is supported by the European Commission through an H2020 grant.

If you are interested in PARSEC, please join community now by taking part in this survey. By answering this survey, you are letting us know what are your needs and you can have an impact on PARSEC support services.

You can follow PARSEC Accelerator on the following platforms:

  • Twitter: @PARSEC_EU
  • Facebook: @PARSEC.Accelerator
  • LinkedIn Group: PARSEC Accelerator


  • Marion Bouvet, EARSC
  • Peter Baumann, rasdaman GmbH

rasdaman Receives US TechConnect Innovation Award

Boston, June 2019 - Every year, the innovation award is a highlight of the TechConnect event in Boston, USA. This year, the rasdaman datacube technology received the prestigious award from the industry prospectors looking for breakthrough technologies.

The TechConnect Innovation Awards identify the top 15% of submitted technologies as ranked by the TechConnect Corporate & Investment Partner Committee. Innovation rankings are based on the impact the submitted technology will have on a specific industry sector. Innovations are submitted globally, ranging from academia over government and corporate research laboratories to small and large industry.

"We are proud about this leadership acknowledgement expressed by TechConnect Boston", says Peter Baumann, Principal Architect of rasdaman and CEO. "The continuous line of innovation awards, such as the NITEC Innovation Award 2018, is a strong incentive for the team to maintain its strong pace of innovation."

For over 20 years the TechConnect World Innovation Conference and Expo has connected top applied research and early-stage innovations from universities, labs, and startups with industry end-users and prospectors. TechConnect has published over 10,000 technical papers, connected over 20,000 innovations with industry partners, provided prospecting to most all Fortune 500 technology companies, and supported most every U.S. Science and Technology agency since its inception.

More information: TechConnect Innovation Awardee

Advancing Coverage APIs: the OGC OpenAPI Hackathon

London, June 2019 - Following weeks of intense preparation by OGC experts, Ordnance Survey has hosted an OGC hackathon in London to get implementation feedback. Coverage work is continuously supported by the rasdaman experts being the prime shapers of the coverage standards. Now work is being consolidated and fed into the OGC adoption process.

With OpenAPI, a new API style has emerged among geo communities. The Open Geospatial Consortium (OGC) is fostering this with dedicated activities, at the same time using the opportunity for homogenizing the existing Web service specifications. In support of this, OGC has organized a Hackathon to advance OGC Application Programming Interface (API) specifications. This hackathon has tested draft OpenAPI-based standards for coverages, map tiles, and processes using a common template based on the OGC API for features.

The Hackathon was run by Standards Working Group (SWG) Chairs, with support from OGC staff. A team from rasdaman participated to give support on all levels. On conceptual level, Peter Baumann as WCS.SWG chair and prime editor of the OGC and ISO coverage standards contributed architectural advice to ensure API consistency and compatibility with the existing standards suite, so as to safeguard existing investment into data and service assets. Vlad Merticariu enhanced the rasdaman coverage reference implementation with OpenAPI access, which subsequently was successfully used by EURAC and Sinergise clients.

The Hackathon ran from June 20th 09:00am through June 21st 05:00pm. It was hosted by Ordnance Survey at the Geovation Hub in London. The event was instrumental to the evolution of the OWS standards to a modern API based approach, setting the course for open geospatial standards for the next decade. The hackathon was organized around Coverages (WCS), Map Tiles (WMS and WMTS), and Processing (WPS). A bridging between coverages and tile services could be demonstrated.

The outputs of the hackathon will inform the development of the OGC API - Common specification and the OGC API - Features specification. Other specifications that will benefit from the work done at the hackathon include the OGC API - Processes specification and the OGC API - Coverages specification.

More information: OGC Open API hackathon

rasdaman newsletter 01/2019

EGU 2019: news and insights about Big Earth Data

  • From Mystery to Mastery: Spatio-Temporal Coverage Standards Explained (Peter Baumann, Vlad Merticariu)
  • Implementation of software components to build web-based decision support systems: the case of the land take pipeline within a geospatial cyberinfrastructure (Giuliano Langella, Peter Baumann, Francesco Vuolo)
  • Big data and machine learning in geosciences (Mikhail Kanevski, Peter Baumann, Sandro Fiore, Kwo-Sen Kuo, Nicolas Younan)

German DIN Supports Foundations of Datacubes

January 2019 - What is a coverage, after all? This fundamental spatio-temporal data structure, encompassing (among others) datacubes, is widely used, based on adopted and implemented OGC and ISO standards like WCS. However, the conceptual foundations are outdated. This is being changed now.

On a high level it is clear: ISO 19123, which is identical to OGC Abstract Topic 6, describes the foundations of coverages. Accordingly, coverages encompass regular and irregular spatio-temporal grids, point clouds, and general meshes. However, this standard - crafted around the year 2000 - is not up-to date and needs urgent revision according to ISO and other experts. Definitions like "a raster is like a grid of pixels on a cathode ray tube" are not exactly representing modern understanding.

Therefore, ISO since several years has set up a New Work Item Proposal (NWIP) based on which project leader Prof. Liping Di at the US George Mason University in the US is looking for a writer of the specification. Peter Baumann, rasdaman CEO, was approached, but could not do it earlier due to urgent work on other standards. In 2018, an agreement was finally reached with DIN, the German standardization body, to establish a national project led by Peter Baumann to establish a new abstract coverage specification.

ISO has decided to split the coverage standard into a pair of companion standards, 19123-1 (abstract foundations of coverages) and 19123-2 (concrete, interoperable coverage definition which allows mapping to a variety of data format encodings). The latter, 19123-2, has been adopted by ISO in 2018 from the corresponding OGC Coverage Implementation Schema (CIS) in an ISO project led by Peter Baumann. For the former, 19123-1, the so-called DIN-CONNECT project has submitted a complete specification by the end of the year, fully according to plan. This coverage specification will now become German national standard and simultaneously will be submitted to ISO TC211 as German input to the project.

"It is extremely important to have a coherent set of standards", says Peter Baumann. "We have put utmost emphasis on establishing a specification that is state of the art, open for the future, and yet compatible with the existing coverage standards ecosystem".

Security, Advanced Clipping, Plus More: rasdaman 9.7 Released

January 2019 - the new version rasdaman 9.7 has been released. Once more a major step forward has been accomplished by all contributors. Most notably, raster/polygon clipping has been addressed in full. Further, multi-dimensional visualization capabilities have been improved.

185 individual code contributions have been submitted to rasdaman, have undergone a thorough quality check, and finally have been incorporated in the codebase. So-called curtain queries and corridor queries allow much more concise selection of data from massive spatio-temporal datacubes. This is an important use case, for example, in aviation: to extract a corridor from a weather datacube specifically along the trajectory of an airplane. This way, the airplane does not need to see a sub-cube (which would be prohibitively large), but exactly the weather information of the points it passes, and at its appropriate time, with some buffer. Of similar importance are curtain queries giving arbitrary vertical profiles.

Specifically for visualization, WMS capabilities have enhanced so as to support three and more dimensions, instead of the flat maps generated earlier.

"We are proud to serve the community with the best available combination of functional power, flexibility, scalability, and open standards", says Peter Baumann, Principal Investigator of rasdaman and continues, "and it is exciting to see the success story of datacubes following their invention by us back in 1992".

As usual, see rasdaman.org for options to download and use rasdaman.

Donation for Karl-Heinz Böhm Foundation "Menschen für Menschen"

Our company has been quite successful in 2018: research projects and license sales, and some spicy technological challenges for the year to come.

Time to share - and so a donation has been made by the company, so: us, to a project supporting fellow humans living in less favoured areas. Concretely, "Menschen für Menschen" is a decade-long project building up infrastructure with and for people in Africa.

We all have worked hard towards our success, and our joint donation now helps others. Our little contribution to the global village. Merry Christmas and a Happy New Year!

your rasdaman team

rasdaman newsletter 12/2018

INSPIRE Taking A Fresh Look at Coverages

Antwerp, September 2018. At this year´s INSPIRE Conference a hot topic was coverages, more specifically: raster data. Experts showed how to use coverages, and solution paths for the remaining issues so that coverages can be served simultaneously by both WCS and SOS.

INSPIRE addresses many types of raster data in Annex II and III - Orthoimagery, Elevation, Geology, Atmospheric conditions and Meteorological geographical features, Oceanographic geographical features, Soil, Land cover, Natural risk zones, Energy resources, and more. The common concept for modelling raster data is adopted from OGC coverages as specified in the Coverage Implementation Schema (CIS) standard, which meantime is also adopted verbatim by ISO as 19123-2. INSPIRE, though, has adopted coverages with certain deviations which turn out to be troublemakers as implementation progresses. Therefore, a group of three renowned experts has teamed up to resolve issues and achieve harmonization with OGC CIS: Jordi Escriu (Carthography and Geographical Institute Catalonia, Spain, INSPIRE expert), Peter Baumann (Jacobs University, coverage expert), and Katharina Schleidt (DataCov Austria, SOS expert). At the INSPIRE conference the group presented their results in the workshop "Practising INSPIRE Coverages - Enhancing your Datacube Implementation Assets".

"Spatio-temporal coverages, including datacubes, comprise a powerful tool to provide INSPIRE and Coperncius data in an integrated, easy to access manner" summarizes Peter Baumann, CIS editor, and adds, "with only few adjustments we can achieve analysis-ready data and, as extra benefit, a substantial harmonization between the SWE and coverage worlds".

As it turned out only rasdaman has the flexibility for the coverage experiments conducted in the investigations, and so rasdaman naturally was chosen as the vehicle for modeling, showcasing, and discussion at the workshop. Participants could recap the examples shown no their laptops. After the workshop, attendants used the opportunity of rasdaman's fair booth to see "datacubes on steroids" in multiple scenarios and with dynamic querying.

Currently, the findings of the INSPIRE coverage expert group are being published and presented to JRC so that the necessary minor adjustments can be discussed and adopted in due course to allow stakeholders to proceed swiftly.

Contact: Dr. Peter Baumann, baumann@rasdaman.com.

Public/Private Datacube Partnership Launched

Datacubes are trending, and with the rasdaman datacube engine they can even be federated across data centers. Users benefit from location transparency and planetary-scale fusion services. Now public and private datacubes get federated - which, however, raises new technical challenges in terms of security and billing.

The BigDataCube project, supported by the German Federal Ministry for Economic Affairs and Energy, aims at advancing the innovative datacube paradigm - i.e., analysis-ready spatio-temporal raster data - from scientific into commercial data centers. To this end rasdaman, "the worldwide leading" (ESA, 2017) datacube technology, is getting installed on the public Sentinel hub CODE-DE as well as in the commercial cloud environment of cloudeo AG. A specific new challenge on which BigDataCube focuses is versatile role-based access control on datacubes, configurable down to the level of single pixels.

Under the lead of Jacobs University the team of rasdaman GmbH (datacube backend), cloudeo AG (commercial geo cloud), and DLR (weather and ocean analytics tool) collaborate on these services. On CODE-DE, the datacube service is going to complement the batch-oriented, Hadoop-based service of CODE-DE with interactive spatio-temporal viewing, fusion, and analytics services on Sentinel satellite imagery, based on the open OGC standards.

At this stage, the cloudeo datacube and the DLR analytics tool on top of rasdaman are operational, and so is the precursor version of the CODE-DE datacube service. This precursor has been established while CODE-DE is preparing the infrastructure for the datacube engine. Based on sample Sentinel 1 and 2 timeseries it offers a broad set of access, formatting, filtering, processing, and general analytics functionality through a variety of common Web clients.

Read more: BigDataCube.org

Contact: Dr. Peter Baumann, baumann@rasdaman.com

What's Your Favourite Big Data Client?

So how do you work with Big Data? Typically, tools come with specific frontends that require a new learning curve. This is not necessary, as rasdaman shows: you can remain in the comfort zone of your well-known tools.

Sometimes you will find that the emerging datacube prototypes, such as Open Data Cube, require you to become a versed python programmer. This is similar to the early Big Data wave where everyone was supposed to become a Java and parallelization geek.

At rasdaman, we work hard for harmonization and building bridges in Earth Observation and the Big Data communities in general. This holds in particular for ease of access through a variety of clients, including for example:

  • Map navigation: OpenLayers, Leaflet, ...
  • Virtual globe: NASA WorldWind, Cesium, ...
  • Web GIS: QGIS, ArcGIS, ...
  • Analysis: GDAL, R, python, ...

Admittedly, we never could implement such a wealth of clients ourselves, and fortunately we don't have to, thanks to the open OGC standards: rasdaman supports them, clients support them, deal done.

Actually, we even have joined the Open Geospatial Consortium with a mission of contributing to modularity. This has led to the highly modular Web Coverage Service (WCS) 2 standard which today is embraced by a large, steadily growing number of tools. For example, we have separated the coverage data (CIS) and service model (WCS) so that coverages today can be served through many different APIs, including WMS, WFS, WPS, and SOS.

This affords a new paradigm: whatever you have been using, continue that while unleashing the potential of superscale datacubes with rasdaman in the back end, transparently. Meaning, of course, that different users can access one and the same datacube with different tools doing different tasks; and that includes consistent updates - rasdaman is a database system, after all.

Contact: Dr. Peter Baumann, baumann@rasdaman.com

SQL Datacube Standard Adopted

November 2018. SQL remains the universal database language, satisfying billions of searches daily - although mostly for business data, bypassing the larger part of Big Data in science and engineering. Now the SQL array ("datacube") extension, MDA (Multi-Dimensional Arrays), has been adopted as International Standard. In future, such multi-dimensional data can be queried just like tables, unleashing a large part of this planet's Big Data to analytics and decision making. Plus, there is an additional, maybe unexpected, benefit.

The SQL database language has a long tradition. With its groundbreaking concept of high-level querying, without the need for programming or a need to know the actual storage details, it has revolutionized data management in the seventies. Although challenged by the !NoSQL movement a few years back SQL remains the primary data access language worldwide. This ongoing success is due to the ISO team managing the SQL standard - it embraces novel requirements and keeps enhancing SQL over time. Of course, with all due diligence and not without deep scrutiny on benefits and quality of proposals.

One such enhancement was proposed by the rasdaman team pointing out that a large part of data served worldwide - namely, multi-dimensional arrays, also known as raster data or gridded data or datacubes - remains outside SQL just because SQL is not capable of addressing them. Such data appear all over the place, like satellite and weather data in Earth science, medical imagery in Life Science, stock ticker timeseries in finances, etc. They all typically represent Big Data, and they all sit in bespoke silo systems without flexible access and analytics capabilities.

The array language proposed was rasql, the one of rasdaman which the team had designed over many years, underpinned by the highly scalable architectures of the rasdaman engine. Following its own, independent research ISO concluded that the rasdaman language indeed was the only suitable for merging into SQL. Hence, rasql became the language of choice. In the time following, Dimitar Misev wrote his PhD thesis under the supervision of Peter Baumann, Professor at Jacobs University, on this topic. His complete opus, consisting of the complete description including its mathematical semantics embedded in SQL, represents impressive 185 pages. The leading SQL experts worldwide scrutinized every detail before it finally got adopted as International Standard in Fall 2018.

"Our team is extremely happy about this success, which is heralding a new era of flexible, scalable Big Data handling" says Peter Baumann, the inventor of datacubes as documented since 1992 by publications and international patents.

But aside from making datacubes queryable there is yet another remarkable advantage MDA brings: information integration. Up to now, relational systems (as well as XML and NoSQL databases, by the way) had a say only on metadata. Traditionally, metadata are seen as small and smart ad queryable whereas "the data" are big, dumb, and not amenable to high-level selection and processing. Consequently, data and metadata have been coexisting in separate silos, with all known consistency issues, maintenance costs, and user unfriendliness. This has changed now: in one and the same SQL query both data and metadata can be queried. From now on, users can be provided with a uniform, integrated information space for all types of business, financial, healthcare, geo, and other data.

"Major players in the database market are very attentive, and I expect SQL/MDA to find its way into the worldwide market quickly. With our scalable rasdaman datacube engine, the first MDA implementation, we already perceive a massive growth in inquiries", explains Baumann who is also shaping datacube standards in the field of geo data, such as satellite imagery and weather forecasts.

Standard reference: ISO IS SQL 9075-15: MDA (Multi-Dimensional Arrays)

Contact: Dr. Peter Baumann, baumann@rasdaman.com

rasdaman newsletter 11/2018

Responding to the EU's General Data Protection Regulations

Yes, we know - we are late; (almost) everybody else has done it already. But here it comes now:

In the spirit of transparency, supporting your rights, and acting in the best interest of the community we're writing to let you know that we're updating our privacy policy in response to the EU's General Data Protection Regulations ("GDPR"). While the GDPR is a European law we are taking this opportunity to clarify and apply our privacy policy to everyone, no matter where you're located.

The terms of use of the rasdaman.org and rasdaman.com websites, the rasdaman community mailing lists, as well as this newsletter are detailed in our terms (what data we collect and why, how long we keep these, why this is accordance with law, etc.). Please note that these may get updated at times, reflecting our increasing insights.

By using the rasdaman resources you are agreeing to these. No further action is required on your side in case of agreement; if you want to unsubscribe from this newsletter use the link at the bottom of this mail, from the rasdaman community mailing lists you can unsubscribe directly.

And, as always, if you have questions or comments feel free to email us at the contact address given at the bottom of this newsletter.

rasdaman newsletter 11/2017

EarthServer: 1+ Petabyte Analysis-Ready Datacubes

November 2017 - EarthServer, the European Union Big Data initiative, has reached the next milestone: Today in the second phase extending through April 2018, the Italian data provider MEEO S.R.L. exceeds 1 Petabyte of spatio-temporal, analysis ready data processed through its operational Earth Observation Data Service.

The intercontinental EarthServer initiative unites Europe, US, and Australia in the quest for scalable datacubes on 3D x/y/t satellite image timeseries and 4D x/y/z/t weather data, based on the European datacube technology, rasdaman. After 6 years project running, a particularly spectacular milestone has been reached: operational real-time data analytics on more than a Petabyte of satellite timeseries through open OGC standard interfaces.

EO Data Service has crossed the 1 PB data providing access and exploitation capabilities to the Copernicus Sentinel-2A (launched on 23 June 2015) and Sentinel-2B (launched on 7 March 2017) satellites. The Sentinel 2 archives grow at a rate of 3TB/day and allow the Earth Observation scientists in monitoring variability in land surface conditions", says Simone Mantovani, CEO of MEEO S.R.L.

With the European partners funded by the European Commission, the EarthServer initiative was launched in 2011 to establish a standards-based, highly scalable framework for providing analysis-ready Earth science data. The underlying distinguishing principle is a geospatial datacube analytics language, the OGC Web Coverage Processing Service (WCPS), and scaling its use into the Petascale.

"It was planned like this, but still it excites me to see the Petabyte frontier finally crossed. Building such a datacube allows for new science and applications using both new and old data”, comments EarthServer coordinator, Prof. Dr. Angelo Pio Rossi from Jacobs University.

"And the next challenge is already being tackled", adds Prof. Dr. Peter Baumann, rasdaman architect and Technical Coordinator of EarthServer. "Now we are working closely with the European Centre for Medium Range Weather Forecast (ECMWF) to unleash their 220 PB archive via rasdaman and OGC datacube standards".

Datacubes for Big Earth Data

EarthServer´s 2-day workshop “Datacubes for Big Earth Data”, hosted by ESRIN from October 19 – 20 in Frascati / Italy attracted a lot of data users working with large volumes of Earth Science data. Scientists and professionals in research and industry were introduced to the Datacube concept for processing and analysing massive sets of environmental and geospatial data.

The workshop contained four major thematic blocks: Datacube initiatives, Perspectives on benefits for large-scale data providers, Future perspectives, and an interactive feedback round at the end of the workshop. The EarthServer initiative, which pushes the idea of a Datacube Federation, appeared as the most advanced collaborations at the event, both in terms of technological capabilities, live performance, and data volumes offered. Its Datacube concept eases and accelerates the handling of large geospatial data volumes and the exchange between Datacube technology providers, data users and large data organisations.

In its wrap-up, ECMWF concluded:

Efficient trimming and slicing operations on Datacubes enables on-demand and analysis-ready access to n-dimensional data along any axes. This feature is an asset in diverse fields of applications, i.e. disaster mitigation, monitoring services, Oil-, Gas- and Mining Applications, Business and Science Applications, etc.)

Speeding and easing data access is one of the most convincing arguments for using Datacubes but there are many more advantages:

  • the ability to compare different data,
  • saving local disc space,
  • more time for exploration and science,
  • better opportunities to share data,
  • providing different data typs organized in a homogeneous and standardized way,
  • user access from different domains,
  • cross-cutting activities between sciences

Data provider´s benefit is:

  • sharing the knowledge how to organize data internally, and the best practise to implement it,
  • data sharing,
  • helping non-experts to make use of complex Earth Science Data,
  • accessing and processing data and data service models dynamically and flexible,
  • Standardization of different tools.

Advantages for Technology providers:

  • processing additional data dimensions (time, elevation, etc.),
  • optimizing hardware vendor´s platforms for data processing

Event link: Datacubes for Big Earth Data Workshop

OGC Embracing Datacubes

November 2017 - Datacubes are receiving attention worldwide as a novel concept making raster data more accessible and easier to analyse. Based on the coverage standards suite of OGC, ISO and INSPIRE

Datacubes present massive spatio-temporal raster data as multi-dimensional cubes, such as x/y/t image timeseries and x/y/z/t weather data. Among others, Petascale timeseries analysis can become simple and extremely fast, as the EarthServer initiative shows.

August this year a spearhead of European and US players has come together and suggested a working group charter. Under the lead of rasdaman GmbH and Jacobs University, global players NASA, DigitalGlobe, European Union Satellite Centre, Envitia, and KEYW Corporation have established this charter which aims at establishing a cross-linked discussion forum at the widest possible level. While originally planned to publicly solicit comments ("public Request for Comments"), OGC has now put it on the agenda of the forthcoming Technical Committee meeting in Palmerston North, New Zealand beginning December. It can be expected that the public Request for Comments will be issued soon after.

The OGC Coverages working group will host the datacube session, given that the OGC coverage standards suite supports both regular and irregular n-D raster data "in a natural way", as Jordi Escriu, INSPIRE Thematic Cluster 3 Facilitator, puts it. Over the years and largely driven by the rasdaman team, OGC has adopted a rich coverage standards suite, centered around the Coverage Implementation Schema (CIS) data model and the service model of Web Coverage Service (WCS) with its geo datacube analytics language, Web Coverage Processing Service (WCPS). Today, these standards are adopted by ISO and INSPIRE as well as a large, growing number of both open-source and proprietary tools on both client and server side. Today, operational installations utilizing the OGC Reference Implementation, rasdaman, exceed a Petabyte and thus underline the maturity of WCS/WCPS based services.

"When I started asking around it was not difficult to gather a critical mass of datacube enthusiasts from several continents", says Peter Baumann, Professor of Computer Science at Jacobs University and inventor of datacube query engines since 1992. "We are more than happy to see this massive interest in a field where we are innovating since decades, and to see a criticl-mass team-up happening".

rasdaman newsletter 08/2017

Upcoming Event: INSPIRE Conference 2017

September 2017 - Europe, in its quest for a harmonized Spatial Data Infrastructure, will hold its annual INSPIRE conference in Kehl (Germany) from 04 to 05 September and in Strasbourg (France) from 06 to 08 September. Rasdaman CEO, Peter Baumann, will present tutorials on INSPIRE coverage handling and WCS during this week, as well as the official OGC and INSPIRE WCS Reference Implementation, rasdaman. Further, live demos will be given at the rasdaman booth in the exhibition area showing cloud processing and intercontinental federation of datacubes.

Workshops (in collaboration with the INSPIRE team):

  • INSPIRE WCS: From Mystery to Mastery (Monday)
  • INSPIRE Thematic Cluster #3 – Pushing implementation of European coverage data and services forward (Friday)


  • Spice Me Up: The Extended OGC Coverage Implementation Schema (Wednesday)
  • rasdaman: Big Data Cubes at Your Fingertips (Thursday)
  • 'Any query, anytime, any size': the rasdaman scalable datacube engine (Friday)


  • Big Datacubes for Agriculture: Insight on a Fingertip
  • Space-Time Datacubes - the Emerging Paradigm for Big Earth Data

Contact: Dr. Peter Baumann, baumann@rasdaman.com

ISO Array SQL Standard Under Adoption

Okayama, summer 2017 - ISO Array SQL, the SQL "datacue" extension, has been finalized and "frozen" in its design at the recent ISO meeting in Okayama, Japan. Following the planned timeline, at the next voting stage only minor textual changes will be considered any more, allowing to complete the new standard still within this year.

"Array SQL" is a common term for ISO 9075 SQL Part 15: MDA (Multi-Dimensional Arrays). This candidate standard represents an extension to the standard database language with multi-dimensional arrays. Currently in the final stage of adoption, SQL/MDA will be a game changer according to international experts in that it enables SQL to be used in environments where its gaps have prevented use so far, such as Earth, Space, and Life sciences. The SQL/MDA specification has been proposed and elaborated by the rasdaman team, following internal scrutiny by ISO. Early on, an industry member of the SQL group - investigating on rasdaman and competing approaches like SciDB and SciQL - concluded that rasdaman was the superior data model and believed that "rasdaman is the only one of the three data models to come to market". Through rasdaman, the concepts are practice-proven with 1000-fold parallelization and on Petascale operational installations.

Following intensive work for achieving a seamless integration with the SQL standard, SQL/MDA will next be voted on by the national delegations as a "Draft International Standard" starting this August, at which stage only editorial changes will be possible any more. Therefore, concepts and details are stable now. According to plan, final adoption is planned for the end of 2017.

A publicly accessible summary of the standard is available, e.g., in: D. Misev, P. Baumann: Enhancing Science Support in SQL. Proc. Workshop Data and Computational Science Technologies for Earth Science Research (co-located with IEEE Big Data), Santa Clara, US, October 29, 2015.

Contact: Dr. Peter Baumann, baumann@rasdaman.com

Agro-Geoinformatics with rasdaman

August 2017 - Increasingly, the advantage of combined spatial and temporal analysis is being recognized, and very much so currently in agroinformatics. Rasdaman is enabling novel agroinformatics services on many levels, and international conferences seek talks on rasdaman and its open-standard approach.

EOFarm is a hitech startup in Greece with a mission in supporting farmers with Big Data Analytics. The open-source rasdaman community platform has enabled a quick deployment of the service while ensuring its scalability and openness. Products offered include Color Composites, Band Ratios and Indices, Vegetation Detection, Canopy Greenness Estimation, Land Surface Temperature, Time series over arbitrary areas of interest, etc. With Landsat8, Sentinels, and !RapidEye the underlying data sets integrate both open and commercial satellite imagery. Due to the success of this service, meanwhile a similar framework has been deployed for water quality monitoring.

In Germany, Spatial Business Integration GmbH and rasdaman GmbH team up in the BigPicture project which focuses on "diagnosis in the field": Big-Data-based determination of causes for satellite image derived and site-specific variations so that targeted measures can be recommended to farmers, such as fertilizer placement, application of plant protection products, choice of species to grow, etc. To achieve high reliability, 500 farmers are delivering ins-situ insight for ground truthing. Further, meteorological data are mixed in. Again, rasdaman is the enabling Big Datacube engine underneath. The project is being supported by the German Federal Ministry of Food and Agriculture.

Meantime, the expertise of rasdaman CEO Peter Baumann in Big Datacubes is being sought continuously. In Kampala / Uganda this August he was invited keynote speaker at the Advances in Geomatics Research Conference where a central topic was how to best use Geo IT for an effective, yet environmentally compatible agriculture. Right after, he presented at the annual Agro-Geoinformatics Symposium at George Mason University in Fairfax, USA. At the INSPIRE conference in Kehl he will give a tutorial on INSPIRE WCS: from Mystery to Mastery, followed by talks at ESA EO Open Science Conference iN Frascati, Italy, and at INTERGEO Berlin, among others.

Contact: Dr. Peter Baumann, baumann@rasdaman.com

Rasdaman CEO elected Chair of IEEE Earth Science Informatics Group

Ft Worth, USA, August 2017 - IEEE being a leading organization for IT professionals is continously seeking expertise which ideally combines research and industry. This August, rasdaman CEO Peter Baumann has been elected chair of the Earth Science Informatics Technical Commitee in the IEEE Geoscience and Remote Sensing Society.

The mission of the Earth Science Informatics Technical Committee (ESI TC) is to advance the application of informatics to the geosciences and remote sensing, to provide a venue for ESI professionals to exchange information and knowledge, and to give technology advice to major national and international ESI initiatives.

As such, it unites over 180 experts from all over the globe. The chair, Peter Baumann, is accompanied by two co-chairs: Prof. Peng Yue from Wuhan University in China and Dr Rahul Ramachandran from NASA, USA.

Election of Peter Baumann as the TC chair goes in line with a special appreciation by IEEE in 2016 when he received the honour of becoming IEEE Senior Member and, more recently, IEEE Distinguished Lecturer.

rasdaman newsletter 03/2017

Webinar: Datacubes as an Enabling "Big Data" Paradigm

March, 06, 2017 - Research Data Alliance (RDA) kindly invites you to take part in the Datacubes as an Enabling "Big Data" Paradigm webinar, presented on March 6 at 14:00 UTC by Peter Baumann, co-chair of the RDA Big Data Interest Group, Geospatial Interest Group, and Array Database Assessment Working Group.

In this webinar Peter Baumann will present the concept of datacubes and services around them, based on practical largescale science applications and several "Big Datacube" standards initiated and edited by the presenter, so as to proliferate knowledge and stimulate discussion. The webinar is open to all data experts, data repository managers, data practitioners and researchers, policy makers and beyond.

While it is free of charge but registration is required. To find out more visit https://www.rd-alliance.org/datacubes-enabling-big-data-paradigm-webinar

If you have any questions, please feel free to contact the webinar organisers at webinars@europe.rd-alliance.org.

rasdaman newsletter 01/2017


December 2016 - Last month INSPIRE officially adopted the OGC Web Coverage Service (WCS) as INSPIRE Coverage Download Service. Jordi Escriu, Facilitator of the corresponding Thematic Cluster 3, calls "WCS the natural way to deliver and exploit coverage data".

The OGC Web Coverage Service (WCS) standards suite defines services on coverages, a concept including spatio-temporal sensor, images (timeseries), simulation, and statistics data. In practice, coverages often represent "Big Data" as orthoimagery, elevation data, weather forecasts, and point clouds. The WCS Core establishes basic functionality all implementations must provide: access to whole coverages or parts thereof via spatial/temporal subsetting. Extensions add optional functionality, such as data format encodings (like GeoTIFF, JPEG2000, NetCDF), service model extensions (such as scaling and reprojection), and protocol bindings (GET, POST, SOAP, and in future REST).

Representatives of industry, administration, and research are positive about the standard model WCS and its reference implementation rasdaman: "The RASDAMAN product is currently the world leading environment in this domain and the standard working horse for OGC standardisation on these innovative data access interfaces", Guenter Landgraf from European Space Agency (ESA) said.

Already during the workshop “Implementation and potential of INSPIRE coverage data and WCS” at the INSPIRE Conference in Barcelona last Fall attendees were impressed by the options and possibilities WCS is offering. In his presentation INSPIRE delegate, Peter Baumann, and Alex Dumitru from Jacobs University showed the ease of using WCS in a hands-on-session doing live ingest and retrieval using rasdaman. Presenter Julia Wagemann from the European Centre for Medium-Range Weather Forecasts (ECMWF) underlined: “Experience from ECMWF on coverages and WCS is a good example of the benefits and the potential of such kind of data and services. Worth to look at and take into account by other organizations and communities." Editor of the INSPIRE WCS specification is James Passmore from British Geological Suervey (BGS) who has gained ample experience in coverage services as participant of the EarthServer project. Peter Baumann, head of rasdaman, was involved as national German delegate to INSPIRE; based on a long-standing experience in building flexible, heavy-lifting services he is editor of "Big Datacube" standards in OGC and ISO.

Read more: JRC EC Europe: Results from the Workshop on Implementation and Potential of INSPIRE Coverage Data and WCS

rasdaman joining EARSC

January 2017 - Starting 2017, rasdaman is full member of the European Association of Remote Sensing Companies (EARSC), a not-for-profit association promoting Earth Observation (EO) technology especially from European companies. Monica Miguel-Lago expressed that EARSC "are delighted to accept your company as new EARSC full Member".

With these activities EARCS fosters development of the European EO geo-information services industry. EARSC is open to full and observer members with an interest in using or providing services delivering information based on observations of the Earth and its environment from any platform; satellite, aircraft or UAV.

"We see high value in joining EARSC and the hitech portfolio aggregated there, it's like meeting with friends", says Peter Baumann, rasdaman CEO. As a key technology provider in Big Earth Data services, rasdaman enables researchers to browse, access, and analyze massive multi-dimensional data sets from a wide range of source. rasdaman is deeply involved in several Earth observation projects, such as the EarthServer initiative, where monitoring the Earth and analyzing EO timeseries data are at the heart. The rasdaman technology allows real-time analytics on 3D and 4D Petabyte cubes for spatio-temporal sensor, image, simulation, and statistics data as well as their intercontinental fusion. This power of data handling can be wrapped into any kind of clients, ranging from direct visual interaction through maps and virtual globes (like OpenLayers and NASA/ESA WebWorldWind) over Web GIS (like QGIS) to high-end analytisc (such as python).

EarthServer: Experts Attest Significant Progress

Brussels, January 2017 - "The EarthServer-2 project has successfully completed its second milestone and has made significant progress towards the eventual aim of producing a comprehensive big data analytics platform for Earth Science", the European Commission currently attested on the occasion of project´s midterm review performed in Brussels.

Among the main innovation activities, rasdaman, the platform technology of all services, has been extended with new functionalities, such as support for GRIB data and NetCDF ingestion. A new metadata engine, FEMME, has been developed and, over the next period, will be fully integrated with xWCPS, which leads to combined array data and metadata retrieval. Search facilities on both XML-encoded metadata, freetext, and OGC coverages will be an important step as well. Work on mixed queries enabling new user interfaces has been done already, and will be contined. Services for marine, climate, earth observation, and planetary data have been created using the NASA/ESA WebWorldWind visualisation API. Live demonstrations of pilots during the review meeting showed that significant progress has been made in producing high-quality interfaces, as well as in ingesting valuable data resources and integrating new providers, such as ANI.

One of the important results achieved is a dramatic reduction of the amount of data downloaded. While the extraction of data with traditional methods requires users to download all data for the time series, up to Terabytes, with rasdaman users can access data with surgical accuracy, thereby reducing data transfer to Kilobytes. Additionally, access is simplified substantially; for example, users only need to click on the globe for retrieving a timeseries values for this point.

Further, the project has achieved important impact on international big data standardization process. For example, INSPIRE has chosen rasdaman as reference implementation for the INSPIRE WCS Download Service. The technical capability gained during the project, enabled the participating SMEs to substantially strengthen their market footprint.

In the next period, the Petabyte frontier will be attacked, further enhancements will be implemented on service as well as technology level, and "Big Datacube" standardization will be progressed further.

Contact: Dr. Peter Baumann

OGC Confirms rasdaman Compliance with Datacube Extension Standards

Mayland/USA, January 2017 - OGC officially confirms that rasdaman has achieved compliance certification on a number of important extensions to the OGC "Big Datacube" standard, Web Coverage Service (WCS).

OGC operates a compliance program which allows implementations of standards to be tested free of cost test through a neutral, verified, suite maintained by OGC (see www.opengeospatial compliance). The purpose is to increase system interoperability while reducing technology risks. Buyers gain confidence that a compliant product will work with another compliant product based on the same OGC standard, regardless of which company developed the product. The rasdaman array analytics engine has passed the WCS test suite for Core and GET/KVP extension already in 2013 and today is official reference implementation in OGC and also INSPIRE. While rasdaman always has been a first implementer of the OGC Big Data standards - and currently offers the most comprehensive support for the WCS suite - now there is official confirmation for the extensions WCS Post, WCS Processing, WCS Scaling, WCS Interpolation, WCS Range Subsetting. By combining the complementary know-how of both partners Big Picture aims at establishing large-scale automated analysis offering individual insights and support for agricultural production in Germany.

These compliance tests have been developed by Fuzhou University in China. "We are very happy about Fuzhou University undertaking the work of establishing such a test suite", says Peter Baumann, WCS suite editor and rasdaman Principal Architect. "This valuable service to the community allows tool providers like us to get an independent compliance confirmation, which of course is worth more than any self assessment".

Work on demonstrating compliance is ongoing: testing rasdaman against the WCS CRS extension is already in an advanced state, and the WCS-T extension is planned to get addressed next.

rasdaman newsletter 10/2016

rasdaman 9.3 Released

Open-source rasdaman v9.3.0 is now available for download, see rasdaman.org. Important feature additions include:

  • Enhanced NetCDF support
  • GRIB format support
  • Advanced JSON-style format-parameters support in encode/decode
  • WCSTImport supports a general coverage recipe that can be used with complex formats like GRIB or NetCDF
  • WCPS uses a brand new and improved parser and implementation by default
  • CRS reprojection now available in both WCS and WCPS
  • automated download, compilation, and deployment through a brand new installer script

Deployment can be done through one of these options:

  • From source - just run the installer script
  • Packages - RPM (CentOS 7) and DEB (Ubuntu Trusty / Debian 8) are available; more distros will come soon
  • Virtual machines - readily configured with a running rasdaman instance

Full details can be found at rasdaman version 9.3 .

Ready for INSPIRE - Reference Implementation rasdaman in Action at Intergeo

Oktober 2016. This year the annual INTERGEO fair was held at Hamburg/Germany from Oktober 11 through 13. The rasdaman company was represented witha fair booth, a congress keynote talk on INSPIRE WCS and a workshop demonstrating solutions for dealing with it. Highlights included realtime intercontinental weather/satellite image fusion and ultrafast on-demand cloud processing.

Jacobs University demonstrated in a one-hour-session the radically simplified handling of spatio-temporal sensory, images, weather data, and more based on Web requests following the open OGC standards WCS, WCS-T, and WCPS. Critically important, all components of the WCS suite are seamlessly harmonized.

In the first part of the presentation, implementation and potential of INSPIRE coverage data and WCS was outlined: Member State institutions have to make their coverage data conformant to the INSPIRE directive before 2020, and coverages occur across many, if not most of the Annex II and III themes. According to INSPIRE experts the list is long indeed: Orthoimagery, Elevation, Geology, Atmospheric conditions and Meteorological geographical features, Oceanographic geographical features, Soil, Land cover, Natural risk zones, Energy resources, and more.

Concretely, in the scope of INSPIRE raster data shall be implemented and made available as coverages according the concepts stemming from OGC coverages, ISO 19123 and other related standards currently in force. To this end, INSPIRE is establishing a Web Coverage Service (WCS) based on the OGC WCS standard adopted in 2012. This INSPIRE WCS has been published in a draft version, and comments from the member states have been collected.

In the second part of the presentation a live demo showed how straightforward coverage services can be. In the talk and at the rasdaman booth visitors experienced live ingestion of NetCDF files and their retrieval as TIFF files. It took just a simple change in one configuration file to switch from mosaicking to timeseries generation in the database. The open-ended capabilities of the OGC geo analytics language, Web Coverage ProcessingService (WCPS), was impressively shown. In the Amazon cloud, a Terabyte of data was analyzed in about 100 milliseconds. Heavy rainfall risk areaswere determined in realtime through automatically orchestrated queries combining ECMWF in the UK and NCI in Australia.

Contact: Dr. Peter Baumann, rasdaman GmbH.

INSPIRE Conference 2016

September 2016. Europe, on its quest for a harmonized Spatial Data Infrastructure, has held its annual INSPIRE conference in Barcelona from 26 to 30 September. Highlight was the presentation of INSPIRE WCS, crafted under the lead of British Geological Survey. INSPIRE Editor Jordi Escriu concluded: "If you have raster data you should use WCS" after experiencing live the ease of ingest and download through INSPIRE WCS Reference Implementation, rasdaman.

On Monday 26, "Choosing the Right INSPIRE Download Service Technologies" initiated discussion. Next, large-scale practice proofs of WCS were presented in the session "Improving access to big data through OGC standard interfaces - a hands on WC(P)S session", based on 150+ Terabyte services and intercontinental datacube fusion. Over the next days, a lot of conference attendees took the opportunity to visit the INSPIRE booth and, next toit, the EarthServer booth.

On Thursday 29, "Integrating INSPIRE into a single seamless information space" highlighted the integration power of the WCS suite. Finally, on Friday 30th "Implementation and potential of INSPIRE coverage data and WCS" summarized and deepened the topic. Live demos based on rasdaman which hasbeen chosen as INSPIRE WCS Reference Implementation.

WCS has proven itself through several years as an OGC standard for regular and irregular grids, such as imagery, elevation data, climate data, andmany more spatio-temporal data. The modular suite of standards enables data access, extraction, download - up to versatile analytics with OGC WebCoverage Processing Service (WCPS). Based on the unifying concept of coverages, data homogenization and service integration is achieved. The OGC coverage data and service model currently is on the way of becoming the lingua franca in Big Earth Data being adopted by INSPIRE as well as ISO. Both the leading open-source and proprietary tools support WCS, with rasdaman being the only fully comprehensive implementation of the WCS suite, in particular with its award-winning multi-dimensional parallel, distributed array engine.

Big Data Help Harvesting More

The German Ministry of Food and Agriculture is funding project "BigPicture - Diagnosis in the field: Big-Data-based determination of causes for satellite-image-derived and site-specific variations" carried out jointly by rasdaman GmbH, Bremen, and Spatial Business Integration GmbH, Darmstadt. Goal is helping farmers to optimize returns and to protect the environment through innovative Big Data technology.

A ceremony marked the official kickoff and underlined importance of this project. "For German agriculture it is important to play in the worldwide ivy league when it comes to hitech services. Projects like BigPicture play an important role due to their outstanding innovation and practical relevance", says Parliament State Secretary of the Federal Ministry for Food and Agriculture, Peter Bleser, while handing out the funding certificate on October 10.

With this handover of the certificates to rasdaman GmbH and Spatial Business Integration (SBI) the Federal Ministry of Food and Agriculture officially announced its financial support of the BigPicture research project through its innovation grants program. Overall volume of this 2.5 years project is above one million Euro. Technically, BigPicture aims at interpreting satellite captured variations in farm fields through Big Data technologies so that targeted measures, such as fertilizer placement, application of pesticides, or choice of species to grow, can be derived. Manifold information will be combined in such analyses, including satellite imagery, weather data, as well as the farmers' experience. This way, the process chain from satellite-based symptoms capturing in the field overdiagnosis up to therapy recommendation will be automated completely.

By combining the complementary know-how of both partners Big Picture aimsat establishing large-scale automated analysis offering individual insights and support for agricultural production in Germany.

Bremen-based hitech company, rasdaman GmbH, is world leader with its multi-award-winning agile Big Data technology for massive spatio-temporal datacubes. Just this summer US magazine CIOReview has chosen high-speed server rasdaman into their list of top Big Data solutions worldwide. Scaling seamlessly from laptops over clouds to intercontinental federations, rasdaman is in operation at leading data centers. The European Space Agency (ESA) operates a rasdaman database with 145 Terabyte, and the European Centre for Medium-Range Weather Forecast (ECMWF) wants to unleash its world's largest climate data archive encompassing 90 Petabyte. International "Big Datacube" standards used rasdaman as a blueprint, such as ISO Array SQL and the Open Geospatial Consortium (OGC) "CBig Geo Data" suite, WCS.

Spatial Business Integration GmbH (SBI) focuses on providing services on a worldwide scale based on satellite and geospatial data for its customers in the agricultural sector, as well as the agro-chemical and finance industries. "We care about the world, which is why we provide information which can protect food, water and nature", stated General Manager Dr. Born recently at a conference attended by delegates from the agricultural, agro-chemical and space industries. Providing intelligent information products derived from geospatial data assists SIB's clients inrealizing their business strategies, ensuring efficient business processes, saving time and money, and managing limited resources such as arable land, nutrients and water. Spatial Business Integration's success is anchored in more than 20 years of experience and expertise in the fields of satellite remote sensing, geospatial technologies, plant sciences, water and agriculture.

The BigPicture project is supported by funds of the Federal Ministry of Food and Agriculture (BMEL) based on a decision of the Parliament of the Federal Republic of Germany via the Federal Office for Agriculture and Food (BLE) under the innovation support programme.


rasdaman newsletter 08/2016

US CIOReview: rasdaman Among Most Promising Big Data Solutions

Published from Fremont, California, CIOReview is a print magazine that explores and understands the plethora of ways adopted by firms to execute the smooth functioning of their businesses. A distinguished panel comprising of CEOs, CIOs, IT VPs including CIOReview editorial board finalized the "100 Most Promising BigData Solution Providers 2016" in the U.S. and shortlisted the best vendors and consultants.

In their July 20 press release where Rasdaman has been ranked one of the 100 Most Promising Big Data Solution Providers by US magazine CIOReview.

"It's a delightful experience to announce Rasdaman as one of the 100 Most Promising BigData Solution Providers 2016," said Jeevan George, Managing Editor of CIOReview. "Rasdaman is recognized for providing highly innovative products that helps to store and query massive, multi-dimensional array of data into the data base, which brings new dimension to the industries in the Big Data arena."

More info: CIOReview rasdaman assessment, CIOReview digital magazine

INSPIRE Conference: Coverages Trending

Europe, on its quest for a harmonized Spatial Data Infrastructure, is about to congregate for the annual INSPIRE conference, this time in vibrant Barcelona in mid-September. Highlight will be the presentation of the INSPIRE WCS, crafted over many months under the lead of British Geological Survey.

On Monday September 26, "Choosing the Right INSPIRE Download Service Technologies" will initiate discussion. Next, large-scale practice proofs of WCS by PML, ESA, ECMWF, and others will be presented in the session "Improving access to big data through OGC standard interfaces - a hands on WC(P)S session", based on 150+ Terabyte services and intercontinental datacube fusion. Over the next days, you may want to visit the INSPIRE booth and, next to it, the EarthServer booth. On Thursday 29, "Integrating INSPIRE into a single seamless information space" will highlight the integration power of the WCS suite. Finally, on Friday 30th "Implementation and potential of INSPIRE coverage data and WCS" will summarize and deepen the topic. Live demos will be based on rasdaman which has been chosen as INSPIRE WCS Reference Implementation.

WCS has proven itself through several years as an OGC standard for regular and irregular grids, such as imagery, elevation data, climate data, andmany more spatio-temporal data. The modular suite of standards enables data access, extraction, download - up to versatile analytics with OGC WebCoverage Processing Service (WCPS). Based on the unifying concept of coverages, data homogenization and service integration is achieved. The OGC coverage data and service model currently is on the way of becoming the lingua franca in Big Earth Data being adopted by INSPIRE as well as ISO. Both the leading open-source and proprietary tools support WCS, with rasdaman being the only fully comprehensive implementation of the WCS suite, in particular with its award-winning multi-dimensional parallel, distributed array engine.

rasdaman newsletter 07/2016

Open Source Parallel rasdaman at VLDB 2016

At the VLDB BOSS workshop this fall it is planned to release rasdaman community BOOST!, the open-source engine that allows parallelizing single queries across clouds, clusters, and data centers.

Help to get this accepted, vote for rasdaman on the workshop page:

... and the timing is really tight: only 3 days left! Thanks for your click to support rasdaman! Please feel encouraged to distribute immediately over all your channels.

rasdaman newsletter 04/2016

EGU 2016 Splinter meeting: EarthServer - the Global Datacube Federation

Meet EarthServer at the splinter meeting "EarthServer: the Global Datacube Federation" at EGU 2016, Tue, 19 Apr, 12:15–15:00 / Room 2.83.

One of today's most influential initiatives in Big Geo Data is EarthServer which is paving the way for flexible, scalable datacube services based on innovative NewSQL technology. Researchers from Europe, the US and recently Australia have teamed up to rigorously materialize the datacube paradigm for Earth Observation, ocean, meteorological, and planetary science.

The platform technology, rasdaman, enables ad-hoc datacube analytics and fusion. Integrated metadata / datacube retrieval is under preparation. Relying on the OGC Big Geo Data standards, WCS and WCPS, "any question, anytime, on any size" becomes possible from open-source tools like OpenLayers, ncWMS, QGIS and proprietary tools like ESRI ArcGIS. NASA WorldWind adds high-end visualization. EarthServer databases have transceded 100 TB and are currently heading towards the Petabyte frontier. Single queries have been split automatically across more than 1,000 cloud nodes.

In this meeting, EarthServer will be presented live: its vision, technology, existing services, and its impact on Big Datacube standards. Questions will be answered like "why is it advantageous for our data center join the federation?", "how can we do it, and how much effort is needed?".

PS: Snacks and refreshments will be provided free of cost.

your rasdaman team

See rasdaman in Action - live at EGU 2016!

The EGU General Assembly 2016 provides an essential opportunity to find out more about the technical innovations and web services developed in the EarthServer project, which uses rasdaman as its platform core. Learn more about spatio-temporal analysis on Big Earth Datacubes in different geo domains. The EarthServer initiative establishes a series of novel high performance datacube services. At the EGU conference the partners will demonstrate these innovations through a series of live demos, poster sessions, and presentations as follows:

  • Monday, 18 April 18
    • rasdaman: poster EGU2016-17171 | Big Geo Data Services: From More Bytes to More Barrels | Baumann | 17:30-19:00 | Hall A
    • Jacobs University: SC43 Standards based Services for Big Spatio-Temporal Data | Baumann | 17:30–20:00 | Room 2.44
    • Tuesday, 19 April
    • EarthServer Splinter meeting of EGU2016 | SMP13 | EarthServer: the Global Datacube Federation I 12:15–15:00 / Room 2.83
    • Jacobs University/rasdaman: poster EGU2016-16985 | Science SQL as a Building Block for Flexible, Standards-based Data Infrastructures | Baumann| 17:30-19:00 | Hall A
    • Jacobs University: PICO EGU2016-3996 | Planetserver | Remote Sensing Data Analytics for Planetary Science with PlanetServer/EarthServer Pio Rossi, Baumann | 17:30–19:00 | Hall X4
    • ECMWF: Poster EGU2016-16899 | EarthServer: Opportunities and Challenges of serving ECMWF´s peta-sized Archive through OGC Web Services | Wagemann/Siemen/Lamy-Thepaut | 17:30 - 19:00 | Hall A
    • Wednesday, 20 April
    • Jacobs University: poster EGU2016-16753 | Datacubes as a Service Paradigm | Baumann | 15:30-17:00 | Hall A
    • Friday. 21 April
    • Jacobs University: oral EGU2016-17298 | The EarthServer Federation: State, Role, and Contribution to GEOSS | Merticariu, Baumann | 11:45- 12:00 | room 2.61

rasdaman newsletter 03/2016

rasdaman 9.2 with more performance and functionality

We are proud to announce availability of rasdaman 9.2.1, which represents yet another important increase in performance, functionality, and OGC standards support. The team has worked hard, committing 216 patches in total since v9.1, see the full details.

Major changes:

  • Scalability:
  • Improved storage support, up to exascale datasets on all filesystems
  • Improved performance and reliability for the rasnet protocol; rasnet is now the default communication protocol.
  • Automated balancing of worker processes
  • Ingestion:
  • WCS-T import user interface improvements: WCS extract recipe, ability to resume and retry imports, improved filtering of input files, improved status messages
  • WCS-T import stable recipe API allowing developers to create their own custom recipes for their specific usecases
  • Various WCS-T enhancements
  • Improved transactional handling in the SQLite/Filestorage backend
  • OGC Web Services:
  • Embedded server for OGC Web Services platform - now it can also run as a standalone executable without the need of an external server container like Tomcat
  • Support for PNG as an export format in WCS
  • User Interface:
  • Revamped interface for the WCS Client + various enhancements
  • Miscellaneous:
  • runtime configuration of logging
  • Greatly simplified installation procedure with rasdaman installer, RPM and DEB packages
  • Over 200 fixes for issues reported

There was a lot of work on improving the installation experience of rasdaman, and v9.2.1 now offers a range of download options:

  • From source: we have a brand rasdaman installer script that greatly automates installation.
  • Packages: RPM (CentOS 7) and DEB (Ubuntu Trusty) are available; more distros will be supported soon. If you are updating RPMs, note that the repository has changed; further details on the RPM installation page.
  • Virtual machines: a preconfectioned virtual machine, ready for use.

We hope you will enjoy the improvements and new features this version brings; as always, all feedback is greatly appreciated!

your rasdaman team

rasdaman Donates to "Menschen für Menschen" ("Humans for Humans")

Dezember 2015. Our company has been quite successful in 2015 - innovation contributions in the rasdaman product as well as in international research projects, as well as substantial license sales.

Time to share - and so a donation has been made by the company (that means: us) to a project supporting fellow humans living in less favoured areas. Concretely, "Menschen für Menschen" is a multi-decade project building up infrastructure with and for people in Africa.

We all have worked hard towards our success, and our joint donation now helps others. Our little contribution to the global village. Merry Christmas and a Happy New Year!

your rasdaman team

rasdaman newsletter 01/2016

How Fast is rasdaman?

January 2016. As the rasdaman array engine is known worldwide, more and more people ask about a comparison with related technology. Now scientific tests have proven: rasdaman is the fastest Big Array Data technology by far.

The setup was immense: at the 2015 Annual Conference of the American Geosciences Union, more than 20,000 visitors were present. Researchers from George Mason University and University of Wisconsin / Madison presented a performance comparison of rasdaman with Apache Hive and Apache Spark, to renowned Big Data technologies. Test case was evaluation of MODIS satellite imagery. The outcome reported showed that rasdaman outperforms both competitors by far.

"Our team is very happy about these findings" says rasdaman Principal Architect, Peter Baumann. "We still have many ideas for speeding up rasdaman even further, in particular on massive datacubes, so we are excited to excite again in future".

Simultaneously, a general Array Database benchmark was presented at an expert workshop on database benchmarking held in India this December. Here rasdaman was compared against SciDB and SciQL, to other Array Databases. The various tests, which are available as open source code on github for reproduction by the community, revealed that rasdaman often is faster by several order of magnitudes. However, some of the tests could not be scaled to the intended "Big Data" situations as SciQL and SciDB could not ingest large datacubes. Notably, known rasdaman databases exceed 130 TB.

Elephant Meeting on Big Data Services

November 2015. While we try to get to grips with the ever-increasing “Big Data” deluge we recognize that adequate Web services are a key prerequisite for ubiquitous, flexible, and fast data access. In a massive concertation effort several large European initiatives have teamed up now to address the service challenge. From 12 through 13 November 2015 an inaugural EUDAT Workshop on services for Big Data held successfully at the Supercomputing Center in Barcelona, Spain. Representatives of three decisive Big Data projects - EUDAT, EarthServer, and EPOS - have come together to discuss innovative alternatives for value-adding services.

To consolidate activities around these specific themes the workshop was divided in several tracks focusing on the topics of Big Data semantics, federated Data Mining, and multi-dimensional Array Databases for large time series. Discussions started by capturing best practices and discussing the current state of development and activities in the respective areas. Questions like: How can data processing be orchestrated optimally or how can scientific workflows make use of EUDAT services were discussed intensively in different working groups.

Peter Wittenburg, scientific coordinator of the EUDAT Data Infrastructure, convened a critical variety of expertise from Europe and the US. Especially the topic of multidimensional arrays was focused by the experts because of playing a major role in scientific and engineering data. In a summary Mark van de Sanden, EUDAT Workpackage Leader, and Peter Baumann, workshop facilitator of the EUDAT Array Database track, pointed out possible roles of EUDAT in the future:

  • IaaS provider: providing a cloud infrastructure to run Array Databases
  • SaaS provider: providing an Array Database as an domain-independent, horizontal service
  • Providing tools for easy data movement between EUDAT DCI domain and User domain
  • Providing domain services (e.g., geo, astro, life sciences) based on a common horizontal platform of array services, thereby leveraging cross-community effects

Peter Baumann resumed his experiences of running large-scale infrastructures in his presentation: "Of course multidimensional arrays do not stand alone, they are intertwined with other data types, but typically they constitute the "Big Data" part. Therefore, it makes sense to integrate arrays into common data management platforms." The flexibility of querying data, achieving data independency, scalability and standards conformance are critical advantages of Array Database technologies. Among the challenges spotted were integration of heterogeneous data types, including arrays, into a single common information space for users. Array intensive domains like the Earth-, Space-, and Life Sciences were considered as possible candidates of future EUDAT services.

OGC Requests Comments on Coverage Implementation Schema

December 2015. The Open Geospatial Consortium (OGC) requests comments on the OGC Coverage Implementation Schema (abbreviated as "CIS"). This candidate OGC Standard complements the OGC Abstract Topic 6 Standard (which is identical to ISO 19123) with a concrete coverage model whose implementations are conformance testable and, hence, are interoperable. ISO has expressed interest to adopt CIS 1.1 as ISO 19123-2.

The Coverage Implementation Schema version 1.1 is the backwards compatible successor of the "GML 3.2.1 Application Schema - Coverages" version 1.0 (abbreviated as "GMLCOV"). The CIS candidate standard has been renamed to reflect the breadth of available data encodings beyond GML and to reduce market confusion. While incorporating all features of GMLCOV 1.0, CIS 1.1 adds important new modeling capabilities, such as coverage partitioning, interpolation, geometry/value pairs, SensorML support, and point clouds. Further, a general model for all combinations of non-regular grids is supported, which includes GML 3.3 coverage types as special cases.

The OGC Abstract Topic 6 Standard defines an abstract coverage model. Coverages represent homogeneous collections of values located in space/time, such as spatio-temporal sensor, image, simulation, and statistics data. Common examples include 1-D timeseries, 2-D imagery, 3-D image time series or spatial models, as well as 4-D climate and ocean data. Generally, coverages encompass multi-dimensional regular and irregular grids, point clouds, and meshes.

The OGC Coverage Implementation Schema complements OGC Abstract Topic 6 with a concrete coverage structure definition of a subset of coverage models in OGC Abstract Topic 6 that can be conformance tested, regardless of the implementations’ data format encoding. CIS enables a wide range of interoperable implementations.

Coverages can be encoded in suitable formats (such as GML, JSON, GeoTIFF or CF-netCDF) and can be partitioned, e.g., for a time-interleaved representation. Coverages are independent of service definitions and therefore can be accessed through a variety of OGC services types, such as the Web Coverage Service (WCS) Standard. The coverage structure can serve a wide range of coverage application domains, thereby contributing to harmonization and interoperability between and across these domains.

Download the candidate OGC Coverage Implementation Schema and provide feedback by 16 January 2016.

rasdaman newsletter 11/2015

Simplified Big Data Handling: New OGC Standard

2015-oct-30: Voting has closed on the OGC WCS-T specification, resulting in the effective adoption of the standard as part of the OGC WCS suite. WCS-T enables Web-based insertion, deletion, and modification of spatio-temporal coverages offered through a WCS-based service.

The OGC Web Coverage Service (WCS) Interface Standard is an open standardwidely implemented in geospatial applications around the world, both open-source and proprietary. The standard's main function up until now has been to support retrieval of geospatial "coverages" data such as spatio-temporal sensor, image, timeseries, simulation, and statistics data. With the new Transaction extension, the standard will support not only retrieval but also transactions such as insert, update, and delete of a coverage on the server.

"The high-performance datacube services running rasdaman showed us the need for enhancing operator support for the maintainers", explains Peter Baumann, Professor of Computer Science at Jacobs University and editor of the standard. "Incremental creation and maintenance of 2-D, 3-D, and 4-D datasets, possibly with irregular grids along space or time, can be challenging and error-prone. With WCS-T, a simple Web request performs all actions necessary, and ensures consistency of the server offering at all times." Simultaneously to writing the specification, WCS-T has been implemented in rasdaman. He explains, "With rasdaman, we have the ideal test vehicle for evaluating new service concepts." Naturally, therefore rasdaman becomes reference implementation lateron. In the intercontinental EarthServer initiative [0], WCS-T is already regularly used to establish datacubesgrowing beyond one Petabyte each.

But there is more to it than just convenience. Effectively, WCS-T closes a gap when it comes to Earth data mash-ups and federations: Coverage subsets can be extracted from one server using any suitable service interface, such as WFS, WCS, WMS, WPS, or SOS and can be fed into another service node via WCS-T.

The team around Baumann already has extensive experience in crafting widely accepted "Big Data" standards. They have developed the whole WCS 2.0 suite which today is supported by MapServer, GeoServer, ESRI, and many others. Recently, OGC WCS has entered the adoption process of both ISO and INSPIRE. Further, the group has initiated ISO work on domain-independent SQL support for multi-dimensional arrays, the forthcoming SQL/MDA standard. WCS data offerings today exceed hundreds of Terabytes, and Array SQL queries have been parallelized over more than 1,000 cloud nodes.

A draft version of the WCS-T standard is available at the OGC Coverages info hub [1], the authoritative standard document will be made available soon on the official OGC WCS download page [2].

[0] earthserver.eu

[1] opengeospatial.org/CoveragesDWG

[2] www.opengeospatial.org/standards/wcs

EarthServer TV Documentary is in 2nd place of the International Science Film Festival

2015-oct-23: The 1h TV documentation Big Earth Data - the digitized planet produced by Jacobs University Bremen and rasdaman GmbH as part of the EarthServer project in cooperation with the german broadcast ZDF/ARTE is in 2nd place of the "International Scientific Film Festival in Szolnok, Hungary. The documentary which has been funded by the European Commission and broadcasted in February 2015 is concerned with the chances and challenges dealing with Big Earth Data.

The contest received 146 films from 32 countries for consideration this year. National pre-juries of local science education and media experts in all participating festival countries have concluded their selection process and the official selection of the Science Film Festival is comprised of 52 films from 17 countries in 2015. Last year the Science Film Festivalreached 600 000 visitors in 13 countries. It is the largest event of itskind worldwide in terms of audience reach. The contest provides forum for such distinct films which can be connected to science, includes scientific results or activities.

It was the 14th year that the International Scientific Film Festival witha wide spectrum of topics has started its way. Since 2001, every other year a film forum emerged on which the creators of film and science had the opportunity to display their work that had a scientific theme. During the last decade scientific educational films, documentaries and portraits about science, short and long features, experimentals with a scientific topic, documentaries dealing with protection of nature and environment, and films dealing with the relationship of science and fake science took place. The aim is to spread the wide spectrum of films to the audience, to university and college students in and outside the country who major in film studies, and to pass more and more of these films to one of the tv channels.

Read more: Filmfreeway.com

Big Data Cube Standard: SQL Standard Draft Established

2015-oct-17: With the NoSQL movement, many voices have been heard that the classical database language, SQL, does not support Big Data adequately. The latest extension of SQL with query support for massive multi-dimensional arrays demonstrates the power of declarative languages in a new realm, Big Data Cubes. Implementations like the rasdaman array analytics engine underpin this with a proof of high performance and scalability.

In practice, arrays appear in manifold ways in science and engineering, but also in social media and business analytics. Examples include satellite imagery, weather forecast data, microscopy data, cosmological simulations, connection information in social networks, and business datacubes. Frequently, such multi-dimensional arrays constitute the "Big Data" part inapplications.

"Due to the lack of array support in SQL, industry and data centers had to resort to self-programmed workarounds which typically lead to 'silo solutions' which are limited in functionality, hard to maintain, and not scalable", observes Peter Baumann, Professor of Computer Science at Jacobs University and inventor of Array Databases. Together with Jim Melton (Oracle) and Dimitar Misev (Jacobs University) he is editor of the SQL/MDA specification. Baumann and his team, including Misev, have implemented rasdaman ("raster data manager"), a highly flexible and scalable Array Database system which meantime is in manifold practical use.

While this may sound like a field for the techies among us, it effectively heralds a disruptive change in how we deal with Big Data. Due to technical limitations, data today get subdivided into "the data" and metadata, both being maintained and served using entirely different paradigms. For example, metadata may sit in queryable SQL databases while data reside infiles offered for download. "Common opinion is that metadata are small, smart, and searchable whereas data are big, clumsy, and only amenable to simple download", explains Baumann. "With SQL/MDA, this age-old subdivision can be overcome, and we can freely mix queries on data and metadata - effectively, we do not need this subdivision any longer." The resulting single, coherent information space will be instrumental for a new era of user-friendly services on large-scale, heterogeneous data assets, and ultimately for our understanding of "Big Data.

At its Tokyo meeting this October, the ISO SQL working group has decided to suggest the meanwhile completed specification for "CD ballot", that is: for comments by the nations participating in ISO. Final adoption as an ISO standard is expected for 2017.

ESA Conference "Big Data from Space" 2016

2015-oct-31: The Second Conference on Big Data from Space (BiDS '16) will be held at the Auditorium of Santa Cruz de Tenerife (Spain) on 15-17 March 2016. The conference is jointly organised by the European Space Agency (ESA), the Joint Research Centre (JRC) of the European Commission, and the European Union Satellite Centre (SatCen), in collaboration with the Instituto de Astrof=C3=ADsica de Canarias (IAC).

The objective is to bring together researchers, engineers and users working in the Big Data from Space sector. The focus of the Big Data from Space Conference is on Data Science, addressing the whole data lifecycle (from data acquisition by space-borne and ground-based sensors to data management, analysis and exploitation). Various space-related domains are concerned, like Earth Observation, Space Science, Solar System Objects, Space Situational Awareness, Secure Societies, etc.

The conference addresses various themes and offers several options for contributions: oral presentations in standard or invited sessions, poster sessions, demo sessions, elevator pitches, etc.

Conference details are available at: congrexprojects.com

Please feel invited to submit, by November 07, two page abstracts via theelectronic submission system, as indicated on the conference web site.

Notification of Acceptance will be provided by December the 20th. Additional information on the Final Program, hotel accommodation, transports, etc. will be provided on the Conference Web site.

rasdaman newsletter 08/2015

Release of rasdaman 9.1 Brings More Performance, Functionality

Version 9.1 of the pioneer Array Database, rasdaman, is now available fordownload. It represents yet another important increase in performance, functionality, and OGC standards support.

After relentless community efforts over the last months, version 9.1 of rasdaman is now available from the open-source portal www.rasdaman.org. Dimitar Misev, a core developer of rasdaman, announced the news on the rasdaman list indicating a series of highly pertinent enhancements.

One set of new features concerns functionality and performance, it includes: External tool support is available for R and python; queries automatically respect null / nodata values; based on user experience, new functionality has been added which specifically eases and accelerates massive timeseries analysis; a new client/server protocol boosting performance and scalability; type management has been simplified by integrating it with the query language.

Further, support for OGC standards has been broadened: WCS-T, the forthcoming OGC standard for Web-based Big Data maintenance and mashups, is fully implemented; support for WMS 1.3 has been added; the WCS Web client offers comprehensive support for WCS Core and all Extensions; and WCPS is prepared for the forthcoming version 2.0 of the standard which is currentlyunder preparation.

Notably, the rasdaman company keeps on contributing code. In addition to manifold detail code enhancements, with version 9.1 null/nodata support has been released from the commercial to the open-source version, a feature which has been repeatedly been asked for by the data science community.These contributions have been made possible through EU funded projects PublicaMundi and EarthServer-2.

At the EarthServer review in Fall 2014, independent reviewers attested, based on "proven evidence", that rasdaman will "significantly transform the way that scientists [...] will be able to access and use data in a way that hitherto was not possible".

EarthServer Project Goes into the Second Round

Pushing the boundaries of Big Earth Data services, the intercontinental Earthserver initiative enables researchers to browse, access, and analyze massive multi-dimensional data sets from a wide range of sources. Big Earth Data at your fingertips - this is the vision of EarthServer for unleashing the potential of Big Data through a disruptive paradigm shift in technology:

  • from isolated silos of data with disparate functionality towards a single, uniform information space
  • from a difficult, artificial differentiation between data and metadata access to unified retrieval
  • from zillions of files towards few whatever-size datacubes
  • from limited functionality to the freedom of asking anything, anytime, any server in a peer network of data centers worldwide.

In phase 1, EarthServer has established open ad-hoc analytics on massive Earth Science data, based on and extending the leading Array Database technology, rasdaman. According to EU Commission and inde pendent reviewers,rasdaman will "significantly transform the way that scientists in different areas of Earth Science will be able to access and use data in a way that hitherto was not possible" as demonstrated by portals with over 230 TB of spatio-temporal data. EarthServer "with no doubt has been shaping the Big Earth Data landscape through the standardization activities within OGC, ISO and beyond".

Now phase 2 of EarthServer has started, with an even more ambitious goal:data centers will provide at least 1 Petabyte of 3-D and 4-D datacubes. Technology advance will allow real-time scaling of such Petabyte cubes, and intercontinental fusion. This power of data handling will be wrapped into direct visual interaction based on multi-dimensional visualization techniques, in particular: NASA World Wind. Following the motto "a cube says more than a million images" EarthServer has set out to redefine the BigData service landscape even more.

This way, critical support will be given to Copernicus and the Sentinel satellite data: a single 3D x/y/t datacube will be constructed for each satellite instrument so that millions of images form a single, simple data space, irrespective of its size resulting. Likewise, each climate dataset will form a single 4D datacube. Access to these cubes is through a clean-slate standards-based query language on n-D grids, OGC WCPS. This yieldsthe agility that any query can be sent at any time, without admin intervention on server side. Multiple cubes can be combined based on parallel, distributed processing. Altogether, the WCPS language allows navigation, extraction, aggregation, and fusion of any-size space/time data cubes using simple, yet powerful query operators. The consortium consists of Jacobs University (Germany, coordinator), , rasdaman GmbH (Germany, SME), , Plymouth Marine Laboratory (UK), European Center for Medium-Range Weather Forecasts (UK),MEEO s.r.l. (Italy, SME), and CITE S.A. (Greece, SME). Additionally, twohigh-profile international organizations participate: NASA (US) and National Computational Infrastructure (Australia).

US Government Looking at Datacube Processing Standards

In face of the Big Geo Data challenges the OGC standards for spatio-temporal keep advancing. Recent emphasis was on the semantic-rich server-side processing of coverages by bridging the Web Coverage Service (WCS) 2.0 and the Web Processing Service (WPS) 2.0. The standards observing group of US ?Federal Geographic Data Committee (FGDC) sees coverage processing as a mandatory standard in future.

Server-side filtering and processing of spatio-temporal coverages is a core trend emerged in 2014. The Web Coverage Service (WCS) standard suite represents a modular framework for versatile access to coverages; a special extension, WCS Processing, allows to send datacube queries to a server using the OGC Web Coverage Processing Service (WCPS) language. Up to now,the WCPS way of processing was separate from the OGC Web Processing Service (WPS). The new "WPS Application Profile - Coverages" (short: WPS-Cov,OGC document 09-045r1) has been adopted unanimously as OGC Discussion Paper by the WPS Standards Working Group. It specifies how to invoke WCS and WCPS functionality using the WPS protocol. A simple parameter mapping enables WCS and WCPS requests over WPS as carrier while remaining fully conformant with all these specifications individually.

In an earlier project funded by the European Space Agency (ESA) such mechanisms have been investigated successfully: a commercial client, ChartLink by Envitia, communicated with an open-source server, rasdaman by JacobsUniversity. "WPS-Cov adds coverage semantics to WPS; on the other hand, WCS is complemented with asynchronous processing, something tremendously useful for complex coverage analytics" explains Peter Baumann, editor of this specification as well as the WCS standards suite.

Hence, with its Web services portfolio OGC offers a tailorable, modular spectrum of standards ranging from simple access with WCS over agile analytics with WCPS to arbitrary algorithms with WPS. The recent specificationcontributed by Jacobs University closes the last gap, arriving at a seamless spatio-temporal Big Geo Data suite.

This sustained activity on flexible, value-adding coverage services has also attracted attention of the US Federal Geographic Data Committee Standards. In its October update published by Julie Binder Maitre the WCS suite components receive particular attention. FGDC reports that in its Fall 2014 meeting the Geospatial-Intelligence Standards Working Group (GWG) voted to "add the WCS Processing Extension as an emerging standard, with the intention to move it to mandated in a future baseline". Peter Baumann assesses, "we expect that after this announcement and 170 Terabyte datacubes queried through WCPS in ESA installations, uptake of OGC WCPS will increase significantly in the near future".

Big Data in the Geosciences

A workshop to be held at the 2015 IEEE International Big Data Conference.

Computing is fundamentally transforming science. We have entered the so-called Fourth Paradigm in which observational data is threatening to overwhelm scientists. There are now numerous examples of the five "V's" of BigData (Volume, Velocity, Variety, Veracity, and Value) impacting the geosciences. The rapid improvements in scientific instruments have led to a massive growth in data production. As data volumes often prohibit transferof such Big Data, it is becoming increasingly important to develop sustainable infrastructure, clever data management and reduction analytics, and community, around Geoscience Big Data. The increasing emphasis on sharing and collaborating has led to a growing focus on discovery and variety.

In addition, scientific data, and the processes used to analyze that data, can differ substantially from those encountered with Big Data in other fields. Moreover, educating the next generation of geoscientists is itself becoming a pressing issue.

This workshop focuses on highlighting the state-of-the-art research surrounding Big Data in the geosciences. The Workshop Chairs and Program Committee, with assistance from the Open Geospatial Consortium, seek to engagethe scientific community in understanding their big data challenges and problems. Finally, we aim to identify gaps in current technologies as well as gaps in the education of new data scientists. This workshop will provide a venue for participants to showcase innovative tools and services, build new collaborations, and hear from application scientists and educators about the challenges they face.

The Program Chairs of the workshop are Tom Narock (Marymount University, USA), Marshall Ma (Rensselaer Polytechnic Institute, USA), and Peter Baumann (Jacobs University Bremen, Germany).

rasdaman newsletter 08/2014

OGC announces Big Data Domain Working Group

13 August 2014 - At the recent OGC meetings in Geneva, members of the OGC formed an OGC Big Data Domain Working Group. This OGC domain group provides an open forum for discussions and standards recommendations for Big Data interoperability, access, and analytics related to geospatial information. To this end, the open forum will encourage collaborative development among participants representing many organizations and communities, and will ensure appropriate liaisons to other relevant working groups inside and outside OGC.

The group will consolidate findings on a public wiki to inform both the OGC membership and the greater public and allow for feedback during the editing phase and after. A report will be submitted to the OGC membership for publication as an OGC Best Practice paper.

Because location-based and geospatial data applications are major contributors to the Big Data deluge, the OGC is positioned to provide guidance on the use of OGC standards in managing such data. Further, with the advent of increased machine-machine communication, interoperability is gaining even more importance. OGC, therefore, is establishing a position addressing Big Data issues, including – but not limited to – science, implementation, market value, and societal effects.

The initial membership of the BigData WG will consist of the following members and individuals with extensive education and experience in Big Data issues:

  • Peter Baumann, Jacobs University (co-chair)
  • John Herring, Oracle (co-chair)
  • Juergen Seib, Deutscher Wetterdienst
  • Stan Tillman, Intergraph
  • Marie-Francoise Voidrot, Meteo France
  • Jeff de la Beaujardiere, NOAA
  • Bruce Gritton, US Navy MetOc
  • Chuck Heazel, WISC (co-chair)
  • Mike McCann, MBARI
  • Pedro Goncalves, Terradue
  • Don Sullivan, NASA
  • Ed Parsons, Google
  • Robert Gibb, Landcare Research New Zealand
  • Jean Brodeur, Geoconnections, NRCAN
  • Jinsongdi Yu, Fuzhou University
  • Arnaud Cauchy, Airbus Defence & Space

These members encourage others inside and outside the OGC to participate in discussions and in preparation of a final report. The OGC® is an international consortium of more than 475 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. Visit the OGC website at: http://www.opengeospatial.org/

rasdaman newsletter 06/2014

Big Science Data Coming to SQL Databases

Bremen, June 17 - Multi-dimensional sensor, image, simulation output, and statistics data make up for most of the "Big Data" in science and engineering. Last week, it has been decided to extend the SQL database language with massive multi-dimensional arrays. Initiated by Prof. Peter Baumann from Jacobs University, work has commenced on the forthcoming standard named SQL/MDA.

SQL has been tremendously successful in running any-size databases in business and administration. However, the "Big Data" in science are structured differently. Instead of simple tables, they often consist of multi-dimensional "data cubes". In Geo sciences, for example, this encompasses 1-dimensional sensor data, 2-D satellite imagery, 3-D x/y/t image timeseries as well as x/y/z geophysical voxel data, and 4-D x/y/z/t weather data. In Life sciences, there is laserscan microscopy and brain scans. And it can grow as large as simulations of the whole universe when it comes to astrophysics.

But SQL is not able to find, filter, and process such arrays, and consequently today arrays largely are maintained outside databases. Recognizing this shortcoming, Peter Baumann, Professor of Computer Science at Jacobs University Bremen, and his group have long been researching on ways to extend SQL appropriately. The rasdaman system which the group has established effectively has coined a new technology, Array Databases. In a recent technology demonstration, more than 1,000 computers have collaborated in a cloud to jointly compute the result of one single database query. This "distributed query processing" means a massive speedup, and research challenges on multi-Petabyte data cubes can be answered that hitherto were unsolvable.

Meantime, international data centers use this tool to allow scientists gain unanticipated insights into their spatio-temporal data cubes, and rasdaman installations can be found at NASA, ESA, British Geological Survey, Plymouth Marine Laboratory, Deutscher Wetterdienst, and many more.

At last week’s meeting in Beijing all national bodies participating in the SQL working group of ISO unanimously have agreed on the importance of arrays in SQL. Following thorough assessment of all options available, the group has accepted the proposal of Baumann for further elaboration. The new standard will be named ISO 9075 SQL/MDA, for "Multi-Dimensional Array"”.

rasdaman newsletter 05/2014

FOSS4G-Europe 2014 - Early Bird Registration extended to May 30 - Final Program released

It's been a busy few weeks as things start to come together for FOSS4G Europe 2014 – but now, admittedly with some delay, we have managed the challenge of compiling an exciting program, which you can find under: http://www.foss4g-e.org/program-glance

To give everyone time to have a look at the schedule before booking, we’ve extended the Early Bird registration deadline to May 30, 2014. In the meantime, many thanks to all who submitted abstracts – it has been a challenge to select the best of all excellent submissions passed in the last weeks. Altogether, the conference received about 78 abstracts in the general and 16 for the academic track. Every abstract has been carefully evaluated by the program committee to obtain a fair ranking. On the whole there has been a very satisfying presence of topics relating to the relevant Open-Source-tools like GeoServer, MapServer, Zoo, rasdaman, Grass, and several more. As well OGC-standards like WMS, WMS, WCS, WCPS, WPS, and SOS have the attention of the community. All in all the bandwidth of proposals shows a good representation of the general trends in the geospatial world. In addition the Local Organizing Committee has secured facilities to run additional workshops and a code sprint during the conference.

A lot of registrations form Europe and beyond have poured in already, so make sure you are part of the crowd in this visionary European geo happening. Sign in now! http://www.foss4g-e.org/

Rasdaman Company releasing Array Booster into Open-Source

Bremen, May 10, 2014 - The donator of the massively scalable open-source rasdaman Array Analytics engine, rasdaman GmbH, today announces the next substantial contribution: The performance-boosting direct file storage will be made available in the open-source rasdaman community. Of course, backward compatibility will be maintained: the new rasdaman version will smoothly run on existing databases.

"We wholeheartedly believe in open source, and we continue contributing to open-source rasdaman community all the time. We are glad that now we can provide this single major breakthrough in addition", says Peter Baumann, architect of rasdaman and CEO of rasdaman GmbH. "Several customers have expressed high interest in this feature. We love databases, but file-based analytics opens up scenarios where users decide not to run databases."

Until now, rasdaman stores its data in a PostgreSQL database. While this is operational in many places on this planet, a direct file-based storage can offer two distinct advantages. First, going directly to file is massively faster; preliminary tests show an improvement over PostGreSQL by almost a factor of two. Of course, PostgreSQL delivers good extra service, but if this is not needed (like often for satellite image archives) the new rasdaman storage will play out its full performance advantage.

A second advantage is simplicity: no PostgreSQL installation is needed, no database demon needs to run, so zero administration costs arise here.

One more time, this confirms the worldwide leading position of rasdaman in the rapidly expanding field of Agile Array Analytics. While research projects appear at the horizon, rasdaman remains the only fully implemented Array Database, operationally used on dozens of Terabytes.

Jacobs University maintains rasdaman community, supported by a worldwide community of contributors and the rasdaman company, which additionally offers the proprietary performance optimized version, rasdaman enterprise. As of this writing, rasdaman community is in its final steps for adoption by OSGeo Foundation. Further, it is the official OGC WCS Core Reference Implementation. In 2013, rasdaman has been awarded the prestigious Geospatial World Forum Innovation Award.

rasdaman info letter FOSS4G-Europe 03/2014

FOSS4G-Europe Welcoming Contributions

Independent software developers, scientists, industry experts, and agency representatives will come to meet at the FOSS4-Europe conference held in Bremen, Germany from July 15 to 17. Contributions can now be submitted on the conference website http://www.foss4g-e.org/ until April 15.

Under this year’s motto “Independent Innovation for INSPIRE, Big Data and citizen participation” the conference series FOSS4G-Europe (Free and Open Source Software for Geospatial - Europe) is uniting the GIS community from a broad spectrum of fields like geodesy, geo information, land management, and remote sensing. Industry and public authorities are expressly invited as they increasingly realize value and economic potential of free and open source products.

Several additional events make the FOSS4G-Europe in Bremen a unique event. A public viewing of the soccer World Championship final will kick off a week packed with events. The week will open with a workshop “Big Geo Data & INSPIRE” especially targeted at public authorities. INSPIRE Annex II and III will be discussed in relation to the pertaining standards of the Open Geospatial Consortium http://www.opengeospatial.org/. Canadian Jeff McKenna, President of the OSGEO Foundation, will open the event with his keynote talk. As a charter member McKenna has got involved with spreading and usage of open-source software for geo information systems for many years. In a plenary, the winners of the international academic NASA World Wind Challenge Europe programming contest will be announced and honored at the conference.

Being in the tradition of the worldwide FOSS4G conference series, this event will bring together open-source experts and enthusiasts from all over Europe and beyond creating a unique, visionary atmosphere.

Altogether, this unique technology demonstration paves the way for services merging data across data centers, nations, continents - ultimately, across the planet.

Conference Dates

General presentations and workshops:

  • Abstract submission: 2014-apr-15
  • Review notification: 2014-apr-30
  • Conference: 2014-jul-13

Scientific Track:

  • Abstract submission: 2014-apr-15
  • Full paper submission: 2014-apr-22
  • Review notification: 2014-may-15
  • Camera ready version: 2014-jun-15
  • Conference: 2014-jul-13
Early bird registration deadline: 2014-may-22

rasdaman newsletter 01/2014

First-Ever Direct Data Mix between US NASA and European ESA

rasdaman connects NASA Ames Research Center and ESA

Geneva, January 17, 2014. Commonly, Big Data are seen as "too big to transport" - however, often the real hindrance comes from security policies data centers have to implement to safeguard themselves. High-ranking international officials in Switzerland now have witnessed secure simultaneous access to NASA and ESA archives based on the rasdaman Big Data engine, effectively bridging the Atlantic in real time.

International delegates gathered in Geneva at the 10th Plenary of the Group on Earth Observations, GEO, to discuss future steps to create societal value from innovative hi-tech tools and data. On this occasion, a first-ever direct and real-time mixing of NASA and ESA data was demonstrated. The Web client on a laptop in Geneva accessed the NASA server to fetch a color image taken by a NASA drone, Ikhana, while it was supporting local firefighters combatting large bushfires in California. From ESA, an elevation model giving height information about the Californian hills was retrieved, originally obtained from a radar satellite. Both were combined in the browser to a 3-D landscape that could be zoomed into and viewed from all sides, similar to a virtual globe like GoogleEarth.

The client made use of the OGC Web Coverage Processing Service (WCPS) standard, a language that allows clients to task servers with processing and filtering on massive spatio-temporal data assets, such as satellite image or climate data sets. Access to the servers on both sides of the Atlantic was accomplished despite both machines are heavily secured by their operators. Not that security was compromised – the resulting work from the GEOSS Architecture Implementation Pilot (AIP) 6 showed that security-interoperable access is possible via so-called Access Management Federation setup. It establishes a well-defined level of trust between parties and supports Single-Sign-On: A user logging into at one organization of the federation will be trusted by all other members of the federation to perform managed access - and nothing else.

An international collaboration made this transatlantic gathering possible. Jacobs University and rasdaman GmbH provided the Big Raster Analytics engine, rasdaman, which NASA Ames Research Center and ESA are running on their operational servers. Secure Dimensions provided the integrated, overarching security tools. MEEO performed data ingest at ESA. Partners met during their collaboration in the OGC standards development and international research collaborations.

Altogether, this unique technology demonstration paves the way for services merging data across data centers, nations, continents - ultimately, across the planet.

OGC Adopts Extensions to Big Geo Data Standard

January 20, 2014. OGC now has adopted a series of functionality extensions to its Big Geo Data standard, Web Coverage Service (WCS). Web services implementing the WCS interface standard offer simple, yet powerful access to spatio-temporal sensor, image, simulation, and statistics data. The standards adopted now focus on data formatting and server-side processing of large-scale information to enable versatile, fast access to both detail and summary information.

The concept of coverages unifies regular and irregular grids, point clouds, and meshes. A coverage can be described as the digital representation of some spatio-temporal phenomenon, be it observed or generated. While ISO 19123 establishes an abstract model of coverages, the OGC GMLCOV standard refines this to a concrete, interoperable specification. The corresponding compliance tests offered by OGC can test compliance of implementations down to single pixel level.

Coverages can be served out via many OGC-based interfaces, such as WMS, WFS, WCS, SOS, and WPS. The broadest functionality set is offered by WCS, ranging from simple subsetting to complex analytics. The latest WCS extensions add in various functionality facets. For the encoding of grids in the GeoTIFF format, the WCS GeoTiff Coverage Encoding Extension [OGC 12-101] has been established. One or more "channels" or "bands" can be extracted conveniently from a multi-band or hyperspectral coverage through the functionality defined in the WCS Range Subsetting Extension [OGC 12-040]. Retrieval of grid coverages in reduced resolution is possible through the WCS Scaling Extension [OGC 12-039]. The WCS Interpolation Extension [OGC 12-049] serves to control interpolation in a server on all operations where interpolation occurs, such as CRS transforms and grid scaling.

A particular answer of OGC to the Big Data challenge is the Web Coverage Processing Service (WCPS) standard. This coverage query language allows clients to send queries in a simple language for evaluation on a server, in clusters, clouds, or federations of data centers. The WCS Processing Extension [OGC 08-059r4] connects the abstract WCPS language into WCS, allowing to send queries as WCS-style requests using any standardized protocol.

While WCS Core and Extensions are domain neutral and generically spatio-temporal, WCS Application Profiles bundle functionality for particular purposes and applications. The WCS Earth Observation Profile [OGC 10-140r1] adopted now establishes such an Application Profile for large-scale satellite imagery.

Altogether, WCS constitutes today's most powerful and flexible service environment for Big Geo Data appearing as coverages. The broad support by industry, academia, and government agencies during the vote underlines the broad worldwide uptake of WCS. A significant number of WCS implementations exists already, and many large vendors are preparing their own products based on WCS. A mapping of the INSPIRE Download Services to WCS is in preparation.

Comprehensive information about OGC's Big Coverage Data standards is provided at http://external.opengeospatial.org/twiki_public/CoveragesDWG/WebHome

rasdaman going Space

Darmstadt, January 16, 2014. The European Space Agency (ESA) has now confirmed selection of a Big Data engine, developed at Jacobs University Bremen in collaboration with a small hi-tech company, to go on board of a satellite. Goal is to demonstrate how satellites can process data on board and participate in planet-wide analytics networks.

Satellite imagery is indispensable for rapid response to disasters such as floods and wildfires, but also for everyday's governmental tasks, like land use monitoring: who is farming what and how much? However, data from a satellite take a long journey: when the satellite is flying over a ground station, it attempts to download as much of the imagery taken as possible. That is passed on to some data center which undertakes a lot of processing, puts the resulting products into its archive, and eventually publishes them. Not always is it easy for users to find the needle, that is: those data they are interested in, from the resulting haystack of data.

Researchers at Jacobs University want to change this now, allowing regular users to get data directly from the source. To this end, this image source, the satellite, will be instrumented to allow flexible processing and filtering of data on board. This is accomplished by the rasdaman system ("raster data manager"), an innovative analytics engine specialized on science and engineering data which is already in operation internationally, among others at NASA Ames and ESA. "A core challenge will be to micro-miniaturize the rasdaman engine", explains Prof. Peter Baumann, architect of rasdaman. Together with his team at Jacobs University and a research spin-off, rasdaman GmbH, he is preparing the system to use less power while fulfilling more tasks. This effort is not just a one-time endeavour: insights gained can help to save substantial amounts of electrical energy in computers on the planet in future.

In ESA's OPS-SAT project, a so-called CubeSat will be launched into an earth orbit. A CubeSat is a small, low-cost satellite which can be quickly and flexibly assembled for different configurations and tasks. CubeSats have been invented by US universities to overcome the high costs and time it takes to launch satellites, effectively hindering them from doing space research. While traditional remote sensing satellites cost hundreds of millions of Euros and have development times of a decade and more, CubeSats can be flexibly adapted to new experiments and launched with moderate costs. Today, the final specification of the satellite has been released for review by the Jacobs software engineers. Launch of OPS-SAT is foreseen for early 2016.

The OPS-SAT model hosts a Linux PC, camera, and GPS so that images and their position on earth can be recorded, processed, and transmitted to earth. Accessing the satellite will be done through open standards well known by geo service providers on Earth, so that no special programs are needed; any regular Web browser is sufficient. Therefore, the researchers see this as a contribution towards democratizing data access: on principle will allow anybody to access such a service via Internet - neither expensive receiving stations, nor huge data centers will be needed.

OGC navigator - Jacobs University supports OGC certification

Bremen. On the occasion of Intergeo 2013 Jacobs University introduces a new service adressed to software vendors: mentoring and conformance guidance for OGC certification.

This competent support implies extensive cost- and time savings. Varieties of geo software make the decision what product should be provided difficult: OGC, ISO, INSPIRE - many requirements software must fulfill to convince customers. Especially conformance with relevant standards will be checked critically. Conversely it´s difficult for suppliers standing out from the crowd and profiling themselves because everyone claims standard conformance. For this reason OGC has established standardized conformance testings - a seal of quality which is an important criteria providing software.

Jacobs University has coined relevant OGC standards for many years. Under the direction of Prof. Dr. Peter Baumann, the research team of the department of Computer Science has developed more than a dozen OGC standard specifications, i.e. the unified coverage model, WCS suite and all conformance testings for coverage services. Particularly with regard to INSPIRE Annex II und III where coverages play a central role, this worldwide unique expertise is of eminent importance for products claiming standard conformance. Jacobs University provides this expertise to general public. Seminars and advisory services will be offered to data and service providers.

“Principally each company is abled to carry out conformance testings for itself because these testings are freely accessible”, Prof. Dr. Peter Baumann, head of Jacobs University reseach team, explains. “However for staff it´s essential familiarizing themselves with underlying facts and practicing the test engine autonomously. This procedure will save time and money. We are cutting corners.”

For vendors this procedure implicates cost savings and accelerates time-to-market. Furthermore researchers offer more intensive testings on demand as ruled by OGC, which means an increase of product quality. Though Jacobs University guaranties strict confidentiality: no product leaves its place of development - only web access to product installation is necessary for testing. All results will only be communicated to the awarding authority. All testings are carried through under full cost-control. Every step of certification can be booked individually as well as advisory service.

Further information: http://jacobs-university.de/ogc and email p.baumann@jacobs-university.de .

rasdaman newsletter 11/2013

FOSS4G Europe Conference at Jacobs University Bremen

Bremen, November 2013. Europe's largest ever event on free geospatial and location-based software will be held at Jacobs University in Bremen, Germany from July 15 through 17, 2014. Also at this conference, the winners of the NASA Worldwind Europe Challenge will present their innovative apps to the public.

More than 500 delegates are expected; notably, the FOSS4G-Europe conference does not only attract software developers and open-source service companies, but also industry and governmental agencies which increasingly recognize the value and cost savings potential of open-source software. "We are proud and excited to have the honour of organizing the first pan-European FOSS4G", says Conference Chair Prof. Dr. Peter Baumann, head of the Large-Scale Scientific Information Systems (L-SIS) Research Group at Jacobs. Adds Organizing Committee Chair and L-SIS group member Susanne Ebrecht, "This event actually will contribute to bridging gaps between open-source and commercial software worlds". The main financial support is coming from the transatlantic Big Geo Data initiative, EarthServer, funded by the European Commission. Additionally, sponsors are being invited from industry, government, and other organizations.

FOSS4G (Free and Open Source for Geospatial) is a conference series held under the auspices of OSGeo. This is a worldwide acting, independent, nonprofit legal entity established to support the collaborative development of open source geospatial software, and promote its widespread use. Its international Board of Directors elected periodically by the OSGeo Charter Members, one of which is Prof. Dr. Peter Baumann. The OSGeo Board now has agreed to hold the FOSS4G-Europe Conference on the campus of Jacobs University in Bremen coming summer. Following an outstandingly successful FOSS4G-CEE, for Central and Eastern Europe, in Bucharest this summer, it was decided to widen scope to a pan-European event next year. Expecting over 500 participants this will be the largest event of its kind in history, only surpassed by the global FOSS4G conference. As part of this activity network, there are continental FOSS4G events held in Africa, Asia, and Latinamerica.

The NASA Worldwind Europe Challenge is an annual programming competition organized by NASA, Patrick Hogan, and Prof. Maria Brovelli, University Como, Italy. The challenge is to develop solutions that serve the spatial data needs of the European Community and respond to the INSPIRE Directive. Solutions sought will use NASA’s World Wind, an open-source virtual globe like Google Earth. Data used should relate to INSPIRE, the Infrastructure for Spatial Information in Europe. Examples of applications include a wildfire management tool, a tool for urban management, tools for weather display and climate research, and others.

The term free and open-source software denotes computer programs which, together with their blueprint (called the program's source code), can be distributed without ownership limitations and can be modified and improved by anybody. This prevents from software vendor dependencies and typically leverages resources from developer communities whose passion it is to continuously improve freely available software. As universities frequently are engaged in open-source projects, it is not uncommon that such projects actually define the state of the art. Prominent examples include the Apache Web server and the rasdaman array database system.

URL: http://www.foss4g-e.org/

OGC Releases Compliance Tests for WCS 2

Bremen, November 2013. The Open Geospatial Consortium (OGC) announced the new compliance test for the OGC Web Coverage Service (WCS) Interface Standard 2.0.

The OGC Compliance Program provides an online free testing facilityto check for proper implementation of the standards, and a process for certification of products that are compliant with OGC standards.

Once an organization's product has successfully passed a test, the organization can acquire a license to use the “Certified OGC Compliant” mark on web pages and other marketing material. The product is also promoted on the OGC Compliant Products Web page. The OGC invites organizations to start the process of getting their products certified. The Certified OGC Compliant mark increases potential buyers' confidence in a product, making the product easier to market and sell.

The candidate for Reference Implementation is rasdaman, available in open-source at http://www.rasdaman.org.

Release of the WCS test suite is in parallel with the release of further test suites for GML, WFS, SPS, and SOS.

Contact: Luis Bermudez, Director of Compliance, lbermudez@…

EarthServer Climate Data Service integrated at the European Space Agency

Bremen, November 2013. The EarthServer Climate Data Service has been successfully integrated at the European Space Agency: a powerful hardware infrastructure consisting of 40CPUs and 120GB RAM hosts the subsystems to handle EO products and model data.

Atmosphere and Climate products collected within the EarthServer project http://www.earthserver.eu as well as EO products provided by the European Space Agency, are presently accessible through the Multi-sensor Evolution Analysis (MEA) platform. The EarthServer platform runs rasdaman as its enabling analytics engine. The detailed list of EO products and model data is available at https://mea.eo.esa.int/data_availability.html.

Single Sign On (SSO) technology is used to authenticate and login in to the MEA system.

MEA provides to users the following functionalities:

• Real time data access and visualization (time series display)
• Data comparison (e.g. same variable from different data sources)
• On-the-fly data analysis (extraction of statistical parameters)

Honors for Jacobs Master Thesis on rasdaman

Bremen, November 2013. Jacobs alumna Mihaela Korte has been honored for her master thesis with the ‘Studienpreis für Mittelstandsforschung ‘ by local wiwib e.V.

Jacobs alumna Mihaela Korte has been honored for her master thesis with the ‘Studienpreis für Mittelstandforschung’. The prize is awarded biannually by local association wiwib e.V (Wirtschaft-Wissenschaft-Bremen)* in cooperation with the universities of the metropolitan region Bremen-Oldenburg.

Mihaela Korte from Moldova graduated from Jacobs University with an outstanding BA and MA in Computer Science. She wrote her master thesis on „Integrating user-defined functions into the querying process of an array database“ whilst also working at rasdaman GmbH with a focus on big data technology.

The 24-year-old was honored for the development of a new software technology which makes it possible to analyze large amounts of science and engineering data simultaneously with different programs. These programs are working collaboratively while fulfilling different tasks. Commonly, such huge data volumes are referred to as Big Data.

“While experts are still discussing which approach is the best for the analysis of data, Korte’s work as laid the foundations for different technologies to supplement each other. Her work has opened new perspectives for research and practical applications, “ comments Peter Baumann, Professor of Computer Science and Korte’s supervisor at Jacobs University. “Areas of application are manifold, ranging from climate analysis to disaster protection”. Mihaela Korte’s programs are meanwhile distributed by the company as part of its raster data manager software product, rasdaman. The Jacobs alumna has since moved to the US where she is working for a global software company.

*wiwib e.V. is a local association for the support of the Faculty of Business Studies and Economics at Universität Bremen. In cooperation with Bremen businesses and universities from the metropolitan region Bremen-Oldenburg it regularly honors outstanding Bachelor and Master theses.

rasdaman newsletter 08/2013

rasdaman 8.5 released

Bremen, August 2013. Jacobs University and rasdaman GmbH are proud to announce release of open-source rasdaman community 8.5.

This version includes numerous improvements and new features. Most important, performance has been increased again, obtaining a speedup of 2 - 3 orders of magnitude in many cases. Additionally, several more OGC WCS extensions have been implemented. See http://www.rasdaman.org/wiki/Download.

Following roll-out of 8.5, rasdaman 9.0 is planned for release this fall. It will contain substantial innovations, such as cloud and cluster parallelization, leading to unprecedented scalability. Another feature many data centers were eagerly awaiting is "in-situ storage": the rasdaman engine can answer queries not only on the database, but also on any pre-existing data archive structure. The rasdaman 9.0 version will be presented to the public this September at INTERGEO 2013 and several more national and international events.

OGC Progressing with Function-Rich Big Data Standards

August 20, 2013. Yesterday voting on the new OGC standard for Coordinate System handling of spatio-temporal data sets has finished with all YES votes. This extends the OGC Web Coverage Service (WCS) suite, together with several further specifications submitted earlier which are disclosed for public commenting. Together, this makes WCS Swiss Army Knife for big and small coverage data, such as sensor data, raster images, timeseries, point clouds, and meshes.

Among the supporters of the new "OGC WCS Extension - CRS" with OGC document number 11-053 are US NOAA and USGS, German AdV, French BRGM and METEO-FRANCE, Australian CSIRO, the European Union Stellite Centre, Oracle, and EADS. "This new standard closes an important gap, as for the first time it allows to treat spatial and temporal coordinates alike" explains Peter Baumann from Jacobs University, the editor of the specification, and details "this will tremendously ease access to and analysis of satellite image timeseries and weather data, for example".

Prior to voting on the adoption of a standard, as with WCS CRS, OGC mandates a public Request for Comment period where the candidate standard is made available to the public and comments are sought. The document editors duly consider all comments, and the revised version goes into voting.

In case of WCS, several further specifications are in this pipeline: WCS Range Subsetting, which allows to extract channels from hyperspectral images or multi-variable climate data sets; WCS Processing, which establishes a bridget between the simple data access protocol of WCS and the advanced filtering and processing query language of the Web Coverage Processing Service (WCPS); WCS Scaling, which allows clients to get downscaled data sets with reduced resolution, such as for Web browser display and thumbnails; WCS Interpolation, which gives clients fine-grain control over any interpolation the server performs during its processing.

A RESTful protocol binding for WCS is in progress by the OGC WCS Standards Working Group and will likely be rolled out soon as well.

"The team has worked hard over the last few years, and now we are reaping the benefits - WCS is the most powerful, comprehensive Big Earth Data service suite available" says Peter Baumann. "Not the least because we have harmonized it across OGC, so that coverages can be exchanged freely between WFS, WCS, WCPS, WPS, and SOS."


rasdaman inventor now OSGeo Charter Member

Bremen, August 2013. Peter Baumann, Professor of Computer Science, has been elected Charter Member of OSGeo.

The Open Source Geospatial Foundation, or OSGeo, is a not-for-profit organization whose mission is to support the collaborative development of open source geospatial software, and promote its widespread use. OSGeo also serves as an outreach and advocacy organization for the open source geospatial community, and provides a common forum and shared infrastructure for improving cross-project collaboration.

"It is an exciting opportunity to support the open source idea in all its variety", says Peter Baumann and explains, "Not only do the Charter Members elect the Board of Directors, but they also actively contribute in discussion." Baumann is well known in OSGeo circles with the Big Data Analytics engine, rasdaman, which is jointly developed by Jacobs University and rasdaman GmbH, a research spinoff. Research around rasdaman has pioneered the area of Array Databases, which is seen as a key Big Data technology by major database vendors. Further, Baumann is editor of international standards in the field and chair of several standards working groups.

More information:

rasdaman newsletter 07/2013

Space-Time Coordinates for Big Data: OGC Voting on Adoption

June 2013. The Open Geospatial Consortium (OGC) has started voting on adoption of the OGC Web Coverage Service (WCS) Coordinate Reference System (CRS) Extension Standard.This is the final step following a successful 60-day public Request for Comment period. Right after voting started, several institutions and companies have already voted in favor of WCS CRS. Voting ends on August 19.

WCS defines an open standard interface to easily access multi-dimensional, spatio-temporal coverages, such as sensor data, satellite imagery, image time series, simulation data, point clouds, and meshes.

The OGC WCS CRS Extension defines an extension to the WCS Core that enables requests to specify that coverages be returned in one of a number of different Coordinate Reference Systems. While the WCS Core defines access only in the coverage’s Native CRS (i.e., the one in which the coverage is stored on the server), the WCS CRS Extension defines how to request and obtain a coverage in a CRS different from the Native CRS. This flexibility is important, for example, in exchanging data between different environments.

The OGC WCS CRS Extension is free and can be downloaded from the OGC Request for Comment page at http://www.opengeospatial.org/

Always Take the Weather With You

June 28, 2013. At a meeting hosted by NASA, foundation was laid for Big Data support in the field of Meteorology, Oceanography, and Aviation. Rasdaman CEO Peter Baumann, being editor of the Web Coverage Service (WCS) standard, was invited to moderate the meeting from a standardization perspective. The rasdaman system will serve as an experimental platform for verifying concepts during development.

There are many reasons why pilots need to know the weather along their flight route, such as to avoid thunderstorms and minimize fuel consumption. However, this information is not always easy to obtain routinely. Similarly, international exchange of information between meteorological institutions is not a standard routine today.

In a sizzling hot New Orleans, an initiative of NASA, UK MetOffice, the French National Geographic Institute, and rasdaman GmbH / Jacobs University met this week at NASA Stennis to establish the core concepts of the so-called "WCS Application Profile - MetOcean" candidate OGC standard. Goal is to establish simple, yet powerful ways for accessing that specific part from the huge reservoir of weather forecast data which is of interest to a particular user. One example is obtaining the weather along an aircraft's flight path in space and time.

The underlying data range from two-dimensional to five-dimensional. Most systems today cannot serve such data, therefore the rasdaman Big Data Analytics engine was selected for trying out and varying concepts prior to their standardization. This will maximize practical usability of the forthcoming standard.

Peter Baumann, CEO of rasdaman and Professor of Computer Science at Jacobs University, was invited to moderate the meeting from an OGC standards perspective. He is chair of the raster-relevant working groups in OGC and editor of the Web Coverage Service (WCS) standards suite which was unanimously chosen as the trial platform for developing the specification.

Big Data from Space… directly to the Users

Frascati, 12 June 2013. How can we make the large amounts of Earth observation data accessible to a wider user community? To explore this idea, some 250 science, industry and policy-making representatives and national delegates from Europe, the US, Australia, China, and Africa met at ESA’s ESRIN space centre in Frascati, Italy for ESA’s first ‘Big Data from Space’ event.

Data from Earth observing satellites and other remote sensing technologies are growing in volume and diversity at an exceptionally fast rate. This poses challenges and opportunities for their quick access, stewardship and applications.

Peter Baumann was invited to present Jacobs University’s research on flexible, scalable search and processing on huge data archives, Overall, more than 50 presentations during the three-day conference stimulated discussion between the different communities involved in the business of providing and manipulating very large-scale data and complex analyses of satellite and in situ Earth observations.

Diversity of space data, the combined use of data from different space missions and the fusion of satellite data with non-space data lead to new types of user applications. This affects the way such data are collected, processed, delivered and preserved. There is an increasing need to serve a wider user community looking to retrieve simple, understandable and immediate information from remotely sensed data that have undergone complex processes and analyses.

Given the overwhelming success of the event, ESA is now considering whether this can become a recurring event.

rasdaman newsletter 04/2013

Spatio-Temporal Coordinates for Big Data: OGC Seeking Comments

April 2013. The Open Geospatial Consortium (OGC) announced a call for public comment on the OGC Web Coverage Service (WCS) Coordinate Reference System (CRS) Extension Standard.

WCS defines an open standard interface to easily access multi-dimensional, spatio-temporal coverages, such as sensor data, satellite imagery, image time series, simulation data, point clouds, and meshes.

The OGC WCS CRS Extension defines an extension to the WCS Core that enables requests to specify that coverages be returned in one of a number of different Coordinate Reference Systems. While the WCS Core defines access only in the coverage’s Native CRS (i.e., the one in which the coverage is stored on the server), the WCS CRS Extension defines how to request and obtain a coverage in a CRS different from the Native CRS. This flexibility is important, for example, in exchanging data between different environments.

The OGC WCS CRS Extension is free and can be downloaded from the OGC Request for Comment page at https://opengeospatial.org/standards/requests/100.

The OGC WCS Standards Working Group will consider all comments when preparing a final draft of the candidate standard. The comment period ends 5 May 2013. The OGC is an international consortium of more than 480 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. OGC Standards empower technology developers to make geospatial information and services accessible and useful with any application that needs to be geospatially enabled. Visit the OGC website at https://opengeospatial.org/contact.

Big Earth Data: The Programming Contest

April 2013. In the context of the ESA BigDataFromSpace Conference [1], the EarthServer initiative [2] is sponsoring a programming contest to establish innovative and surprising applications on multi-dimensional Earth data. The contest is supported by Jacobs University, rasdaman, ESA, and the European Commission [3].

Students worldwide are invited to build apps/clients for Big Earth Data. A variety of large datasets , ranging from 2-D through 4-D, is provided for your perusal.

No need for downloading, though - the rasdaman Array Database System gives versatile, fast query access on massive multi-dimensional raster data based on the OGC Web Coverage Processing Service (WCPS) open query standard. Clients compose query strings (typically from graphical user interaction), send them to the server for evaluation, and visualize the results coming back. To ease client development, a versatile Web development toolkit is provided, offering input widgets like sliders and gauges as well as output widges for diagramming, image display, and more.

Any common client platform technology (Android, iOS, HTML5, X3D, ...) is acceptable as long as there is open-source support for it.

Winners will surprise and impress the expert jury with a tremendously ( cool | geeky | useful | novel )+ client/app. The first three winners receive a money award and have the opportunity of presenting their development to a worldwide audience of experts in the "BigDataFromSpace" conference plenary.

A team of geeks is curiously awaiting for your ideas, and is prepared to assist where needed.

rasdaman 8.4. released

April 2013. Version 8.4 of the rasdaman Array Analytics Engine [1] has been released. Concurrently, this version is made available for the open-source variant, rasdaman community [2], and the proprietary enhancement, rasdaman enterprise [3].

The international team was breathtakingly busy adding features and closing tickets. Among the new features is a classification operator suitable for elevation coloring, but also for content-based image analysis. Maybe most important, the whole release process has been scrutinized and the trac wiki has been opened to the public for writing. For the community this means they have more overview and improved ways of impacting direction of development.

"By introducing semantic versioning we now can provide stable releases better while, at the same time, allowing developers to collaborate on new features" says Dr Beccati, senior software developer from Ferrara / Italy who has introduced these new features into the release management process.

Meantime development continues. This summer, rasdaman 9.0 will be released bringing along substantially enhanced functionality and novel parallelization features.

The rasdaman technology has coined the technology of Array Databases [4] and still today is the most advanced and operational system. It is reference implementation of the OGC Web Coverage Service (WCS) Core and Web Coverage Processing Service (WCPS). In a unique way rasdaman combines performance, scalability, flexibility, and functional power.

Rasdaman community is available for download as source code, installable RPM packages, and as preconfectioned Virtual Machine. For ways to obtain rasdaman enterprise, see the company website [3].

rasdaman newsletter 08/2012

The OGC adopts a Unified Coverages Encoding Standard

August 2012. The Open Geospatial Consortium (OGC®) has adopted OGC GML Coverages (GMLCOV) 1.0.1 as an official OGC standard. OGC "coverages" are a unifying, standardized concept for the digital representation of phenomena that vary over space and possibly time. Examples of such geospatial coverages include Earth images, referenced and non-referenced rasters, curvilinear grids, and point clouds.

A corrigendum version has now been issued for the OGC GML 3.2.1 Application Schema - Coverages, nicknamed GMLCOV. Based on OGC Abstract Topic 6 (which is identical to ISO 19123), GMLCOV provides a unified method for representing OGC coverages in GML and any other suitable data format, such as GeoTIFF and NetCDF, and in a variety of dimensions, including 2-D x/y, 3-D x/y/t and x/y/z, and 4D x/y/z/t dimensions. As GMLCOV is independent of a particular service definition, it allows coverage data to be exchanged through different types of services that implement these OGC standards. As such, GMLCOV integrates Geography Markup Language (GML), the OGC Sensor Web Enablement (SWE) Common Standard, the OGC Web Coverage Service (WCS) Interface Standard, and others.

The new version, GMLCOV version 1.0.1, provides several enhancements, such as versatile encoding options that include binary formats with or without GML header. To better support this, modularization of the specification has been streamlined.

The OGC coverage model is widely used. For example, the European Commission has supported the development and unification of OGC coverages because these coverages are an important element in the standards framework for INSPIRE (the European legal framework for a common Spatial Data Infrastructure). Also, the widely used GMLJP2 standard, which augments JPEG2000 imagery data with GML-encoded geospatial metadata, relies on the GMLCOV standard.

GMLCOV is free and can be downloaded from at https://portal.opengeospatial.org/files/48553. The OGC is an international consortium of more than 460 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. OGC Standards empower technology developers to make geospatial information and services accessible and useful with any application that needs to be geospatially enabled. Visit the OGC website at http://www.opengeospatial.org/contact.

DREAMing Service Innovation Forward

While "Big Data" increasingly is on the radar of all of us, another big wave is approaching us from behind: the quest for substantially more flexible, powerful, and user-friendly access to geo information. The DREAM (Decision Support and Real Time Earth Observation Data Mangement) project, which is funded by the European Space Agency (ESA), has set out to establish novel solutions or data centers and their users. Now DREAM has finished a concise requirements study paving the way to innovative services.

August 2012. The DREAM consortium, represented by rasdaman GmbH, has proposed several standard drafts extending WCS with essential functionality like image scaling, band selection, and reprojection. These drafts are currently being discussed within OGC, with a planned adoption aiming at the end of 2012.

The DREAM project addresses two core issues of geo service provision: on the one hand, integrated management of high-volume, heterogeneous satellite data; on the other hand, user-driven, quick, and flexible filtering and processing of data stored in such large-scale archives gets tackled. Operational installations will be provided for EUSC (European Union Satellite Centre) near Madrid, EMSA (European Maritime Safety Agency) in Lisbon, the ESA data center in Frascati, and NLR (National Aerospace Agency) in the Netherlands. In the first project period, which has ended now, requirements have been elicited stemming from the divergent missions and day-to-day requirements of these data centers.

Following an ESA requirement, DREAM puts strong emphasis on the use of open, interoperable standards. Not only does this mandate the use of standards-conformant tools; in addition, DREAM actively contributes to advancing standardization by providing new specification components. Focus here is on the main OGC standard set supporting multi-dimensional raster data, the Web Coverage Service (WCS) suite. The GML Application Schema for Coverages (GMLCOV) establishes a common conceptual model of coverages (including raster data), while the WCS Core defines basic service components. WCS extensions add further functionality facets which a vendor may choose to implement. In DREAM, particular emphasis is on the data extraction and processing facilities, such as the Web Coverage Processing Service (WCPS) raster query language.

The rasdaman raster analytics engine, available for free on at http://www.rasdaman.org, provides the platform for developing the new OGC standards specifications, such as support for data formats like LIDAR and GML JPEG2000. Right now, rasdaman is already the OGC candidate reference implementation for WCS. Further, rasdaman has been included in the OSGeo Live set of particularly recommended open-source geo projects and as the only multi-dimensional raster database supporting PostgreSQL and PostGIS.

Latin American Industry Looking At Rasdaman and OGC Standards

July 2012. The markets of Latin America grow continuously. A main reason leading to this increasing economic importance of South American countries lies in their role as world-wide suppliers of oil, gas, and mineral resources. At the NRDM (Natural Resources Data Management) conference in Rio de Janeiro, representatives of industry and research discussed about challenges in exploration data management. In his presentation, Dr. Peter Baumann, managing director of rasdaman GmbH, convincingly argued the potential for cost savings by using open standards. His presentation titled “Geo images, sensors and Web Statistics: meet the Standards“ is available on request.

Multinational players like Shell, Chevron, Petrobras and Dell presented their view on data interoperability and integration. Especially exploration industry showed high interest in the multi-dimensional capabilities and the scalability of the Big Raster Analytics Server, rasdaman. The international professional audience attested rasdaman GmbH unanimously its leadership in geo technologies.

rasdaman newsletter 05/2012

Raster Engine rasdaman 8.3.1 released

The new rasdaman version 8.3.1 has passed the conformance test suite of the OGC WCS Core standard and is now a candidate reference implementation. As usual, rasdaman is available in an open-source and a proprietary variant. The open-source rasdaman community can be downloaded from www.rasdaman.org; additionally, it can be tested and installed via OSGeo Live (live.osgeo.org).

New features of the optimized version 8.3.1 include GDAL-based import through the rasgeo component which supports practically all relevant image formats. Further improvements address new platforms (OS-X), increased hyperspectral support, a web client toolkit (raswct) for rapid client development, integration with 52 North WPS, and general usability as well as numerous fixes. Landcare Research New Zealand have contributed, as well as NOAA people, St. Petersburg Polytechnic University, Ferrara University in Italy, Jacobs University, and rasdaman GmbH. This underlines how much of a community project rasdaman meantime has become. Currently the project continues with extending the OGC Web Coverage Service (WCS) with further specification modules, including their implementation in rasdaman along with conformance tests. This way, rasdaman has become an indispensable tool for the advancement of geo service standards.

German Innovation Award 2012 for Big Data Analytics

The research spin-off, rasdaman GmbH, has been awarded the INNOVATIONSPREIS-IT (Innovation Award IT) 2012 in the category “BEST OF 2012 Open Source”. The national jury thus acknowledged the world technology lead of the high-performance raster analytics engine, rasdaman (“raster data manager”), selected from over 2,500 applications.

The rasdaman technology enables fast and flexible access to multi-dimensional sensor, image, and statistics data of unlimited volume; examples include 1D time series, 2D satellite images, 3D x/y/t image timeseries and x/y/z geophysics data, as well as 4D climate data. Being domain independent, rasdaman equally well supports application domains in Earth, Space, and Life sciences, to name but a few. Among the distinguishing features is the capability of answering ad-hoc questions to perform filtering the desired data from large-scale archives and to process the extracted data into the shape users need them. This includes fusion of data coming from different locations and having different resolution, extent, and structure. The enabling idea is to combine standard database query languages with image processing languages, resulting in a novel, powerful means to access massive archives over the Web. Meantime this principle has found its way into international standards – the Open Geospatial Consortium (OGC) has crafted its raster query language, WCPS, along the rasdaman concepts. While rasdaman has been developed over 17 years into a stable, mature technology, international efforts have emerged with similar goals, most remarkably the US SciDB initiative. Under the patronage of German national commissioner for information technology, Cornelia Rogall-Grothe, and IBM Germany the INNOVATIONSPREIS-IT is in its ninth year of scouting IT innovations.

OSGeo recognizes Effort in Big Geospatial Analytics

Being selected by the Open Source Geospatial Foundation (OSGeo) to become part of their “OSGeo Live” demonstration DVD dignifies a project as being particularly topical, valuable, high-quality, and innovative. The rasdaman geo analytics engine has been added to OSGeo Live 5.5 as the only geo raster server capable of handling unlimited volumes of multi-dimensional “Big Data”.

OSGeo Live is a self-contained bootable DVD, USB thumb drive or Virtual Machine based on Xubuntu that allows to try a wide variety of open source geospatial software without install¬ing anything. It is composed entirely of free software, allowing it to be freely distributed, duplicated and passed around. OSGeo Live can be downloaded from live.osgeo.org and several mirror sites. “This rewarding acknowledgement of rasdaman in the geospatial Big Data world underlines the importance of going the next technology step, beyond mere map serving; geo analytics is indispensable in today’s flow of geospatial information”, comments Peter Baumann, principal architect of rasdaman and well-known editor of OGC raster service standards. In parallel, rasdaman has been given “incubation” status in OSGeo’s open-source geospatial software branding program. Being acknowledged by OSGeo means that not only the software itself, but the project overall has been thoroughly examined in all aspects. Criteria considered include geo relevance, code quality, documentation, strictly open-source code provenance, community, and project management. Every project gets assigned a voluntary mentor; for rasdaman, Bruce Bannerman from the Australian Bureau of Meteorology has kindly accepted this task.

OGC Adopts Enhanced Spatio-Temporal Coverage Service

The Open Geospatial Consortium (OGC) Web Coverage Service (WCS) Standard establishes an open, interoperable service interface for accessing, filtering, and processing of multi-dimensional spatio-temporal coverages, such as sensor time series, satellite imagery and image time series, and climate data. Following the complete overhaul of WCS in 2010, leading to version 2.0, now an enhanced corrigendum version 2.0.1 has been produced.

En detail, WCS actually comprises a modular suite of standards based on the common WCS Core 2.0.1 becoming available now. The GML Application Schema for Coverages defines OGC’s common coverage model, allowing coverages to freely be exchanged between services implementing different standards such as GML, SWE, WCS, and WPS. Further service extensions include the Web Coverage Processing Service (WCPS) standard which establishes a multi-dimensional raster query language. The Earth Observation Application Profile (EO-WCS) is a candidate standard which specializes WCS for the satellite imagery domain. “The new version satisfies stakeholder requests received and offers significantly improved functionality, flexibility, and usability”, says Peter Baumann, chair of the WCS Standards Working Group and editor of the specifications. Within the next few days, OGC will make new specification available on the OGC website where it can be found at www.opengeospatial.org/standards/wcs. The OGC is an international consortium of more than 435 companies, government agencies, research organizations, and universities participating in a consensus process to develop publicly available geospatial standards. OGC Standards support interoperable solutions that "geo-enable" the Web, wireless and location-based services, and mainstream IT. OGC Standards empower technology developers to make geospatial information and services accessible and useful with any application that needs to be geospatially enabled. Visit the OGC website at www.opengeospatial.org.