dhp-blogpost-standardimage

Workflow: INSPIRE Planned Land Use for the Trento Province

At the INSPIRE Conference 2013 HALE workshop in Firenze, we presented two case studies, where current published data was used to create INSPIRE-compliant data sets. This post explains how we created Planned Land Use data for the Province of Trento. A more detail version of this workflow is also available as a PDF, and you can see the accompanying slides on slideshare.

The data used for this example is the land use official plan of the Province of Trento, approved in February 2006, which consists of:

  1. a shapefile with the land use plan, including two updates, one approved with a regional decree of February 2013 and the other one was the previous one, approved in July 2011.
  2. a shapefile with the most current hydro-geomorphological risk zones.
  3. main official documentation (law text).

The INSPIRE PLU conceptual schema corresponds to a dataset that corresponds to a spatial planning document. Geographic information as well as the informative or descriptive parts contained in a spatial planning documents are taken into consideration in the LandUse data application schema.

Original Map of the Planned Land Use in the Province of Trento

Original Map of the Planned Land Use in the Province of Trento

Data Preparation

In most data harmonisation workflows, a data preparation step is required. In this case, this involved creating a HILUCS code mapping table and a resource for the official documentation.

In Trento, the PLU is classified with a local classification, since Inspire requires to use HILUCS classification for land use a reclassification is needed. A classification operation can be done directly in HALE or by loading a CSV file containing the correspondence between the original and the HILUCS classes. Regarding the PLU of the Province of Trento the following classification mapping has been used.

COD_TOT

DESCRIPTION

HILUCS CLASSIFICATION

1

Aree Residenziali 5_1_PermanentResidentialUse

2

Aree Produttive

2_SecondaryProduction

3

Aree Ricreative

3_4_4_OpenAirRecreationalArea

4

Aree Agricole

1_1_1_CommercialAgriculturalProduction

5

Improduttivo

6_3_1_LandAreasNotInOtherEconomicUse

6

Campeggi

5_3_OtherResidentialUse

7

Depuratori e Discariche

4_3_3_WasteTreatment

8

Aree Sciabili

3_4_3_SportsInfrastructure

9

Aree a bosco, prato e prateria alpina

1_2_Forestry

101

Strade di importanza primaria

4_1_1_RoadTransport

102

Ferrovie

4_1_2_RailwayTransport

103

Strade di importanza secondaria

4_1_1_RoadTransport

Another preliminary operation that would help us during the mapping process is the creation of a library of information and links for the documentation of the plan. These information are stored in a CSV file containing the basic properties requested in the Insprire schema for OfficialDocumentation, in particular:

  1. an identifier
  2. legislationCitation: reference to the document that contains the text of the regulation
  3. DocumentCitation: citation of scanned plans and structural drawings being sometimes georeferenced and sometimes not (raster images, vector drawings or scanned text)

The content of the CSV file for the use case of the PLU of the Province of Trento is:

ID documentCitation legislationCitation

Doc_1

http://pguap.provincia.tn.it/GIS/tiff/uso.html

http://www.delibere.provincia.tn.it/CercaSpecifica.asp?Modalita=Delibere&anno=2013&numero=228

Doc_2

http://www.delibere.provincia.tn.it/CercaSpecifica.asp?Modalita=Delibere&anno=2011&numero=1551

Doc_3

http://pguap.provincia.tn.it/pdf/Piano/0IndiceGenerale.pdf

The Alignment in HALE

The very first operation to do in HALE is to load the source schema, the source data and the target schema. This operation is very easy, select the correspondent element to import from the menu File → Import → …

HALE can load source schema from different sources and in particular from any GML/XML, shapefiles and CSV files. Considering the source data and the target schema there are different FeatureType that has to be considered in this example, in particular:

  1. ZoningElement: contains the information of the geometries of the Land Use
  2. SpatialPlan: contains all the information related to the official plan
  3. OfficialDocumentation: contains the links and reference for the available documentation
  4. SupplementaryRegulation: contains the information of additional regulation which supplement the zoning.

The first type to map is the LandUse shapefile which contains the basic information for both the ZoningElement and the SpatialPlan. All the mapping is based on the retype operation. The Retype function expresses that a source and a target type are semantically equal: for each instance of the source type, an instance of the target type is created. Property relations only take effect in the context of a type relation. First a type relation must be defined, then property relations between the involved types can be specified.

Initial HALE setup after loading source & target schema, source data

Initial HALE setup after loading source & target schema, source data

Mapping Process for PLU.ZoningElement

The operations to execute to map the original field of the shapefile to the properties of the types of the PLU target schema are basically rename, assign, date extraction, generation of unique ID and generation of the mandatory inspireID.

One of the most powerful operation in HALE is the possibility to integrate groovy script to map different properties.

Using the Groovy Script function to extract a date from a string

Using the Groovy Script function to extract a date from a string

Mapping Process for PLU.SpatialPlan

With the same source type it is then possible to map also the SpatialPlan. In this case, a merge operation has to be used to set a relation between the types. This function merges multiple instances of the source type into one instance of the target type based on one or more matching properties.

The basic operations for the mapping process are similar to the ones used with ZoningElement. HALE gives the possibility to generate the mandatory inspireID property which requires some more information on the origin of the data.

Using the Inspire Identifier Function in HALE

Using the Inspire Identifier Function in HALE

Link between ZoningElement and SpatialPlan

Since all land use polygons are part of one plan we can directly use the assign operation to assign plan → href in ZoningElement the name of the reference plan.

 

Mapping Process for PLU.OfficialDocumentation

A very important request in the Inspire schema for PLU is the link and reference to the official documentation of the plan. For the example only web links are used as listed in the CSV file. The basic operation to start with here is retype then other basic operation like rename, assign and generate ID are used to provide at least all the mandatory information.

Link between ZoningElement and OfficialDocumentation

Use a groovy script to assign the reference to the officialDocumentation of the features in ZoningElement, since ZoningElement contains two different update of the land use polygons, the official documentation will be different for the two and it is based on the date of the update.

Link between SpatialPlan and OfficialDocumentation

Since all the documents included in the OfficialDocumentation are related to the plan it is necessary to link all of them to SpatialPlan → officialDocumentation → href. In this case we have to create two other new instances of the same property officialDocumentation and assign to each of them the link to the documents .

 

Mapping Process for PLU.SupplementaryRegulation

The last information to be mapped is the additional regulation on hydro-geomorphological risk mapping. This information is stored in an other shapefile and the retype operations is used to map the contained elements to SupplementaryRegulation.

Link between SupplementaryRegulation and SpatialPlan

Since all risk zones polygons are part of one plan we can use directly assign operation to assign to plan → href in SupplementaryRegulation the name of the reference plan.

Link between SupplementaryRegulation and OfficialDocumentation

Since all risk zones polygons are part of the same update of the land use plan we can use directly assign operation to assign to officialDocument → href in SupplementaryRegulation the name of the reference documentation.

Final mapping for the SupplementaryRegulations type.

Final mapping for the SupplementaryRegulations type.

NOTE: the detailed description step by step of the alignment process is reported in the following presentation: http://www.slideshare.net/silli/2013-05-24hydrologisinspirepres

 

dhp-blogpost-standardimage

HALE 2.6.0 brings integration with FME

Last week, we published the new HALE 2.6.0 release. There is one particular feature that we have added in collaboration with Safe Software that I’d like to highlight: The integration of HALE and the CST engine as a GML Writer in the upcoming FME 2014 release. This integration reflects what I and other had been doing for a while now:

  1. Start FME and use the readers to import from a wide range of formats such as an Esri File Geodatabase,
  2. Perform operations that are only possible in FME, such as Geometry calculations,
  3. then write out a simple feature style output schema using a GML or SHP writer,
  4. then start HALE to map the data to the actual target schema, e.g. an INSPIRE Application Schema,
  5. and lastly, use HALE/CST to create the final GML product.

In other words, you can use HALE to either perform the mapping to the complex schema after all other data preparation in FME, or you create a subset of a complex data set before you go to FME. To make the new integrated workflow work, you’ll need FME 2014 and HALE 2.6.0 installed. These are the steps after installation of both tools:

1. Start FME. Add readers and transformers as needed.

2. Add the HALE GML/XML Writer.

fmehale01

3. Open the Parameters for the HALE GML/XML Writer and set at least a *.halex project file location and the path to the HALE executable:

fmehale02

4. Add a Feature Type to the Writer, e.g. by right-clicking on the canvas and selecting “Insert Feature Type…”. Either import a Feature type from any data set or, after creating the Feature type, manually define its schemas using the “User Attributes” tab in the Feature Type’s properties.

5. Connect the last transformer in the workspace to the writer as appropriate to have a complete workspace such as this one:

fmehale03

6. Set any necessary additional writer attributes, such as the number of features you want to use in HALE for interactive transformation. Also pay specific attention to the “Execution Mode” setting. “Schema Transformation” will just execute the project given in Step 3, while “Update Mapping” will launch the HALE UI to enable you to create/update a halex project. “Auto” switches between these two modes depending on whether a file is already present in the indicated halex location:

fmehale04

7. Execute the workbench, and see either HALE or CST get fired up in the final writing phase!

A press release giving more information is also available.

INSPIRE 2013: First day presentations – Code Lists, XQuery, GeoUML and other stuff

While I had a lot of meetings and booth duty, I managed to attend some presentations that turned out to be quite insightful.

First, I visited Michael Lutz’ talk entitled “INSPIRE Data Specifications – What’s New? What’s Next?”. A core change has been made in the way that code lists are handled. In the preparation of our HALE Workshop, we had already found that the way that Code Lists are used in Annex II and II has been changed from the way it was done before – instead of having a CodeList type with a namespace indicating from which list the value was taken, there is now a ReferenceType whose href attribute point to a value in a published code list. These code lists will be made accessible through registries, of which the first one is the official INSPIRE codelist registry. Code Lists can now also be extended in several ways by INSPIRE implementers, e.g, to add narrower values in a classification. Micheal then proceeded with explaining the work that has been started on the Registry provision. The INSPIRe Registry already provides themes, code lists and application schemas in various formats such as HTML, XML, Atom and JSON. We’ll adapt HALE so that it will be able to work with the new resources as soon as possible.

Michael Lutz INSPIRE Presentation

The next presentation that I saw was Christine Giger’s “Tips & Tricks for Spatial Data Harmonization”. She focused on exploring options for data harmonisation to satisfy INSPIRE and other standards, and suggested that XQuery could be a useful tool for schema transformation and other aspects of data harmonisation. XQuery builds on top of Xpath as a selection/filter language and on several elements of XSLT for the generation part. XQuery, as a mainstream IT tool, has quite good tool support, e.g. in Zorba. To cover spatial operations, an extension called ExPath Geo can be used. Christine showed examples how to use XQuery and summarized her assessment that typical problems in data transformation can be covered. Of course, authoring the XQuery (a functional, turing-complete language) statements is not trivial, and I am not yet aware of an interactive authoring environment. As such, it might be interesting to extend HALE’s XSLT/XPath support to also enable XQuery authoring.

Christine Giger INSPIRE Presentation

Giuseppe Pelagatti presented his group’s work in his talk titled “Application of The Geouml Tools for The Production and Validation of INSPIRE Datasets“. They built a catalogue viewer to be able to explore the INSPIRE schemas, in a very similar fashion to what HALE’s schema explorer provides. However, Alberto, Giuseppe and their colleagues based their tool directly on the UML model, which provides more information than the GML Application Schemas. It would be really nice to collaborate with them to maybe bring actual UML support back into HALE, after our earlier experiments with bringing in XMI were not very successful. The talk also explained work that the group did in the field of connecting INSPIRE with spatial database backends, essentially performing vertical mappings. Here I though it might also be worthwhile to collaborate, to bring DB support to HALE.

INSPIRE Presentation

The final talk I was able to attend was Astrid Feichtner’s “Testing of INSPIRE Datasets“. She explained the current state of work in a project at the Bavarian Surveying Authority to perform AAA and INSPIRE dataset conformance testing, which goes well beyond Service Testing and validation performed on an XML Schema basis. The project, where the implementation partner is Interactive Instruments, was quite impressive. OCL constraints from the original UML model are evaluated, e.g. testing allowed spatial relationships and verifying references. The current limitations of the project were mainly scalability and completeness of spatial tests. I thought that providing this tool as part of the GDI-DE testsuite or otherwise making it accessible to the community might be a very valuable prospect. One major caveat I have however is that all tools I know of that create INSPIRE Geodata are based on compliance with the current GML application schemata. As the information in the schemas is only a subset compared to what is in the UML model, these data sets will not comply to all aspects of the regulation. This wouldn’t need to be the case, even without moving on to XSD 1.1 there would be ways to include more information in INSPIRE XSDs.

INSPIRE 2013: Full Agenda for the HALE Workshop

We (Silvia Franceschi, Simon Templer and me) look forward to welcome you at the HALE workshop at INSPIRE conference. This post brings you the full agenda of the workshop and provides links to associated materials and software.

Resources:

Agenda:

  1. Introduction to the workshop and background presentation (Thorsten, 15 minutes)
  2. Basic HALE Mapping: Converting the INSPIRE Planned Land Use data for the Trento Region of Italy (Silvia, 40 minutes)
    1. Loading resources
    2. Analysing schemas and data
    3. The 3 R’s of basic mapping: Retype, rename, reclassify
    4. INSPIRE Mapping Functions
    5. Dealing with references
    6. Data export and usage
  3. Advanced HALE Mapping: Provide INSPIRE Hydrophysical Waters Data from MERIDIAN2 UK data (Simon, 20 minutes)
    1. Mapping inheritance
    2. Functions and their documentation
    3. Additional Export functions: HTML, XSLT, Excel (CSV/XLS)
    4. One more thing…
  4. Q&A, Discussion

You will get the resources at the workshop on a nice USB stick as well, but of course it is always a good idea to prepare to get the optimum from an event. See you in Florence on Monday, 24th of June!

HALE Workshop @ INSPIRE Conference

Visit beautiful Firenze in Italy and learn about how to interactively transform complex and large Geodata sets at our HALE Workshop!

In the workshop, we will demonstrate the steps of a high-quality data harmonisation process by showing how data can be transformed to comply to the INSPIRE Data Specification for Land Use, and how the published data can be used as valuable input to downstream activities. The software and data for the showcases will be made available to all participants of the workshop, in order to provide them with a real world hands-on experience.

Intended Audience and any Required Skills or Interests

This workshop addresses policymakers and decision makers that have an interest in powerful open source tools to support their harmonisation needs. It also addresses developers and scientists with an interest in the latest approaches for schema mapping and data transformation.

Get more information on the Workshop here.

Reserve the time slot: Monday, 24th of June, 11:00 to 12:30

dhp-blogpost-standardimage

FOSS4G Community Selection Candidates include a HALE Workshop!

FOSS4G, the largest GIS Open Source Conference, is doing a community selection process right now on the workshops that will take place at this year’s conference – with a deadline of 6th of June. Help us by checking out the selection of workshops there and if you would like to see more distribution of HALE or visit the workshop there, be sure to vote for the workshop! It’s pretty far down on the list with the following description:

Create and Use INSPIRE Harmonised Geodata with HALE

In the workshop, we will demonstrate how data can be transformed to comply to the INSPIRE Data Specification for some spatial data themes defined in Annex I, II and III, and how the published data can be used as valuable input to downstream activities.

More information on the workshop proposal is available here.