Thursday, 15 November 2012

Working with Google Maps API v3 and Geoserver WMS overlay of differing projection

With Google's web mapping API continuing to improve and their map display performance setting the standard, it makes sense for every GIS architect and developer to familiarise themselves with what is on offer.  Often the first problem in doing so, however, is finding a way to display your own data together with Google's base maps (well you need to make your investigation personal don't you?)

This post describes how to display WMS data from Geoserver as an overlay on top of  a Google base map using the Google Maps API v3. For added interest, the source data for the overlay is stored in the British Ordnance Survey's OSGB projection (EPSG:27700) within a SQL Server 2008 database.

I found two ways to overlay data from my Geoserver on top of a Google basemap using the Google Maps javascript API:
  • google.maps.ImageMapType for displaying OGC WMS data as tiled images.  This method may suit relatively static data sets that cover large areas. WMS layers may be tiled and cached for improved display performance using GeoWebCache.
  • google.maps.KmlLayer for displaying more dynamic (query driven) data covering smaller areas.
A note on projections
Google maps is happy receiving data in WGS84 projection (EPSG:4326) and displaying it in the Google Mercator projection (EPSG:3857, formerly 900913).  I found that life became much easier when I accepted that I needed to provide the Geoserver maps/kml in WGS84.  I also did a lot of investigation using the Proj4js library to formulate requests but produced some pretty odd looking maps (messy tile edges).  Serving the data in WGS84 greatly simplified things, bypassing the need to re-project coordinates within javascript and relying instead upon Geoserver's built in coordinate re-projection facilities as illustrated.

The javascript below is called on the page load event to populate a map div within a web page.  It creates a map centered on Lincolnshire and displays Google's roadmap as the base layer.

To add a custom overlay of census output areas from Geoserver on top of that we define a new Google ImageMapType which includes the getTileUrl function which will make one WMS request to our Geoserver for each map tile to display. This involves taking the coordinate of the top left hand corner of each map tile (as a screen coordinate) and calculating its bounding box coordinates as Latitude/Longitude. We can then build the individual WMS requests for adding the custom overlay to the map.

The code also demonstrates the use of a kmlLayer as an overlay.

//myMap.js
//javascript to create web map displaying Google base map
//with custom overlays from geoserver wms and kml

(function () {
  window.onload = function () {

    // Create a MapOptions object with the required properties for base map
    // Centered on Lincolnshire, UK
    var mapOptions = {
      zoom: 9,  
      center: new google.maps.LatLng(53.3567, -0.244), 
      mapTypeId: google.maps.MapTypeId.ROADMAP
    };

    // Create the base map 
    map = new google.maps.Map(document.getElementById('map'), mapOptions);

    //Define custom WMS layer for census output areas in WGS84
    var censusLayer =
     new google.maps.ImageMapType(
     {
       getTileUrl:
      function (coord, zoom) {
        // Compose URL for overlay tile

        var s = Math.pow(2, zoom);  
        var twidth = 256;
        var theight = 256;

        //latlng bounds of the 4 corners of the google tile
        //Note the coord passed in represents the top left hand (NW) corner of the tile.
        var gBl = map.getProjection().fromPointToLatLng(
          new google.maps.Point(coord.x * twidth / s, (coord.y + 1) * theight / s)); // bottom left / SW
        var gTr = map.getProjection().fromPointToLatLng(
          new google.maps.Point((coord.x + 1) * twidth / s, coord.y * theight / s)); // top right / NE

        // Bounding box coords for tile in WMS pre-1.3 format (x,y)
        var bbox = gBl.lng() + "," + gBl.lat() + "," + gTr.lng() + "," + gTr.lat();

        //base WMS URL
        var url = "http://mywebserver/geoserver/lincs/wms?";

        url += "&service=WMS";           //WMS service
        url += "&version=1.1.0";         //WMS version 
        url += "&request=GetMap";        //WMS operation
        url += "&layers=census_oa_2001"; //WMS layers to draw
        url += "&styles=";               //use default style
        url += "&format=image/png";      //image format
        url += "&TRANSPARENT=TRUE";      //only draw areas where we have data
        url += "&srs=EPSG:4326";         //projection WGS84
        url += "&bbox=" + bbox;          //set bounding box for tile
        url += "&width=256";             //tile size used by google
        url += "&height=256";
        //url += "&tiled=true";

        return url;                 //return WMS URL for the tile  
      }, //getTileURL

      tileSize: new google.maps.Size(256, 256),
      opacity: 0.85,
      isPng: true
     });

    // add WMS layer to map
    // google maps will end up calling the getTileURL for each tile in the map view
    map.overlayMapTypes.push(censusLayer);

    // define kml layer
    // set base WMS URL for kml request
    var kmlUrl = "http://mywebserver/geoserver/lincs/wms/reflect?";

    kmlUrl += "&version=1.1.0";              //WMS version 
    kmlUrl += "&layers=study_area_poly";     //WMS layers
    kmlUrl += "&viewparams=analysis_id:177"; //analysis_id parameter for query filter
    kmlUrl += "&styles=";
    kmlUrl += "&format=kml";

    var kmlOptions = {
      map: map,              // the map that the kml layer will be added to
      url: kmlUrl,           // the url for the kml data
      preserveViewport: true // do not adjust map location and zoom
    }; 

    // add the kml layer to the map
    studyLayer = new google.maps.KmlLayer(kmlOptions);

  };

})();

The above WMS request includes parameters to return the overlay tiles in WGS84 projection and to make areas without data to be completely transparent.  The viewparams parameter used by the KML example allows for dynamic layer creation using parameterised sql views from Geoserver.


The image illustrates the Google roads basemap with census areas (grey polygons) overlain as WMS tiles and kml data (green grid) overlain as feature data.

One of the nice features of using kml layers is that the Google Maps API automatically provides support for feature identification via info windows.

The actual web page html is provided below:

<!DOCTYPE html />
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Google Map with Geoserver WMS Overlay</title>
    <link type="text/css" href="Style/maps.css" rel="Stylesheet" media="all" />
    <script type="text/javascript" src="http://maps.google.com/maps/api/js?sensor=false"></script>
    <script src="Scripts/myMap.js" type="text/javascript"></script>
  </head>
  <body>
    <h1>Google Map with Geoserver WMS and KML Overlay</h1>
    <div id="map"></div>
  </body>
</html>

You might question the sanity of producing web maps that require reprojection on-the-fly and I would have to agree; anything that introduces processing and delay into web mapping is to be avoided, but sometimes you just have to go with what is available.

It would be very nice to see the Google Maps API introduce support for other OGC standards such as WFS; this would enable the creation of more interactive maps and improved support for dynamic mapping of the data, especially where we are dealing with feature data that covers larger areas.

Thanks to Opinionated Geek for the html encoder, needed to make the html display correctly with Alex Gorbatchev's syntax highlighter.

Saturday, 15 January 2011

Using Powershell to load Ordnance Survey Code-Point CSV data

Another off-topic jaunt, but I have been playing with Microsoft's Powershell recently and quite enjoyed it.

Powershell is an interactive scripting language with strong ties to the .Net framework - meaning that .Net code can be accessed whilst scripting.  I found it easy to get going with and used it to import code-point data from 120 CSV files into a SQL Server database.

The script is based on Doogal Bell's excellent blog post in which he provides a C# program to import code-point.  I borrowed Doogal's .Net Coordinates library which was downloaded and accessed directly from within the script, simply by using the Add-Type command to locate the DLL file.  This library enabled me to easily convert the OSGB coordinates into latitude and longitude.

The script is provided below and could be used by a simple cut and paste and changing the first few lines to reference your own environment.

# Add reference to DotNetCoords
Add-Type -path "C:\Users\GrahaMorgan\Software\.net code\DotNetCoords.dll"

# Specify location of codepoint csv files
$dataLocation = "C:\Data\Ordnance Survey\codepoint_gb\Code-Point Open\Data"

# set up database connection
$connectionString = "Server=GRAHAMORGAN-PC\Spatial1; " +
                    "Database=pafdb; " +
                    "Integrated Security=SSPI;"
          
# Connect to database and open it
$sqlconnection = New-Object System.Data.SqlClient.SqlConnection $connectionString
$sqlconnection.Open()

$sqlcmd = New-Object System.Data.SqlClient.SqlCommand
$sqlcmd.Connection = $sqlconnection 

# get hold of all csv files
$files = get-childitem $datalocation *.csv | foreach-object {$_.name}

foreach ($file in $files)
{
    $filename = join-path $datalocation $file
    write-output("Processing " + $filename)

    $data = import-csv $filename -header ("postcode","B","C","D","E","F","G","H","I","J","easting","northing","M","N","O","P","Q","R","S")
    foreach ($row in $data)
    {  
        $OSRef = new-object DotNetCoords.OSRef($row.easting,$row.northing)
        $LatLng = $OSRef.ToLatLng();
        $latLng.ToWGS84();
              
        $sql = "insert into codepoint_coords (postcode, longitude, latitude, coordinates) " +
               "VALUES ('{0}', {1},{2}, geography::STPointFromText('POINT({1} {2})', 4326))" -f $row.postcode, $latLng.longitude, $latLng.latitude
         
        # Execute sql insert      
        $sqlcmd.CommandText = $sql
        $res = $sqlcmd.ExecuteNonQuery()
    }   
}

$sqlconnection.close()

Microsoft have made Powershell available from within SQL Server and it will be interesting to see what people do with it.  I guess one obvious use would be to take advantage of the database import tools which would probably provide much faster performance.  On the other hand, the flexibility of the script to manipulate the data, such as changing coordinate system is pretty nice.

Note syntax highlighting via Alex Gorbatchev's open-source SyntaxHighlighter; instructions on MLA Wire.

Tuesday, 5 January 2010

Fun with Manifold GIS

This post may not fall strictly within the Enterprise GIS topic but I have just completed my first project with Manifold GIS . This was a little piece of work to create a series of maps for a county sports partnership in the UK; we chose Manifold purely based upon price, and curiosity...

First off - Manifold was upto the job and the client was delighted with the results.  As someone who has used GIS technologies of all shapes and sizes for almost 20 years I have to admit that I was rather stumped by the user interface but, with patience, I was able to find all of the functionality that I needed.

Highlights for me were:
  • A great price, at just USD$400
  • Windows based client application 
  • Is able to read and write a variety of file and database formats
  • Includes a host of analysis functions
  • Supports several Windows scripting/programming languages
Limitations for my purpose were:
  • Was not intuitive for me to use
  • Help documentation not always easy to find and used unfamiliar terms
  • Cartographic facilities were not bad but didn't provide fine grained control
  • Some processing steps went into long loops and failed
  • Exports to PDF but not to GeoPDF yet
I think the Manifold team have done a great job producing this functionality at the price.  Next I want to explore Manifold further to get a better understanding of the customisation facilities and scalability for larger batched processing.

Monday, 14 December 2009

Enterprise GIS is not a Package

It seems that some organisations continue to regard GIS as a commodity application package: something that can purchased, plugged in, and voila - GIS for the business.  I started thinking about how these organisations may have come to adopt this view.

Perhaps there are 3 broad views of GIS:
  • Desktop application
  • Standalone system
  • Enterprise capability
Looking back, GIS has traditionally been provided by 'fat client' desktop applications such as MapInfo, ArcGIS, GeoMedia, Manifold, gvSIG, etc. (there is a nice list of GIS software on Wikipedia).  These applications have in general been geared towards the use of proprietary file based formats such as shp, tab, e00, dwg, etc.  As GIS grew from project based applications to departmental systems, they evolved to employ multi-user databases for storage, but still maintained the two-tier, fat-client architecture.  GIS vendors marketed their GIS Systems as replacements to the drawing office paper and mylar maps i.e. as standalone, seamless map repositories.

I think those original standalone GIS systems can be thought of as the first beasties that crawled out of the sea and into a new world of land and air.  They took the first step from paper to the digital world - but that was just the beginning.  The data has evolved to take advantage of its new surroundings; we realised that we have many other datasets that could be indexed geographically, and many other areas of the business that could benefit from the power of spatial technologies to integrate, analyse and visualise data.  GIS was no longer just the digital drawing office, it provided capabilities that were sought after across the whole organisation.

This evolution has been driven by a desire to speed up business workflows, enhance services, reduce costs and improve decision making.  These drivers have required that location should be captured and used during every task in a workflow where it helps - and that turns out to be pretty much everywhere.With this realisation it is clear that GIS is no longer a standalone system - it is a set of enterprise-wide capabilities that need to be accessible to all business systems.

I was rather puzzled recently when a UK Police force went to the market for a 'GIS System'.  In talking with the procurement officer, he appeared to regard this as any other commodity purchase; something that could be acquired and plugged in within 30 days to replace their existing GIS - software vendors only need apply.  Well good luck to them.  You can easily imagine how central to police operations geography is (CBS's TV show 'The District' featured GIS scenarios prominently). Buying a standalone GIS is not going to provide them with the benefits that they will want and expect, but until organisations evolve their definition of GIS then I guess some procurement folks are going to struggle to buy what is really needed.

Wednesday, 9 December 2009

Enterprise Architecture and GIS

Enterprise Architecture is a continuous practice of aligning the resources of a business with the objectives of that business.  At a broad level Enterprise Architecture considers:
  • Business Architecture
    Commercial model, process model, organisation mode
  • Technology Architecture
    Application mode, infrastructure model
  • Data Architecture
    Data models, information products
GIS is very much relevant to Enterprise Architecture because it can both enable and hinder each of these tiers.  Business processes have traditionally needed to work around limitations imposed by GIS technologies because the transfer of spatial data between systems, departments, and hence successive steps in a workflow have been so problematic.  This has been caused primarily by limitations within the technology and data tiers. 

The concept of a 'GIS System' is starting to evaporate as spatial facilities emerge within other components.  Databases (e.g. Oracle, SQL Server, MySQL, Postgis) and data formats (e.g. GML, GeoPDF) are rapidly maturing and are available for use by a much wider community than the traditional 'hard core GISers' and thus we are starting to see components of the IT stack become spatially-aware.  This is the natural evolutionary path for GIS and is the model that we should be designing for.  As these technologies mature I think  it is important that the Enterprise Architects and IT Architects are aware of the current and emerging opportunities, and limitations, afforded by spatial technologies.   

Tuesday, 29 September 2009

AGI 2009 Geocommunity Conference - Mainstream? Not yet.

A good AGI GeoCommunity conference for sure with some thought provoking presentations.  There were lots of familar faces and it was good to catch up over a beer or two.  I was wondering who would be there this year and what the community would consider the burning questions of the age to be...

It wasn't as I had expected.  Steven Feldman, the conference chair, opened the first day with a solid introduction, explaining that the central theme would be 'What' people are doing with GIS, and hoping that over the next year we would be considering more 'Why' questions.  This year's conference would not really consider the 'How' as GIS has now merged with the mainstream, and so presumably then, implementation problems are a thing of the past.  At least that is how I heard it and I must admit that this sentiment does not ring true to me. Not yet.

In my own travels as a Geospatial Solution Architect I haven't noticed GIS implementation challenges lessen very much - sure some of the standards are beginning to take hold but the problem areas of real multi-user concurrent data editing, application interoperability, data integration, data licensing, and GIS within enterprise architecture are still very much significant challenges.  I would have loved to see an implementation stream cover progress in these areas.

The conference was not organised along the lines of the AGI Special Interest Groups which perhaps in itself indicates that we may need to review the topic areas and roles of the SIGs to ensure that they remain relevant to the community. As someone that feels that technical developments are of great interest I enjoyed the Geoweb stream which addressed some relevant (or potentially relevant) technical developments, principally in the area of mashups.  I would love to hear more about developments in data storage, server based services (and issues of scalability and availability), systems and data integration - technical issues relevant to enterprise deployment of GIS capabilities.

The AGI Northern Group has organised a follow up conference (Where2.0Now?) in Harrogate on 10th November on the subject of Web2.0 and disruptive technologies.  The AGI Technical SIG is also in the process of organising a conference for early next year on the subject of Mobile GIS.  I look forward to both of these events and to hear more of the technical developments effecting our industry.

GIS as a member of the mainstream?  Based upon the content and attendance of this conference we can't possibly argue that GIS is mainstream yet.  Sure, many more people use spatial data and technologies daily in the form of SatNav and online web maps but the IT mainstream was not at the conference.  There were no stands from the mainstream IT players such as IBM, Microsoft, Oracle, FileNet, Google, Informatica, Accenture, Logica, Dell, Sun, Embarcodero, Adobe, etc, etc. in the small exhibit hall.  When we start to see these vendors and integrators showing up in numbers then we will know that GIS has arrived.  I am not so sure what we will all do after that though!

Thanks to the conference organizers.

Sunday, 18 May 2008

What is this Enterprise GIS Thing?

I suppose if I am going to blog on about this topic then I should really lay out what it is I am talking about, so here goes. It is not going to be straight forward though but hopefully, as my ideas develop, I’ll be able to refine this over time.

“Enterprise GIS” is one of those terms that seems to have been around for a long time, at least since the late 1990’s but it has always had more to do with marketing hype than any commonly understood concept. During my time at Convergent Group in the US we certainly toted many solutions as being Enterprise GIS (and some were pretty cool) but today I am taking a different slant on the notion. To begin with I don’t believe that there is such a thing as “an Enterprise GIS” – that is, there is no single application or architectural component that provides all the GIS capabilities that a large organization could need.

In my experience the term is most commonly used for large spatial data repositories; hubs, warehouses and clearinghouses are often labelled as Enterprise GIS. These central stores of geospatial data can vary enormously in their sophistication, ranging from simple file shares that act as dumping grounds for file based formats, to spatially enabled databases fed by automated feeds from business applications. Repositories are often accompanied by metadata catalogues to assist potential users find and evaluate data – but in the end these are still read-only systems and only provide a subset of capabilities.

To me, the definition of “Enterprise GIS” is a set of capabilities (or services) that are baked into the software infrastructure (the ‘stack’) in order to be made available to any application that needs to use them. What are those capabilities? Well basically they provide the ability to work with spatial data i.e. to capture, transform, transfer, store, analyse, visualise, maintain, validate, manage, secure and distribute spatial data. ‘Baking them into the stack’ means that they are made available to any business application without any dependency on proprietary ‘fat’ desktop client software.

Thus it is not possible to purchase an Enterprise GIS; these capabilities need to be architected and assembled using a blend of platform software, supplemented by specific technology components from the specialist GIS houses. This involves careful selection of appropriate standards and policies to support data interoperability between applications and to enable reliable ‘Create Read Update Delete’ access to spatial data via any channel including desktop, web, sensor, mobile, EAI, etc.

Enterprise-wide GIS capabilities need to exhibit specific ‘enterprise class qualities’ i.e. they must be deployed in a server environment and possess the following qualities:

  • Scalable – they can grow to support volume and performance demands for current and future (unforeseen) needs;
  • Reliable – can operate in an unattended fashion to produce consistent and predictable results;
  • Available – can be configured to support High Availability e.g. support clustering and fail over;
  • Flexible – configuration can be changed to support evolving needs of the business with relative ease, i.e. without significant downtime or impact on business operations;
  • Secure – all access to data functions must be securable in line with corporate policies; includes support for directory services, delegation and encryption;
  • Measurable – it must be possible to monitor all aspects of the operation of these capabilities such that administrators can easily determine that they are functioning correctly and to support problem identification and resolution using standard tools (e.g. HP OpenView);
  • Standards-based – In order to support use by many applications, these capabilities need to be accessible via open standard protocols and formats;
  • Discoverable – in future posts I’ll explore this idea more, but the crux is that as we build a set of commonly available geospatial services, they will need to be described, and those descriptions made in an accessible fashion.

This doesn’t sound like GIS today, and in 20 years time may be redundant anyway as the capabilities are increasingly adopted by the major platform vendors. But at the present time I believe that providing spatial capabilities within the common platform is still a highly desirable and achievable vision.