Large GPS Track Logs

I had the need to display GPS tracks in DeepEarth. GPS tracks can contain thousands of points. DeepEarth has three update modes called ElementUpdate, PanOnlyUpdate and TransformUpdate for showing features such as tracks:

• ElementUpdate recalculates the point positions on every map movement. This produces an accurate track display but gets slower as the number of points increases.
• PanOnlyUpdate recalculates point positions during panning and hides features while zooming. Not too useful for me and didn’t seem to show anything anyway.
• TransformUpdate draws the tracks to the map once then scales and pans the vector graphic in synchronization with the map. This makes it very fast. Sadly the scaling code is flawed. Lines disappear as you zoom in and sections of the tracks become distorted, almost looking like calligraphy.

I wasted many evenings trying to get the scaling in TransformUpdate mode working before giving up. I then turned my attention back to the ElementUpdate mode to see where the bottleneck is.

The LogicalToPixel Function

LineStringControl objects represent the GPS tracks and they draw as lines using multiple points. When points are added to a LineStringControl they are transformed from longitude and latitude into logical coordinates. This is an intermediate coordinate system that can easily be converted into pixel coordinates later.

When the map is panned or zoomed the CoordTransform.LogicalToPixel function is called for every point in line. This function uses some simple mathematics to work out where the point should be on the map in terms of pixels. This is the bottleneck.

A single zoom or pan operation can result in this function being called several times. If a GPS track has 7,000 points that is a lot of calls. My testing showed that even the very first mathematical operation caused considerable slow down. Overall this function had several problems:

• Using 64-bit division
• Creating three Point objects and then discarding two for each call
• Using Math.Log and Math.Pow functions

Optimizing LogicalToPixel

The first step was to separate out the mathematical operations into two groups:

• Those that can be executed once per movement of the map
• Those that have to be executed once per point

I split the function into two new functions, LogicalToPixelPrepare and LogicalToPixelPerform to allow the two groups to be called separately.

Next instead of creating two temporary Point objects and then throwing them away, two global (to the class) Point objects are created and continually reused. This eliminates any overhead of object creation.

Finally I saw that I could eliminate both Math.Log and Math.Pow, as they were not necessary.

The last step was to modify the GeometryLayer.UpdatePathData function to call LogicalToPixelPrepare once and then LogicalToPixelPerform for every point. This reduces the transformation code needed for every point down to four multiplications and two additions.

Conclusion & Download

By finding the bottleneck and looking at way to optimize it, I was able to gain a significant speed increase for my 7,000 point test GPS track. I think the performance is close to that which could be obtained from the TransformUpdate mode, but without the scaling flaws found in that mode.

Download a patch that can be applied to the Silverlight 4 branch of DeepEarth (currently at revision 49789).

www.youtube.com/watch?v=22MOWEAzmoY

This article assumes you already have the Flex 3 SDK and FlashDevelop 3 installed and working. See my post from two years ago for help.

Compilation Options

There are two ways OpenScales can be compiled. Firstly the source code can be included with your application code. This approach is good for debugging OpenScales and learning how it works. My understanding of the OpenScales LGPL license is that this will make your application also licensed as LGPL (or a compatible license). However I’m not a lawyer so if that aspect is important to you then you will need to take a closer look.

The second way to compile OpenScales is as a library (SWC file). This library can then be linked with your application to produce the final Flash file.

Compiling OpenScales Into a Library

1. Download and install the ExportSWC plugin for Flash Develop.

2. Create a new Flex 3 project with no package name.

3. Check out OpenScales using Subversion into the folder above the project folder. Note that if you use a different location the paths in the rest of this tutorial will need to be adjusted accordingly. I recommend TortoiseSVN if using Windows.

4. Go to Project -> Properties -> Compiler Options. Click on the Additional Compiler Options field and then the small “…” button.

5. Enter:

-namespace http://openscales.org ..\OpenScales-1.1\src\openscales-fx\src\main\flex\META-INF\manifest.xml

Note the hyphen at the start of the line.

6. Click on the Classpaths tab and add the following Classpaths:

..\OpenScales-1.1\src\openscales-core\src\main\flex
..\OpenScales-1.1\src\openscales-fx\src\main\flex
..\OpenScales-1.1\src\openscales-proj4as\src\main\flex
C:\Program Files\FlexSDK\frameworks\libs
C:\Program Files\FlexSDK\frameworks\locale\en_US

then close the dialog window. Note if you copied the Flex 3 SDK to a different location then choose the correct location when adding the libs and locale classpaths.

7. Download GTween, extract the zip file and copy GTween_xx_xx.swc into the lib folder for the project.

8. In the Project tree view right click on the GTween SWC file and choose “Add To Library”. The name will turn blue.

9. In the Project tree expand “libs” and “locale”, right-click on each SWC file in turn and choose “Add To Library”. The names will turn blue.

10. Click on the Build SWC toolbar button to generate the SWC file.

You may get two warnings for the OverviewMap.mxml file, which causes the build to fail. I think this is a bug in FlashDevelop because warnings shouldn’t cause the build to fail, only errors should. However the warnings can be disabled in the compiler options. I don’t know for sure if the Overview Map control is usable because of this.

The generated SWC file will be in the bin subfolder. In FlashDevelop create a new Flex 3 project and copy the OpenScales SWC file into the lib folder, then add it to the library by right-clicking on the name in the Project tree and choosing “Add To Library”. OpenScales can now be used.

Compiling Application Code With OpenScales Code

1. Create a new Flex 3 project with a suitable package name.

2. Check out OpenScales using Subversion into the folder above the project folder. Note that if you use a different location the paths in the rest of this tutorial will need to be adjusted accordingly. I recommend TortoiseSVN if using Windows.

3. Go to Project -> Properties -> Compiler Options. Click on the Additional Compiler Options field and then the small “…” button.

4. Enter:

-namespace http://openscales.org ..\OpenScales-1.1\src\openscales-fx\src\main\flex\META-INF\manifest.xml

Note the hyphen at the start of the line.

5. Click on the Classpaths tab and add the following Classpaths:

..\OpenScales-1.1\src\openscales-core\src\main\flex
..\OpenScales-1.1\src\openscales-fx\src\main\flex
..\OpenScales-1.1\src\openscales-proj4as\src\main\flex


then close the dialog window.

6. Download GTween, extract the zip file and copy GTween_xx_xx.swc into the lib folder for the project.

7. In the Project tree view right click on the GTween SWC file and choose “Add To Library”. The name will turn blue.

Now you can add OpenScales controls to Main.mxml and build a Flex/ActionScript 3 project.

It is possible to add the GTween source code to the project instead of the SWC if needed.

You may get two warnings for the OverviewMap.mxml file, which causes the build to fail. I think this is a bug in FlashDevelop because warnings shouldn’t cause the build to fail, only errors should. However the warnings can be disabled in the compiler options. I don’t know for sure if the Overview Map control is usable because of this.

For completeness my MXML and ActionScript test code follows. Note that I avoided using MXML to create any controls, instead preferring to use ActionScript. This is just a personal preference.

Example Main.mxml file:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml"
xmlns:os="http://openscales.org" layout="horizontal" horizontalAlign="left"
backgroundColor="#FFFFFF" backgroundGradientColors="[#FFFFFF, #FFFFFF]"
paddingTop="0" paddingLeft="0" paddingRight="0" paddingBottom="0"
creationComplete="OSMap.main();">
</mx:Application>

Example OSMap.as file:

package com.britishideas
{
  import mx.controls.*;
  import mx.core.*;
  import mx.events.FlexEvent;
  import flash.events.*;
  import flash.utils.*;
  import flash.display.Sprite;
  import org.openscales.fx.FxMap;
  import org.openscales.core.Map;
  import org.openscales.core.layer.osm.Mapnik;
  import org.openscales.core.handler.mouse.DragHandler;
  import org.openscales.core.handler.mouse.WheelHandler;
  import org.openscales.core.control.PanZoomBar;

  public class OSMap
  {
    private static var map:FxMap;
    private static var mxmlApp:Application;
    private static var osmap:Map;

    public function OSMap()
    {
    }

    public static function main():void
    {
      mxmlApp = Application(Application.application);
      map = new FxMap();
      map.width = 840;
      map.height = 840;
      map.zoom = 10;
      mxmlApp.addChild(map);
      map.addEventListener(FlexEvent.CREATION_COMPLETE, MapCreated);
    }

    private static function MapCreated(e:FlexEvent):void
    {
      osmap = map.map;
      var MapnikLayer:Mapnik = new Mapnik("Mapnik", true);
      osmap.addLayer(MapnikLayer);
      var dragHandler:DragHandler = new DragHandler(osmap, true);
      var wheelHandler:WheelHandler = new WheelHandler(osmap, true);
      osmap.zoom = 10;
      osmap.addControl(new PanZoomBar());
    }
  }
}

Map Scales

A map scale looks something like 1:1000. This means that for every 1 inch on the map there are 1000 inches in the real world. The units don’t matter, for example it also means that 1 meter on the map is 1000 meters in the real world. The value 1000 shown in this example is called the Scale Denominator.

Maps (in the context of this discussion) are generated using pixels. This is true even if printing because ultimately the printer has to print the pixels onto the paper (assuming a raster output). At low resolutions the pixels will be easily seen. Not so at high resolutions.

It is therefore useful for us to know the size of a pixel in meters. If we know this then we can work out the map scale and set the map scale.

Universal Transverse Mercator

The world is round and not flat (no comments on this please!) and therefore we need a way of converting a section of it onto a flat piece of paper. The method of converting a curved section of the world to a flat representation is called the projection. There are many different projections to choose from and software available to convert data using the different projections. Universal Transverse Mercator (UTM) is a useful projection because it’s units are in meters. Typical map data uses latitude and longitude in degrees, which are difficult to use in calculations. However meters are much easier to work with.

The first step therefore is to get the bounding box of your map in degrees and convert it to meters in UTM. Mapnik can also do this for you using the projection class.

A bounding box is defined by the two coordinates that locate two opposite corners of the rectangle which defines the area in the map. The UTM zone is not needed, and we will assume the map area fits within a single UTM zone.

Maps For Screens

We assume that a pixel on a screen is 0.28mm on each side. This may or may not be true in reality, but it’s called the “standardized rendering pixel size” and is defined in the OpenGIS Styled Layer Descriptor (SLD) Implementation Specification. This assumption is made because software typically doesn’t know the pixel size on a screen and it’s a reasonable assumption. If the actual pixel size is known then it should be used. 0.28mm = 0.00028 m

We know the width of the area we wish to draw in meters from the UTM bounding box. We will call this map_realwidth_m. We also know the width of the map in pixels, because we choose that based on our requirements for the map. For example if the map is to go on a web page then the maximum width is probably something like 800px. We will call that image_width_px. Therefore:

scale_denominator = map_realwidth_m / (image_width_px x 0.00028)

The scale denominator is simply the ratio of the real world area width to the map width.

To summarize, for a bounding box in UTM coordinates and a given map width in pixels you can now calculate the scale of the map. However by rearranging the formula a UTM bounding box width can be calculated from a specific map scale and image width in pixels. By choosing a coordinate for the center of the map or one corner the UTM bounding box can be fixed.

Here is an example.  The width of the UTM bounding box is 1600 meters. The image width is 800px. Therefore:

scale_denominator = 1600 / (800 x 0.00028) = 7143.
Map scale is 1:7143.

For a map scale of 1:2000 and an image width of 800px, the UTM bounding box width will be:

map_realwidth_m = 2000 x (800 x 0.00028) = 448 meters

Maps For Printing

Generating maps for printing is the same as for a screen except the resolution is different. For example a pixel size of 0.00028m corresponds to a resolution of approximately 90.7 Pixels Per Inch (PPI). This is a relatively low resolution and will likely appear quite poor when printed. Printing has a much better resolution than a screen. For example printing an 8″ x 10″ map might need 300 PPI to get enough detail. Large posters might be printed at 150 PPI. Note that sometimes PPI is referred to as Dots Per Inch (DPI) although DPI is specifically related to printing whereas PPI is more generic.

The first step in creating a map for printing is to determine the PPI to use. This is defined by the size of the print and the printer. For the examples that follow we will use 300 PPI, but any value can be used.

Next we need to work out the pixel size for the resolution.

300 PPI = 1 / 300 inches per pixel. 1/300 x 25.4mm/inch = 0.0846mm = 0.000085 meters

Now we need to work out the width of the map in pixels. For 300 PPI and a width of 7.5 inches we get:

image_width_px = 300 x 7.5 = 2250px

We can now put these values into the map scale formula to calculate the scale from the UTM bounding box, or calculate the UTM bounding box width from the scale. For a UTM bounding box that is 1600 meters wide:

scale_denominator = 1600 / (2250 x 0.000085) = 8366
Map scale is 1:8366

XML Stylesheet

The Mapnik XML stylesheet tells Mapnik what to draw for different map scales. The rules are enabled by specifying a maximum and minimum scale denominator.

When creating maps for printing it is important to remember that internally Mapnik is assuming the map is for the screen and therefore has a different scale denominator. To get the scale denominator which Mapnik thinks is being used:

# zoom to bounding box then…
internal_scale_denom = mymap.scale_denominator()

Use this value when constructing the rules for the XML stylesheet.

Notes

If you tell Mapnik to render a specific UTM bounding box to an image of a specific dimensions then it will expand the bounding box in one direction so it has the same proportions as the image to be created. This can change the width of the UTM bounding box and therefore the scale calculation. The solution is to base any scale calculations off the actual bounding box. Mapnik can give you the values by calling the envelope() function on the map object.

The scale is an approximation and might only be valid for the center of the map. The larger the area represented in the map the more distortion there will be and therefore less accuracy when using the scale.

OpenStreetMap uses zoom levels numbered one to 18. This file shows the relationship between the zoom levels and the scale denominators.

In the 100 or so houses along this beach are the homes of some of the most famous people in Hollywood. Under the California Constitution (Article X, Section 4) the public is allowed access to certain areas of beach, but the rich and famous apparently want to keep it all for themselves. They even bulldozed the sand to protect their homes but ruined the public section, before being made to undo the damage.

This beach has two public access walkways, but they can be hard to find. They are located between houses and are narrow. According to some reports the local residents try to obsure the locations and put up false signs to deter people.

Last weekend (August 2009) I went to the beach armed with my GPS unit. I recorded the western public access and I have now added it to OpenStreetMap. Within a few minutes it was rendered (showing the location, steps and the gate).

Now everyone will be able to put this onto their maps and GPS units, easily find this public access point and enjoy free parking and a quiet beach!

Surprisingly a search turned up very few options.  There is, however, a service that already does everything I want, and it’s called Geo.ly. So why not use it I hear you ask? Well there are two large problems. Firstly the twitter interface doesn’t seem to be working right now and appears to have reliability problems. Secondly about 20% of the time the service will give you a URL to a map that doesn’t work – all you see is a blank page.

After a bit more fruitless searching I sat down and wrote my own PHP code that pulls everything together. Here is how it works.

There is a free text messaging service called TextMarks that converts text messaging to/from HTTP requests and responses. All that’s needed to send and receive text messages is to write a PHP (or CGI) script and put it on a web server somewhere. I tried it and it really is a piece of cake.

Next was to convert the text description of a location into latitude and longitude. For this Google provides a free geocoding system, providing that the results are shown on a Google map. After a bit more work I had this working and it can accept a wide variety of input descriptions. For example the ones I listed above plus things like “Grand Canyon National Park”. Perfect.

The final piece of the puzzle was to automatically post the tweet with the location of the Google map using the Twitter API. I found that Twitter automatically shortens URLs in tweets using the bit.ly service.

So now I can send a text message with a rough description of my location, the location is converted to coordinates, a tweet is generated with a link to the map and a confirmation text message is sent back to my phone. This typically takes about 15 seconds. Nice!

Below is the PHP script. Use at your own risk. To install:

1. Copy to somewhere on your webserver
2. Go to TextMarks and create a new TextMark for your service
3. Create a TextMarks account and edit the configuration. Disable messaging and any public options for your TextMark – after all you don’t want other people posting locations to your twitter account.
4. Edit the script and enter your Twitter username and password. Also enter your Google Maps API key (which is free).

Note that there is no authentication used in the script. This is because the script is not linked to, uses an obscure name (not twitter.php, which is just an example name), and the TextMark is private. However, if desired, the TextMarks service can pass the phone number to the script allowing for only specific phones to use the system.

A public version would, of course, need more input checking, user authentication, etc.

<?php
// script to take a location description from a text message,
// generate a URL to a map and then tweet the URL
// the result is returned as a text message



// (C) Copyright Andrew Ayre, 2009
// andy at britishideas dot com



// call from TextMarks using something like:
// http://www.mydomain.com/twitter.php?args=\0



// This program is free software; you can redistribute it and/or
// modify it under the terms of the GNU General Public License
// as published by the Free Software Foundation; either version 2
// of the License, or (at your option) any later version.



// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with this program; if not, write to the Free Software
// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
// See: http://www.gnu.org/licenses/gpl-2.0.html



// twitter username and password
$twitter_username = 'myusername';
$twitter_password = 'mypassword';
// API key for Google Maps
$google_mapsapikey = 'myapikey';



// max time to wait for tweet responses from twitter in seconds
$twitter_timeout = 30;
// max time to wait for Google geocoding service in seconds
$google_timeout = 30;



// get text message contents
$args = $_GET['args'];
// encode
$args = urlencode($args);



// geocode input
$req = curl_init("http://maps.google.com/maps/geo?q=$args&output=json&oe=utf8&sensor=false&key=$google_mapsapikey");
curl_setopt($req, CURLOPT_RETURNTRANSFER, true);
curl_setopt($req, CURLOPT_CONNECTTIMEOUT, $google_timeout);
$output = curl_exec($req);
curl_close($req);



// parse geocoding result
$result = json_decode($output, true);
if ($result == NULL) {
echo "Geocoding failed";
return;
}



// get address and coordinates
$address = $result['Placemark'][0]['address'];
$lon = $result['Placemark'][0]['Point']['coordinates'][0];
$lat = $result['Placemark'][0]['Point']['coordinates'][1];



// construct Google URL
$mapurl = urlencode("http://maps.google.com/maps?q=$lat+$lon&mrt=yp");



// send to twitter - based on
// http://morethanseven.net/2007/01/20/posting-to-twitter-using-php/
$tweet = "I am somewhere near $mapurl";
$req = curl_init('https://twitter.com/statuses/update.xml');
curl_setopt($req, CURLOPT_CONNECTTIMEOUT, $twitter_timeout);
curl_setopt($req, CURLOPT_RETURNTRANSFER, true);
curl_setopt($req, CURLOPT_POST, true);
curl_setopt($req, CURLOPT_POSTFIELDS, "status=$tweet");
curl_setopt($req, CURLOPT_USERPWD, "$twitter_username:$twitter_password");
$output = curl_exec($req);
curl_close($req);



// return result via text message
if (empty($output))
{
echo "Failed: $address, $lat, $lon";
} else {
echo "Tweeted: $address, $lat, $lon";
}
?>


Get JOSM

First download the current version of JOSM, which is a Java based editor. You will also need to install Java 1.5. You can also read more about JOSM here and here.

Start JOSM. On the command line this is something like:

java -jar -Xmx512M josm-latest.jar

There are some JOSM plugins that I think are essential, and here is how to add them:

• Start JOSM
•  Go to Edit -> Preferences
• Click on the plugins tab (looks like a wall socket/outlet)
• Click on “Download List”
• Check/tick the following: utilsplugin, validator, waydownloader
• Click on “OK”
• Restart JOSM

Next you need to tell JOSM your OpenStreetMap username and password.

• Go to Edit ->Preferences
• Click on the connection settings tab (looks like a planet)
• Enter your email address and password
• Click on “OK”

Next install the WMS Plugin for JOSM that will allow you to view Yahoo satellite images.

Download a Section of the Map

Next you will want to download the part of the map that covers your local area.

• Click on the Download From OSM toolbar button
• Check/tick “Download as a new layer”
• Drag the map with the right mouse button, zoom with the mouse wheel or Ctrl + arrow keys
• Drag a box with the left mouse button to select an area to download. Keep it small for now.
• Click on “OK” and wait a bit. It could take a few minutes so don’t give up too quickly

You will now be looking at a black screen with lots of lines on it. Zoom in to a few streets using the magnifying glass tool on the toolbar.

Get the Satellite Images

The next step is to get the satellite images so we can see how the streets compare with the real world.

• Go to  WMS -> Yahoo Sat (may have a slightly different name depending on the JOSM version)

At the top right under the Layers heading a new layer should appear called “Yahoo Sat” or something like that. Wait a bit and the images should start to appear.

Edit the Map

It’s now time to start editing. It should be clear if a street is not aligned with the Yahoo images. If it isn’t then you can click on the Select tool on the toolbar and start dragging the nodes (small yellow boxes). Position a road over where it is in the satellite image.

Upload!

Once you have fixed a few roads click on the Upload to OSM toolbar button.

• Enter a description for the change
• Click on “Upload Changes”

Once complete you can go to your OSM account page and view your edits. The URL is http://www.openstreetmap.org/user/myusername/edits. Replace “myusername” with your user name. You should see your first edit!

After waiting a few minutes your changes should start to appear on the map. Mapnik (the default renderer) renders the lower zoom levels more frequently, so you should see your changes appear in stages and only in some zoom levels. After a day or so all the lower zoom levels should show your changes.

Some Notes

The Yahoo satellite images are not always in the right place, but it seems from my experience they are pretty good. However this is something to keep in mind.

Along with fixing streets, it’s also possible to add in schools, malls, hospitals, businesses, rivers, streams, parks, etc. all by using the satellite images. There is plenty to do!

And here is the map today:

As you can see there are less roads. We found that many of the roads just don’t exist. Also after talking to some cabin owners we met, some roads have apparently been cut in two to build more cabins.

Comparing with the Google map we can see that we have missed a section of roads in the North East of Summerhaven, so we will need to go back for those sometime soon. Looking more closely at the Google map I can see many mistakes. Roads connecting when they don’t and the wrong names used.

Interestingly it seems Garmin have also used the incorrect TIGER data for their City Navigator NT. It contained a huge error which I corrected as soon as I saw it several weeks ago on the OSM map.

We marked on the map the locations of restrooms and parking areas, something which doesn’t appear on other maps.

Here is a screenshot showing our GPS trails overlaid on the TIGER data. As you can see the TIGER data is total nonsense.

We have now compiled a list of about half a dozen specific items to look at when we next visit, however I can say that the OSM map must be the most accurate map of Summerhaven that is publically available, even if it isn’t completed yet.

1. Large Scale Features

Adding national forests, parks and preserves, wilderness areas and the trails inside those areas for Arizona. I am perhaps 70% done with this goal. The benefit is vastly improved recreation maps for the six million people in the state. The area is huge (larger than the UK) and the total number of nodes is probably in the 100,000s.

2. Medium Scale Features

Adding key features in the city of Tucson. This includes major rivers and washes, large buildings such as hospitals, malls and big box stores, realigning primary and secondary roads. To date I have added at least one of each of these features. The benefit is a good overall map for the one million people who live in the Tucson area.

3. Small Scale Features

I plan to completely map in as much detail as possible the nine square miles or so around my home. This includes realigning all the streets, adding buildings which are not houses, adding recreation areas, ensuring all streets are named, etc. Of the three goals this is probably the most work. The number of people living in the area isn’t that much, but I intend this to be an example for future Tucson mappers.

The answer is to use the tag waterway:riverbank and build the river up from polygons that are next to each other. This diagram shows:

Seems simple enough. But what about bridges. The bridge:yes page explained. Just cut up the road and mark the section on the bridge with bridge:yes. The layer option was a bit ambiguous so to be sure I added layer:1 for the bridge.

So I tried added a section of the Pantano Wash in Tucson. After waiting 15 minutes or so the result was added to the map:

Looking good. Now to add some more.