workflow.md

How to import

Getting started

Creating an import account

• OSM best practices require that you do not use your normal OSM account for the imports. Create a new account for this purpose. Usually, it's your existing OSM username followed by _imports (e.g. tekim_imports).
  • Tip: If you use GMail, you can create email aliases by adding +osmimport to your email address, and use this alias to sign up for the additional OSM account. This makes it easier to log into the correct account, without having to create a new email address!
  • Example: [email protected] can become [email protected] - They both refer to the same email inbox, but can be used to create unique logins. Post your import account username in this ticket.
• Remember you should log into the OSM Tasking Manager (see below) with your 'imports' username but you will also need to use that same username from any data upload from JOSM.

Getting familiar with JOSM

To contribute to this project, you need to use the JOSM editor. Here are some resources to get you started:

• JOSM Website - https://josm.openstreetmap.de/
• LearnOSM - http://learnosm.org/en/josm/
• Mapbox Mapping wiki - https://www.mapbox.com/blog/making-the-most-josm/

Tools for the import

Before you begin, make sure your tools are setup and working:

1. OpenStreetMap account with the _imports suffix.
2. Working JOSM installation, with the Conflation plugin installed from Preferences > Plugins.
3. DRCOG imagery added to JOSM:
   • Imagery > Imagery preferences - select the +TMS button to add a new TMS imagery source.
   • Add the URL: tms:http://tiles.openaerialmap.org/user/5bc8ba7fa8a0140005ce649b/{z}/{x}/{y}@2x
4. (Optional) Text editor with advanced find/replace functionality, such as BBEdit, Notepad++, or Atom.

Import workflow

Selecting a task in the Tasking Manager

• Go to http://tasking-manager.mapsarecool.com - Select a project, typically the one on top/highest priority unless told otherwise (i.e. at an importathon).
• You will see the main project screen shown below. This example is for the Idaho Springs area pilot project
• You can review the general instructions by clicking the Instructions tab
• If you are ready to start mapping click on the Start contributing button from the Instructions tab or click the Contribute tab.

• On the Contribute tab select a specific task from the map or click the Take a task at random button or click a task on the mapview.
• Make sure your JOSM is running with Remote Control Enabled, and Choose Edit with JOSM

• This will populate your JOSM with the OSM data in your task area.

Getting the DRCOG Building Data for your Task

• Each task has a pregenerated osm-format map data file associated with it
• This pregenerated file contains the building shapes as well as attributes about each building such as address and height
• Your job is to review the pre-generated OSM file in JOSM and make some adjustments prior to uploading the file to the OpenStreetMap database

• You can just open, but we suggest saving, the DRCOG .osm file by clicking the link in the Extra Instructions section shown in the image above.
• You can ignore the Tip about downloading the .gpx file in order to see the current task boundary. The pregenerated OSM data file for the task will only contain buildings within the task boundary
• Open the DRCOG .osm data file in JOSM; depending on your JOSM install method you may simply be able to click on the JOSM file in your Downloads directory
  • CAUTION If you have worked on multiple tasks for this project you may have multiple .osm files in your 'Downloads' directory; be sure to open the correct file for your current task
• Once you have opened the .osm file you should see something like the below image; this is for the the task #11 from the Idaho Springs pilot project

• If you don't see both data sources (drcog...osm & Data Layer x), make sure your JOSM remote control is enabled, restart JOSM and try again from the selecting Edit in JOSM from the Tasking Manager step above.
• You may also have an imagery layer as some projects may have a preferred imagery to use for comparison that will automatically load. If not, bring in Bing and any others you prefer; most likely existing data will be mapped/aligned to Bing. Ideally align to available GPS traces; where there is none or still is questionable, align imagery and DRCOG data to existing mapping or Bing where there is little to no existing data.

Pre-checking the data before merging

The overall workflow is to "merge" the DRCOG data into your OSM Data Layer, while reviewing both data layers for possible conflicts. Some quality checks before merging will make things easier.

1. With the .osm file layer activated (no other OSM map data visible), run the JOSM Validator (SHIFT+V) to look for errors. Common errors at this stage can include multiple buildings with the same address (usually an error where a house and a garage or shed have been given the same address tags). Refer to the section below on fixing common errors for solutions and guidelines.
2. While still in the new .osm file layer, Select All (CTRL+A) to select all objects, and then Deselect Nodes (SHIFT+U) to select all of the buildings. In the Selection window on the right side of the screen, scroll through the list of selected buildings to look for malformed street names. Examples of bad street names include:
   • unexpanded abbreviations (Ct instead of Court, or Ln instead of Lane)
   • occasional errors in the automatic import, such as the street name Steele Street being expanded to Streeteele Street
3. Do not upload bad data to OSM, fix it before importing, or flag the task as having problems:
   • Use github issues to flag concerns, make sure you indicate the task number in your issue.
   • Leave comments in the tasking manager.
4. Remove multi-part buildings from the .osm file layer before importing (buildings with multiple polygons side-by-side), these require special handling and are not being imported at this time. If you are already well trained in multi-part buildings and 3D building tags you may proceed with caution.

Reviewing the data before uploading

• Make sure you have the DRCOG dataset active, select a building and use Merge selection under the Edit menu (CTRL+SHIFT+M).
• Merge the building into the OSM Data Layer as shown in image below.

• Now switch to the OSM Data Layer (right click on "Data Layer 1" and select "Activate"), there will be 3 scenarios:

1. If there is no existing OSM data for the building, examine tags to make sure there are no obvious mistakes like incomplete addresses, impossible heights, building type make sense for the area/imagery.
   • Then delete the building from the DRCOG layer (i.e. once it's merged into your OSM layer, it can safely be deleted from the import layer and once all buildings are gone from the import layer, you're done).

If there is an existing building in OSM then you must decide how to conflate the two datasets (NOTE: Preserve the work of previous mappers wherever possible).

2. If existing buildings in OSM are of higher quality:

   • Copy the tags from the imported building as necessary.
   • Delete the imported building from both layers (i.e we're only saving tags from DRCOG).

3. If the imported data are of higher quality:

   • Select both buildings and use the Replace geometry tool (CTRL+SHIFT+G) under Additional Tools menu.
   • Conflate the tagging by selecting which to use if there are conflicts (example: building type).
   • Delete the building from the import layer (the original and DRCOG should now be one geometry on your OSM Layer).

• Do not worry about editing any other nodes such as points of interest contained within the building shape.
• If there are any problems you don't know how to deal with, do not proceed. Instead, flag the task in the tasking manager for a more advanced user to look at.
  • Use github issues to flag concerns; include the task number in your issue.
  • Then unlock your task on the tasking manager and pick a new area to work on, leaving a comment there as well so the next mapper is aware.
• Once you have merged the DRCOG buildings that need to be added to OSM and copied tags or replaced geometry to existing OSM data; you can prepare to upload. If following the workflow you should know your area is finished when you've deleted (merged) all the data from the DRCOG dataset.
• Delete the DRCOG dataset (layer) from JOSM: right-click the "drcog...osm" layer from the layers menu and select Delete - if you get a warning, make sure you did not miss any buildings (i.e. this layer should be completely empty if you have followed the workflow).

Finally, upload your data

Before uploading your OSM Data Layer, do one more check - Run JOSM Validator (SHIFT+V), and if there are errors, fix them before uploading.

• Select and activate the OSM Data Layer.
• Click the Upload button, the green up arrow button.

• If you see a "Suspicious data found" warning, double check that none need to be addressed, then click "Continue upload".

• Use a changeset comment (will be preloaded via the tasking manager): DRCOG Planimetrics Import #DRCOGPlanimetrics #[City or County][Feature] https://wiki.openstreetmap.org/wiki/Denver_Planimetrics_Import and source: Denver Regional Council of Governments https://data.drcog.org/dataset/building-roofprints-2014.

• If you see an Authorization window asking you to log in to OpenStreetMap, log in and remember to use your _imports username.
• Go back to the Tasking Manager and click Mark task as done. If necessary, leave a comment so if another mapper validates your edits they are aware (i.e. one building from satellite imagery not in import data; left alone).

Fixing common problems

Is it really a building?

Watch out for situations where something that isn't a building may have been traced as one. Do not import these "not a building" buildings. Common cases include:

• Parked tractor trailers getting traced as sheds.
• Boats, sandboxes or garden boxes in yards getting traced as sheds or garages.
• Buildings that were traced during construction, and have the type building=foundation/ruin which is not a valid OSM building type.
  • If the building exists in the imagery, change the building type to building=house or whatever building type is appropriate for the structure.
  • Check the height tag. If you see height=0, then either update the height information if you can determine it through other imagery, or simply delete the height tag.

Duplicate Addresses

Duplicate addresses can occur in a few situations, and are easy to fix.

1. The most common case is when a house and an accessory building are given the same address. This often occurs when the accessory building is not a building=garage or building=shed (which have addresses automatically removed), but something uncommon, such as a gazebo.
   1. The fix is to simply remove the address tags from the accessory building before importing.
2. Another common case is when multiple buildings in an apartment complex have the same address. In this case, if you believe this to be accurate, the duplicate addresses may remain (it is possible for multiple structures to have the same address).
   1. In some cases, the address field may be the same, but the street name is something like addr:street=Logan Street Bldg 4. In this case, remove Bldg 4 from the street name, and add a reference tag to indicate which building number the building is: ref=4.

Building "slivers"

• Check for small building parts that should be joined to the main building. We've already found a few examples of these in the data (see LA Building Import - issue #19), so make sure you keep an eye out for these. To join small parts, select both polygons and select Tools > Join overlapping Area.

• If it's a small sliver, it makes sense that the "proper" data is on the larger object. Delete all the tags on the sliver and join the two objects.
• If it's larger, like a strip mall split into pieces, then do:
  • start_date -> None if multiple or the one option if there's only one
  • height -> largest number
  • ele -> largest number
  • building:units -> none if different

Bad addresses (Advanced Repair)

If during your initial review of the .osm file in JOSM you find that there is a consistently mis-named street, it can be helpful to repair it in the .osm file before importing. Example: All of the buildings on Elm Street are labeled "Elm St".

1. Close the .osm file by right clicking on it and selecting Delete. If you are partway through the import and have already made changes to the .osm file, you should save before closing.
2. Open the .osm file in an advanced text editor such as BBEdit, Notepad++, or Atom.
3. Use the editors find and replace functionality to find all instances of the error and replace with the correct text.

Example: You see the following way in the .osm file (with a bad street name):

  <way id='-162872'>
    <nd ref='-155297' />
    <nd ref='-155299' />
    <nd ref='-155301' />
    <nd ref='-155303' />
    <nd ref='-155297' />
    <tag k='addr:city' v='Denver' />
    <tag k='addr:housenumber' v='66' />
    <tag k='addr:postcode' v='80203' />
    <tag k='addr:state' v='CO' />
    <tag k='addr:street' v='Logan St' />
    <tag k='building' v='residential' />
    <tag k='height' v='8' />
    <tag k='source' v='DRCOG PLANIMETRICS DATA' />
  </way>

Then in your text editor you could search for the text v='Logan St', and replace with the text v='Logan Street' to repair all instances of the error in the .osm file.

Tip: This can be used to repair many small problems in the .osm file, such as unexpanded street suffixes, or strange import errors, such as Steele Street being expanded to Streeteele Street (real example).

What to watch out for

When not to import a building

1. The building already exists in OSM, well traced, with complete tags.
   • In this case, simply add any useful tags from the DRCOG building to the originally drawn building, and delete the building from you .osm file.
   • If the building exists in OSM, but the DRCOG trace is higher quality, then you can combine the buildings using the Replace Geometry command (CTRL+SHIFT+G). When asked to combine the tags, use the original mapper's tags whenever possible.
2. The building is a 3D or multi-part building.
   • These are not formatted correctly in the current .osm files, and should not be imported as is. Correcting the multi-part buildings into compliant OSM 3D buildings is an advanced task outside the scope of this import (proceed with caution).
3. The OSM data layer includes updated information about why the building may not exist anymore, such as a construction=residential area indicating that the buildings may have been demolished.

Validate the import with your eyes before uploading!

• Run the JOSM validator. Check for any errors it detects.
• Inspect everything else with a critical eye! Don't trust that the validator or FIXME tags will catch everything. There may be other bugs that only you can detect. Use your human smarts!

Conflating with existing data

• Look for any overlaps with existing buildings. Existing buildings in OSM are probably more up-to-date than our imported data. Do not assume the imported data is better, mostly likely it is worse!
• Carefully combine the import data with the existing OSM data. If you aren't sure about some tags, ask someone! Especially ask the original mapper!

How to validate the data

• We are working to get more up-to-date aerial imagery... for now, try several of the default sources.
• See if the street is on Mapillary or OpenStreetCam; there's JOSM plugins for that!
• Contact mappers that have added information like POIs in the area - maybe they know or can look?
• Go out and check it out yourself! Take a field trip!
• DO NOT USE GOOGLE MAPS OR GOOGLE STREET VIEW.

Identifying new buildings with imported and existing data

• The imagery was purchased by DRCOG. This is the same imagery used as a reference for tracing the imported data. In some areas, the default imagery sources are more updated than the imported data and we can use it to identify newer buildings not found in the imported and/or existing data.
• Newer building should be identified and flagged. Tag the building with fixme and add a note: Appears in satelite imagery. We can verify this with more recent planimetric data.
• Likewise, buildings maybe demolished and does not exist anymore, delete them.

Communicate communicate communicate!

How to ask for help

• Search the Training Material in the DRCOG Import Drive.
• Look on the wiki; if you can't find an answer, create issues on this github repo.
• Or ask questions on the OSM #Colorado Slack Channel.
• Contact DRCOG or OSM-Colorado.

How to share your progress

• Make sure you upload your changes with the recommended comment and mark your task on the tasking manager as complete. Leave a comment there as well if it would be helpful (i.e. a few buildings were difficult to tell which dataset was more accurate, etc.).

How to communicate with other mappers

• Be responsive to comments that other OSM users make on your changesets. If there is a problem with your imports, stop and fix the problem before continuing.
• JOSM GeoChat feature.
• Come to one of the next OSM related meetup groups in Colorado.
• Befriend other mappers on openstreetmap.org!

Thanks to the LA Buildings Import

• This import workflow guide is based on the guide published by the Los Angeles building import at https://github.com/osmlab/labuildings/blob/master/IMPORTING.md.