A Nod to Passion
A few weeks back I BARTed, with bicycle, under San Francisco Bay to retrieve a Getaround-enrolled van in Emeryville. I’d agreed to move the belongings of an expatriate performance artist from under tarps in the backyard of an Oakland home – the bungalow was to be de-verminized – to an indoor storage facility. I’d forgotten that Sten was a writer when I met him and picked a particulatarly awful day to maneuver twenty-some boxes of books and vinyl from under the tarps and out through the yard and a comicly-constricted driveway. From there, over the Bay Bridge, marrying his fortunes into the misfortunes of my own stored studio equipment. Did I mention that I also needed to move, lock, stock, and barrel, all my stuff into a larger but cheaper, for the moment, unit? And return the van? And bicycle to UC Berkeley for Franck Billé’s 3:30p lecture, Somatic States: On Cartography, Geobodies, Bodily Integrity?
What may come across as complaint is, in reality, a prelude to gratitude for a brief exchange whose impact has stayed with me for twenty-some years. One year for each box schlepped.
I recall a conversation with Sten during a period where he was developing a piece about Antonin Artaud, 1996. The word passion kept creeping in. Dimly cognizant of its use in “the passions of Christ” the word had mostly contemporary meaning to me. As Admiral Murrah writes in a blog post about the etymology of passion, “it’s present use is one describing an intense desire … often sexual in nature … an irrational force that is … irresistible”. Sten directed me to the origins of the word, which Francis Berger delineates:
Our modern word passion finds its origin in the Latin pati from which later came the Late Latin words passio and passionem which was then incorporated into the Old French in the tenth century to describe the suffering and physical pain Christ experienced between the night of The Last Supper and his death. By the early thirteenth century the sense of suffering and endurance was extended to martyrs and, eventually, to suffering in general. The word was incorporated into the Greek from Latin to form the word pathos meaning emotion, pity, or compassion. In fact, it is only through these forms that the original meaning of passion survives in our minds. When we feel compassion, we don’t want to seduce someone. Nor do we feel particularly zestful or enthusiastic. Rather, we feel and experience the sufferings of another. The modern definitions of passion as sexual love or enthusiastic devotion appeared some time during the Renaissance and since then the original meaning of the word as signifying suffering and the endurance of suffering have become archaic and have all but evaporated from the contemporary mind.
Which brings me around to maps, the subject of the nuts and bolts explication of contemporary technologies to follow. Isn’t it natural, necessary, when one gets discouraged by evaluating one’s personal effort-to-reward ratio, to disengage? I found it so, and stepped back from attending geography and cartography conferences, mapathons, meetups, even my beloved geobreakfasts.
There remain things in my mind’s eye that I want to make visible, to prototype, to distribute, to discuss if they prove worthy. Discuss more vigorously when they don’t. Maps seem still to be the best way to bring those things out of darkness.
Sten, thank you for your artistry and for helping me to see that maps are my passion, in that evaporated meaning.
Tilting at TileMills
From late 2013 to early 2015 I put a great deal of effort into a self-paced deck & workshop about assembling a workbench from which to explore open source geospatial software and open geospatial data. The 60d nail in the coffin of that project was Mapbox’s end-of-lifing TileMill in their move to vector tiles but, in retrospect, my Geostack project was the very definition of Quixotic.
The shift to vector tiles has altered the component parts necessary for a contemporary geostack. For the way I work – making local interventions to software and data, sometimes offline on public transit, sometimes being, for all intents and purposes, offline due to an ISP on the skids – I need a self-contained platform.
Hello Tegola
James Gardner, in the introduction to his Geovation post about building your own static vector tile pipeline, mentions Tegola as an alternative strategy. It looked a viable, if rapidly evolving, candidate for what I needed. Hearing from Gretchen Peterson after I submitted a few pull requests to improve its documentation affirmed that I’d found a project addressing both front- and back- end concerns. I’ve put it through its paces and hitched my wagon to it. Here are some basics on how you can get started.
Assumptions
Unlike the Geostack presentation, which spelled out installation proceedures for OS X, Ubuntu, and Windows users, this treatment is exclusively OS X. I’ll assume you have basic familiarity with installing software via the Homebrew package manager and GitHub, and that you already have PostgreSQL installed with PostGIS extensions. If not, I’ll refer you to my Geostack deck which recent tweets suggest still helps people with setting up those basics. If your operating system is some flavor of unix you likely know the substitutions you need to make, e.g., which package manager is appropriate to your operating system. I have divested myself of all Windows software and have no resources to assist users of that operating system.
Component Installation
Note: Screendumps reflect package version numbers at the time I wrote this post. Version numbers that don’t increase over time are more worrisome than those that never change, so don’t be alarmed should you see, e.g., a 1.9.3 in place of a 1.9.1.
Go Go Go
As of this writing, Tegola supports one data provider, PostGIS, and Imposm3 has supplanted the execrable1 osm2pgsql as a reliable and efficient way to import OpenStreetMap data into your spatially-enabled database. Imposm3 and Tegola are both written in the Go programming language so we’ll start by installing Go via Homebrew.
$ brew install go
==> Downloading https://homebrew.bintray.com/bottles/go-1.9.1.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring go-1.9.1.sierra.bottle.tar.gz
==> Caveats
A valid GOPATH is required to use the `go get` command.
If $GOPATH is not specified, $HOME/go will be used by default:
https://golang.org/doc/code.html#GOPATH
You may wish to add the GOROOT-based install location to your PATH:
export PATH=$PATH:/usr/local/opt/go/libexec/bin
==> Summary
🍺 /usr/local/Cellar/go/1.9.1: 7,639 files, 293.7MB
$
Translating:
- a
gosubdirectory will be created in your home directory - edit your
.bash_profileto include$(go env GOPATH)/binwithout wiping out what is already there, e.g., for brevity,
export PATH=":$PATH:$(go env GOPATH)/bin"
Source your updated path:
$ source ~/.bash_profile
Imposm3
As I write this, Imposm3 does not follow a contemporary release model. Binaries for Linux are available from https://imposm.org/static/rel/ but for OS X you’ll need to install one dependency (LevelDB) via Homebrew –
$ brew install leveldb
==> Installing dependencies for leveldb: gperftools, snappy
==> Installing leveldb dependency: gperftools
==> Downloading https://homebrew.bintray.com/bottles/gperftools-2.6.1.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring gperftools-2.6.1.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/gperftools/2.6.1: 103 files, 4.1MB
==> Installing leveldb dependency: snappy
==> Downloading https://homebrew.bintray.com/bottles/snappy-1.1.7.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring snappy-1.1.7.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/snappy/1.1.7: 18 files, 115.2KB
==> Installing leveldb
==> Downloading https://homebrew.bintray.com/bottles/leveldb-1.20_2.sierra.bottle.tar.gz
######################################################################## 100.0%
==> Pouring leveldb-1.20_2.sierra.bottle.tar.gz
🍺 /usr/local/Cellar/leveldb/1.20_2: 26 files, 1.1MB
– then use go get to install Imposm3 from their GitHub repo.
$ go get github.com/omniscale/imposm3
$ go install github.com/omniscale/imposm3/cmd/imposm3
It wouldn’t hurt to check that your installation and path are working at this time.
$ imposm3 version
0.4.0
$
Data: Mapzen’s Metro Extracts
Two years ago I was in the middle of a three week stint in Timișoara, Romania.
For nostalgia’s sake I’ll continue by generating a custom Mapzen Metro Extract for the region. Feel free to use one of their prepopulated extracts if you’re eager to get on with it or generate your own custom extract (it may take a few hours before it’s ready).
Download the OSM PBF version from Raw OpenStreetMap datasets.
Tegola-OSM
Terranodo’s provided a GitHub repository called tegola-osm that holds scripts and configuration files designed to help you bootstrap OpenStreetMap data into an instance of Tegola. In the time I’ve been working with Tegola, the OSM repo has been deleted, along with Issue and Pull request history, and rebuilt; indeed, the whole Tegola ecosystem is very much a work-in-progress. Let’s clone and modify it.
$ cd ~/Repos/erictheise/
$ git clone git@github.com:terranodo/tegola-osm.git timișoara
$ cd timișoara
You’ll likely want to put your work under source control so create your own GitHub repo … well, raise my rent if GitHub didn’t just replace ‘ș’ with ‘-‘! Guess we’re still not living in an i18n world. Anyway, feel free to check my repo for files discussed in this post. In particular, edit your .gitignore now so that it looks something like:
# JetBrains IDE files
.idea
# OS X files
.DS_Store
# OSM data snapshots
data/
# Files containing sensitive information
tegola.toml
dbcredentials.sh
We’ll discuss its contents below. Change your remote origin to point to your repo, not mine, and push.
$ git remote set-url origin git@github.com:erictheise/timi-oara.git
$ git commit -m 'Keep credentials out of the repo.' .gitignore
$ git push -u origin master
For the sake of organization I’ll create a data subdirectory where I can store my .pbf file locally. I’m not going to be precious about the snapshot, thus data/ is one entry in .gitignore.
$ mkdir data
$ mv ~/Downloads/ex_Mv2vB28df5Tso1ViL1nyEgn9npM5r.osm.pbf data/timișoara.osm.pbf
Import the metro extract. First we’ll glance at the Getting started portion of the Tegola docs and sidestep the inexcusable advice to give the tegola role PostgreSQL superuser privileges. Log into your PostgreSQL cluster as an administrator:
$ psql
psql (9.6.5, server 9.6.5)
Type "help" for help.
postgres=# CREATE USER tegola WITH PASSWORD 'jw8s0F4';
CREATE ROLE
postgres=# CREATE DATABASE timisoara OWNER tegola;
CREATE DATABASE
postgres=# \c timisoara
You are now connected to database "timisoara" as user "postgres".
postgres=# CREATE EXTENSION postgis;
CREATE EXTENSION
timisoara=# CREATE EXTENSION hstore;
CREATE EXTENSION
It should go without saying: use your own secure password. Next we’ll glance at the Imposm3 README and tweak their imposm3 import command to work with the configuration files pulled down from tegola-osm and the database we’ve just created.
$ imposm3 import -connection postgis://tegola:jw8s0F4@localhost/timisoara -mapping imposm3.json -read data/timișoara.osm.pbf -write
$ imposm3 import -connection postgis://tegola:jw8s0F4@localhost/timisoara -mapping imposm3.json -deployproduction
I’m suppressing the import’s output but at completion your database should look like this:
timisoara=# \d
List of relations
Schema | Name | Type | Owner
--------+-----------------------------+----------+----------
public | geography_columns | view | postgres
public | geometry_columns | view | postgres
public | osm_admin_areas | table | tegola
public | osm_admin_areas_id_seq | sequence | tegola
public | osm_amenity_areas | table | tegola
public | osm_amenity_areas_id_seq | sequence | tegola
public | osm_amenity_points | table | tegola
public | osm_amenity_points_id_seq | sequence | tegola
public | osm_buildings | table | tegola
public | osm_buildings_id_seq | sequence | tegola
public | osm_landuse_areas | table | tegola
public | osm_landuse_areas_gen0 | table | tegola
public | osm_landuse_areas_gen1 | table | tegola
public | osm_landuse_areas_id_seq | sequence | tegola
public | osm_other_areas | table | tegola
public | osm_other_areas_id_seq | sequence | tegola
public | osm_other_lines | table | tegola
public | osm_other_lines_id_seq | sequence | tegola
public | osm_other_points | table | tegola
public | osm_other_points_id_seq | sequence | tegola
public | osm_place_points | table | tegola
public | osm_place_points_id_seq | sequence | tegola
public | osm_transport_areas | table | tegola
public | osm_transport_areas_id_seq | sequence | tegola
public | osm_transport_lines | table | tegola
public | osm_transport_lines_gen0 | table | tegola
public | osm_transport_lines_gen1 | table | tegola
public | osm_transport_lines_id_seq | sequence | tegola
public | osm_transport_points | table | tegola
public | osm_transport_points_id_seq | sequence | tegola
public | osm_water_areas | table | tegola
public | osm_water_areas_gen0 | table | tegola
public | osm_water_areas_gen1 | table | tegola
public | osm_water_areas_id_seq | sequence | tegola
public | osm_water_lines | table | tegola
public | osm_water_lines_gen0 | table | tegola
public | osm_water_lines_gen1 | table | tegola
public | osm_water_lines_id_seq | sequence | tegola
public | raster_columns | view | postgres
public | raster_overviews | view | postgres
public | spatial_ref_sys | table | postgres
(41 rows)
timisoara=#
Although their documentation implies that you don’t need Natural Earth Data if you’re only interested in OSM, you do. The provided natural_earth.sh script will download and import this data for you.
You need to take precautions to keep your database credentials from creeping into your repository. There’s an undocumented clause in natural_earth.sh such that you can supply your credentials in a file named dbcredentials.sh. That file’s included in my .gitignore so if you choose to go that route, create dbcredentials.sh and insert the correct values for DB_USER and DB_PW.
$ cat dbcredentials.sh
DB_USER="postgres"
DB_PW="postgres_user_password"
$
Alternatively, edit the correct values for DB_USER and DB_PW into the script, run it, but revert your changes after a successful run.
$ ./natural_earth.sh
$ git checkout -- natural_earth.sh
Similarly, you also need to run the undocumented osm_land.sh script which can also access dbcredentials.sh. These take a non-trivial amount of time to download.
Terranodo’s documentation says to “Execute postgis_helpers.sql against your OSM database.” Practically, this means one of two things. From the command line:
$ psql -U tegola -W -d timisoara -a -f postgis_helpers.sql
/* helper functions that should be installed alongside the OSM import */
BEGIN;
BEGIN
-- Inspired by http://stackoverflow.com/questions/16195986/isnumeric-with-postgresql/16206123#16206123
CREATE OR REPLACE FUNCTION as_numeric(text) RETURNS NUMERIC AS $$
DECLARE test NUMERIC;
BEGIN
test = $1::NUMERIC;
RETURN test;
EXCEPTION WHEN others THEN
RETURN -1;
END;
$$ STRICT
LANGUAGE plpgsql IMMUTABLE;
CREATE FUNCTION
COMMIT;
COMMIT
Alternatively, you could copy and paste those commands within a psql session.
Do the same with postgis_index.sql.
Tegola
Tegola is distributed as a binary. Just this weekend, the team’s released v0.4.0 to replace the antiquated v0.3.2 (13 Mar 2017) release. Do check the aforementioned Getting started page to see the most recent OS X release. I downloaded tegola_darwin_amd64, moved it to /usr/local/bin, renaming it tegola-0.4.0, and made it executable:
$ mv ~/Downloads/tegola_darwin_amd64 /usr/local/bin/tegola-0.4.0
$ chmod +x /usr/local/bin/tegola-0.4.0
At the risk of repeating myself, it’s poor practice to expose your database credentials via a public repository, so I’m going to move the supplied tegola.toml configuration file to tegola.toml-example, add tegola.toml to .gitignore, then copy the example back to tegola.toml. It may prove to be a bit of dance, keeping the two in sync apart from the credentials, but diffing the two locally should only reveal information you want to keep private.
$ git mv tegola.toml tegola.toml-example
$ cp tegola.toml-example tegola.toml
$ git commit -m 'Keep credentials out of the repo.'
$ git push
Finally, start your tegola server –
$ tegola-0.4.0 -config=tegola.toml
– and browse to http://localhost:8080/capabilities where you’ll hopefully see:
I’m crossing my fingers for you.
From there, you’ll need to fiddle a bit to coax Tegola into rendering your map in all its glory. Assuming you’re using a metro extract, find its center in a psql session –
# SELECT ST_AsText(ST_Centroid(ST_Extent(ST_Transform(geometry, 4326)))) from osm_transport_lines;
st_astext
------------------------------------------
POINT(21.2292561688563 45.7534523467679)
(1 row)
#
– enter those values at the appropriate place in tegola.toml –
[[maps]]
name = "osm"
attribution = "Tegola OSM" # map attribution
center = [21.2292561688563, 45.7534523467679, 8.0] # optional center value. part of the TileJSON spec
[[maps.layers]]
– and restart the server. In my experiences with Tegola I’ve found that various layers have issues. Your server may start without error, but keep an eye on your terminal as you interact with the map viewer at http://localhost:8080. In my case, hitting my server root produces
2017/11/12 22:18:51 handle_map_style.go:167: layer (providerLayerName: water_areas_gen0) has unsupported geometry type (<nil>)
in my terminal. For expediency’s sake I’ve disabled the problematic layer by commenting out the corresponding lines in tegola.toml –
# # Water Areas
# [[maps.layers]]
# name = "water_areas"
# provider_layer = "osm.water_areas_gen0"
# min_zoom = 3
# max_zoom = 9
– and restarting my server.
Because you’re starting with a fresh instance, there’s no tile cache and it may be some seconds, even minutes, before you see much; a bit like getting the blood flowing after your leg’s gone to sleep. Once my map appeared, I selected a (fractional!) zoom level of 11.5 et voilâ:
Tegola’s built-in map viewer displays the known layers at left – they can be toggled on and off – and includes the useful Inspect Features toggle at lower left. This can provide a flood of information, but it’s useful for exploring unfamiliar datasets. Modesty bedamned, I’ll point out that it might benefit from fully implementing the strategy used in my Rrose plugin for Leaflet, displaying the information popup above the feature when nearer the bottom of the viewport. Otherwise, there’s a tedious dance between revealing partial information, finding a point from which the map can be dragged up, then trying to locate the POI again. It can require multiple iterations to get at all the information contained in the popup.
But the point of this exercise, if you’ve been keying it all in, is to enable you to build your own viewport where the popups and everything else behave exactly the way you want them to. If you’re not quite committed to setting up your own instance of Tegola you can visit Terranodo’s public instance to see it in action.
Next Steps
For me, next steps will be to use my local Tegola-based stack as a foundation from which to explore the limits of styling, manually, through Maputnik, an open source Mapbox GL editor, and through interactivity via JavaScript; to run Mapzen’s Tangram; and to dig deeper into the capabilities of OpenLayers and Cesium. Those will likely be the subjects of future posts.
It’s been suggested I offer workshops on a vector tile geostack, or create another self-paced deck on the subject. Presently my inclination is to avoid this temptation in the struggle to rework my personal ratio of effort-to-reward. See above.
Here’s to passions, yours and mine.
Footnote
1 In several years of exposure to osm2pgsql I never encountered a version that worked under Windows, not once; witness the warnings and strikethroughs on its wiki page. The OS X version was unstable and it was my greatest source of anxiety between one workshop offering to the next. It should never have been allowed within earshot of an argument for making the switch to OpenStreetMap.