![Screenshot of a leaflet map showing Leicester Square, London, with a popup giving geographic information and a satellite view.](/post/2020-12-19-londonmapbot-leaflet_files/leicester-square.png)
Closest I’ve been to Leicester Square since the start of lockdown.
tl;dr
I recently made a Twitter bot with R, {rtweet}, MapBox and GitHub Actions – londonmapbot – that tweets images of random coordinates in London. I decided to explore them interactively by creating a simple {leaflet} map. You can jump directly to the map).
The bot
I built the londonmapbot Twitter bot as a fun little project to get to grips with GitHub Actions. An action is scheduled every 30 minutes to run some R code that (1) selects random coordinates in London, (2) fetches a satellite image from the MapBox API, (3) generates an OpenStreetMap URL, all of which are (4) passed to {rtweet} to post to the londonmapbot account.
The outputs have been compelling so far. The composition is usually ‘accidentally’ pleasing. Sometimes landmarks are captured, like The Shard, The Natural History Museum and V&A and Heathrow.
![Screenshot of a tweet by londonmapbot showing a satellite view of the area round London Bridge Station and The Shard.](/post/2020-12-19-londonmapbot-leaflet_files/shard.png)
The Shard looks pointy even in 2D.
I was wondering whether the bot has ‘found’ other landmarks that I hadn’t noticed or whether it’s found my house. The londonmapbot source code doesn’t have a log file for all the coordinates it’s generated, so I figured the easiest way to get this information and explore it would be to grab all the tweets – which contain the coordinates as text – and then map the results.
Packages
I’m loading the tidyverse for data manipulation with {dplyr}, {tidyr} and {stringr}. {rtweet} greatly simplifies the Twitter API and the objects it returns. We’ll use it to fetch all the tweets from londonmapbot.
I’m using a few geography-related packages: {sf} for tidy dataframes with spatial information; {geojsonio} to read spatial files in geojson format; {PostcodesioR} to fetch additional geographic data given our x-y information; and {leaflet} to build interactive maps from spatial data.
library(tidyverse)
library(rtweet)
library(sf)
library(geojsonio)
library(PostcodesioR)
library(leaflet)
A particular shoutout to rOpenSci for this post: {rtweet}, {geojsonio} and {PostcodesioR} have all passed muster to become part of the rOpenSci suite of approved packages.
Fetch tweets
{rtweet} does all the legwork to fetch and parse information from the Twitter API, saving you loads of effort.
The rtweet::get_timeline()
function is amazing in its user-side simplicity. Pass the account name from which to fetch tweets, along with the number of tweets to get (3200 is the maximum).
lmb_tweets <- get_timeline("londonmapbot", n = 3200)
lmb_tweets[1:5, c("created_at", "text")] # limited preview
## # A tibble: 5 x 2
## created_at text
## <dttm> <chr>
## 1 2020-12-29 09:36:20 "51.5519, -0.33\nhttps://t.co/ica9ZypBLS https://t.co/5gS…
## 2 2020-12-29 08:59:46 "51.4392, 0.1636\nhttps://t.co/2DsbbLYIDG https://t.co/KV…
## 3 2020-12-29 06:28:48 "51.6773, -0.2555\nhttps://t.co/1hu2VoxCBF https://t.co/b…
## 4 2020-12-29 05:56:15 "51.674, -0.4042\nhttps://t.co/HMhmUVVrIn https://t.co/mP…
## 5 2020-12-29 05:31:03 "51.4451, 0.1058\nhttps://t.co/nWuqy4s7am https://t.co/qv…
{rtweet} has a function to quick-plot tweets over time. There’s meant to be a tweet every half-hour from londonmapbot, but GitHub Actions has been a little inconsistent and sometimes fails to post.
rtweet::ts_plot(lmb_tweets) + # plot daily tweets
labs(
title = "@londonmapbot tweets per day",
x = "", y = "", caption = "Data collected via {rtweet}"
)
Extract tweet information
The dataframe returned by {rtweet} contains nearly 100 columns. For our purposes we can minimise to:
- the unique tweet identifier,
status_id
, which we can use to build a URL back to the tweet - the datetime the tweet was
created_at
- the tweet text content, from which we can isolate the latitude and longitude values
- the
media_url
to the MapBox image attached to each tweet - the full OpenStreetMap link in each tweet via
urls_expanded_urls
lmb_simple <- lmb_tweets %>%
filter(str_detect(text, "^\\d")) %>% # must start with a digit
separate( # break column into new columns given separator
text, # column to separate
into = c("lat", "lon"), # names to split into
sep = "\\s", # separate on spaces
extra = "drop" # discard split elements
) %>%
mutate( # tidy up variables
lat = str_remove(lat, ","),
across(c(lat, lon), as.numeric)
) %>%
select( # focus variables
status_id, created_at, lat, lon,
osm_url = urls_expanded_url, media_url
)
lmb_simple[1:5, c("status_id", "lat", "lon")] # limited preview
## # A tibble: 5 x 3
## status_id lat lon
## <chr> <dbl> <dbl>
## 1 1343853346478841862 51.6 -0.33
## 2 1343844145518026752 51.4 0.164
## 3 1343806154397409280 51.7 -0.256
## 4 1343797964645523456 51.7 -0.404
## 5 1343791619049480192 51.4 0.106
Reverse geocode
Tweets from londonmapbot are really simple by design; they only have the latitude and longitude, a link to OpenStreetMap and a satellite image pulled from the MapBox API. It might be interesting to provide additional geographic information.
{PostcodesioR} can perform a ‘reverse geocode’1 of our coordinates. Give latitude and longitude to PostcodesioR::reverse_geocoding()
and it returns a list with various administrative geographies for that point.
lmb_geocode <- lmb_simple %>%
mutate(
reverse_geocode = map2(
.x = lon, .y = lat,
~reverse_geocoding(.x, .y, limit = 1) # limit to first result
)
) %>%
unnest(cols = reverse_geocode) %>% # unpack listcol
hoist(reverse_geocode, "postcode") %>% # pull out postcode into a
hoist(reverse_geocode, "admin_district") # pull out borough
lmb_geocode[1:5, c("lat", "lon", "postcode", "admin_district")] # limited preview
## # A tibble: 5 x 4
## lat lon postcode admin_district
## <dbl> <dbl> <chr> <chr>
## 1 51.6 -0.33 UB6 7QT Ealing
## 2 51.4 0.164 DA5 2DJ Bexley
## 3 51.7 -0.256 WD6 5PL Hertsmere
## 4 51.7 -0.404 WD24 5TU Watford
## 5 51.4 0.106 DA15 9BQ Bexley
The object returned from the reverse geocode is a nested list that we can tidyr::hoist()
the geographic information from. Here we grabbed the postcode and ‘administrative district’, which for our purposes is the London borough that the point is in.
Convert to spatial object
Right now we have a dataframe where the geographic information is stored as numeric values. We can use the {sf} package to convert and handle this information as spatial information instead.
Basically we can use {sf} to ‘geographise’ our dataframe. It can add geometry (points in our case), dimensions (XY, meaning 2D), the maximum geographic extent (a ‘bounding box’ that roughly covers London) and recognition of the coordinate reference system (‘4236’ for latitude-longitude).
The sf::st_as_sf()
function performs the magic of converting our tidy dataframe into a tidy spatial dataframe. You’ll see that the print method provides us the extra spatial metadata and that our geographic information has been stored in a special geometry
column with class sfc_POINT
.
lmb_sf <- lmb_geocode %>%
st_as_sf(
coords = c("lon", "lat"), # xy columns
crs = 4326, # coordinate reference system code
remove = FALSE # retain the xy columns
)
lmb_sf[1:5, c("status_id", "geometry")] # limited preview
## Simple feature collection with 5 features and 1 field
## geometry type: POINT
## dimension: XY
## bbox: xmin: -0.4042 ymin: 51.4392 xmax: 0.1636 ymax: 51.6773
## CRS: EPSG:4326
## # A tibble: 5 x 2
## status_id geometry
## <chr> <POINT [°]>
## 1 1343853346478841862 (-0.33 51.5519)
## 2 1343844145518026752 (0.1636 51.4392)
## 3 1343806154397409280 (-0.2555 51.6773)
## 4 1343797964645523456 (-0.4042 51.674)
## 5 1343791619049480192 (0.1058 51.4451)
Map it
London boundary geojson
Coordinates for londonmapbot tweets are selected randomly within a rectangle roughly within the boundary of the M25 motorway. We can grab a polygon of the Greater London boundary to see which points fall within the ‘true’ extent of London.
To do this, I’m using a set of boundaries for England’s regions2 via the Office for National Statistics’s Open Geography Portal API. The polygons are ‘ultra generalised’ to be represented by very few points (every 500m). This means it doesn’t follow the exact boundary of London, but that’s okay: it’s only being used as a guide and we get the benefit of a small polygon file.
ldn_sf <- geojson_read(
paste0( # API endpoint for NUTS1 geography geojson
"https://opendata.arcgis.com/datasets/",
"01fd6b2d7600446d8af768005992f76a_4.geojson"
),
what = "sp" # read as spatial object
) %>%
st_as_sf() %>% # convert to sf object
filter(nuts118nm == "London") # London polygon only
ldn_sf[, c("nuts118nm", "geometry")] # limited preview
## Simple feature collection with 1 feature and 1 field
## geometry type: MULTIPOLYGON
## dimension: XY
## bbox: xmin: -0.5097014 ymin: 51.28676 xmax: 0.3340242 ymax: 51.69188
## CRS: unknown
## nuts118nm geometry
## 1 London MULTIPOLYGON (((-0.01191868...
{ggplot2} has a geom for quick-plotting of {sf} objects, so we can check the boundary.
ggplot(ldn_sf) +
geom_sf() +
labs(
title = "London boundary",
subtitle = "Ultra-generalised NUTS1 extent",
caption = "Data collected via ONS Open Geography Portal API"
)
Build map with {leaflet}
You can build up layers in {leaflet} in a similar kind of way to a {ggplot2} graphic. The base map is applied with addProviderTiles()
, followed by the London boundary with addPolygons()
, with the points added as circle-shaped points with addCircleMarkers()
.
lmb_map <- leaflet(ldn_sf, width = '100%') %>%
addProviderTiles("CartoDB.Positron") %>%
addPolygons( # generalised London boundary
color = "black", weight = 2,
opacity = 1, fillOpacity = 0.2
) %>%
addCircleMarkers( # locations as points
lng = lmb_sf$lon, lat = lmb_sf$lat, # xy
radius = 5, stroke = FALSE, # marker design
fillOpacity = 0.5, fillColor = "#0000FF", # marker colours
clusterOptions = markerClusterOptions(), # bunch-up markers
popup = ~paste0( # dynamic HTML-creation for popup content
emo::ji("round_pushpin"), " ", lmb_sf$lat, ", ", lmb_sf$lon, "<br>",
emo::ji("postbox"), lmb_sf$admin_district,
", ", lmb_sf$postcode, "<br>",
emo::ji("bird"), " <a href='https://twitter.com/londonmapbot/status/",
lmb_sf$status_id, "'>Tweet</a><br>",
emo::ji("world_map"), " ", "<a href='",
lmb_sf$osm_url, "' width='100%'>OpenStreetMap</a><br><br>",
"<img src='", lmb_sf$media_url, "' width='200'>"
)
)
The markers, which are blue dots, have rich pop-ups when clicked. The information is generated dynamically for each point by pasting HTML strings with the content of the dataframe. Props to Matt Kerlogue’s narrowbotR, which uses this emoji-info layout in its automated tweets.
To keep the design simple and uncluttered, I’ve intentionally used a muted base map (‘Positron’ from CartoDB) and limited the amount of pop-up content.
In the pop-up you’ll see information from the tweet, including the satellite image and printed coordinates; URLS to the original tweet and OpenStreetMap; plus the reverse-geocoded info we got from {PostcodesioR}.
Since there are thousands of points, it makes sense to cluster them with markerClusterOptions()
to avoid graphical and navigational troubles. Click a cluster to expand until you reach a marker.
The map
lmb_map
If you can’t see the satellite photos in each pop-up you may need to change browser.
And no, it hasn’t captured my house yet!
Development
I made this for my own amusement and as an excuse to use {PostcodesioR} and reacquaint myself with {leaflet}. If I were going to develop it, I would make a Shiny app that continuously refreshes with the latest tweet information. I may revisit londonmapbot in future, or create a new bot; in which case the reverse geocoding capabilities of {PostcodesioR} could come in handy for providing more content in tweet text.
Session info
## ─ Session info ───────────────────────────────────────────────────────────────
## setting value
## version R version 4.0.2 (2020-06-22)
## os macOS Mojave 10.14.6
## system x86_64, darwin17.0
## ui X11
## language (EN)
## collate en_GB.UTF-8
## ctype en_GB.UTF-8
## tz Europe/London
## date 2020-12-29
##
## ─ Packages ───────────────────────────────────────────────────────────────────
## package * version date lib source
## askpass 1.1 2019-01-13 [1] CRAN (R 4.0.0)
## assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.0)
## backports 1.1.8 2020-06-17 [1] CRAN (R 4.0.0)
## blob 1.2.1 2020-01-20 [1] CRAN (R 4.0.2)
## blogdown 0.19 2020-05-22 [1] CRAN (R 4.0.0)
## bookdown 0.19 2020-05-15 [1] CRAN (R 4.0.0)
## broom 0.7.0 2020-07-09 [1] CRAN (R 4.0.2)
## cellranger 1.1.0 2016-07-27 [1] CRAN (R 4.0.2)
## class 7.3-17 2020-04-26 [1] CRAN (R 4.0.2)
## classInt 0.4-3 2020-04-07 [1] CRAN (R 4.0.0)
## cli 2.2.0 2020-11-20 [1] CRAN (R 4.0.2)
## colorspace 2.0-0 2020-11-11 [1] CRAN (R 4.0.2)
## crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.0)
## crosstalk 1.1.0.1 2020-03-13 [1] CRAN (R 4.0.0)
## crul 1.0.0 2020-07-30 [1] CRAN (R 4.0.2)
## curl 4.3 2019-12-02 [1] CRAN (R 4.0.0)
## DBI 1.1.0 2019-12-15 [1] CRAN (R 4.0.0)
## dbplyr 1.4.4 2020-05-27 [1] CRAN (R 4.0.2)
## digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.2)
## dplyr * 1.0.0 2020-08-10 [1] Github (tidyverse/dplyr@5e3f3ec)
## e1071 1.7-3 2019-11-26 [1] CRAN (R 4.0.0)
## ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.0)
## emo 0.0.0.9000 2020-05-24 [1] Github (hadley/emo@3f03b11)
## evaluate 0.14 2019-05-28 [1] CRAN (R 4.0.0)
## fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.0)
## farver 2.0.3 2020-01-16 [1] CRAN (R 4.0.0)
## forcats * 0.5.0 2020-03-01 [1] CRAN (R 4.0.2)
## foreign 0.8-80 2020-05-24 [1] CRAN (R 4.0.2)
## fs 1.5.0 2020-07-31 [1] CRAN (R 4.0.2)
## generics 0.1.0 2020-10-31 [1] CRAN (R 4.0.2)
## geojson 0.3.4 2020-06-23 [1] CRAN (R 4.0.2)
## geojsonio * 0.9.2 2020-04-07 [1] CRAN (R 4.0.2)
## ggplot2 * 3.3.2 2020-06-19 [1] CRAN (R 4.0.2)
## glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2)
## gtable 0.3.0 2019-03-25 [1] CRAN (R 4.0.0)
## haven 2.3.1 2020-06-01 [1] CRAN (R 4.0.2)
## hms 0.5.3 2020-01-08 [1] CRAN (R 4.0.2)
## htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.2)
## htmlwidgets 1.5.3 2020-12-10 [1] CRAN (R 4.0.2)
## httpcode 0.3.0 2020-04-10 [1] CRAN (R 4.0.2)
## httr 1.4.2 2020-07-20 [1] CRAN (R 4.0.2)
## jqr 1.2.0 2020-11-13 [1] CRAN (R 4.0.2)
## jsonlite 1.7.2 2020-12-09 [1] CRAN (R 4.0.2)
## KernSmooth 2.23-17 2020-04-26 [1] CRAN (R 4.0.2)
## knitr 1.30 2020-09-22 [1] CRAN (R 4.0.2)
## labeling 0.4.2 2020-10-20 [1] CRAN (R 4.0.2)
## lattice 0.20-41 2020-04-02 [1] CRAN (R 4.0.2)
## lazyeval 0.2.2 2019-03-15 [1] CRAN (R 4.0.0)
## leaflet * 2.0.3 2019-11-16 [1] CRAN (R 4.0.0)
## leaflet.providers 1.9.0 2019-11-09 [1] CRAN (R 4.0.0)
## lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.0)
## lubridate 1.7.9.2 2020-11-13 [1] CRAN (R 4.0.2)
## magrittr 2.0.1 2020-11-17 [1] CRAN (R 4.0.2)
## maptools 1.0-2 2020-08-24 [1] CRAN (R 4.0.2)
## modelr 0.1.8 2020-05-19 [1] CRAN (R 4.0.2)
## munsell 0.5.0 2018-06-12 [1] CRAN (R 4.0.0)
## openssl 1.4.3 2020-09-18 [1] CRAN (R 4.0.2)
## pillar 1.4.7 2020-11-20 [1] CRAN (R 4.0.2)
## pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.0)
## PostcodesioR * 0.1.1 2019-08-24 [1] CRAN (R 4.0.2)
## purrr * 0.3.4 2020-04-17 [1] CRAN (R 4.0.0)
## R6 2.5.0 2020-10-28 [1] CRAN (R 4.0.2)
## Rcpp 1.0.5 2020-07-06 [1] CRAN (R 4.0.2)
## readr * 1.4.0 2020-10-05 [1] CRAN (R 4.0.2)
## readxl 1.3.1 2019-03-13 [1] CRAN (R 4.0.2)
## reprex 0.3.0 2019-05-16 [1] CRAN (R 4.0.2)
## rgdal 1.5-18 2020-10-13 [1] CRAN (R 4.0.2)
## rgeos 0.5-3 2020-05-08 [1] CRAN (R 4.0.0)
## rlang 0.4.9 2020-11-26 [1] CRAN (R 4.0.2)
## rmarkdown 2.5 2020-10-21 [1] CRAN (R 4.0.2)
## rstudioapi 0.13 2020-11-12 [1] CRAN (R 4.0.2)
## rtweet * 0.7.0 2020-01-08 [1] CRAN (R 4.0.2)
## rvest 0.3.6 2020-07-25 [1] CRAN (R 4.0.2)
## scales 1.1.1 2020-05-11 [1] CRAN (R 4.0.0)
## sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.0)
## sf * 0.9-4 2020-06-13 [1] CRAN (R 4.0.0)
## sp 1.4-2 2020-05-20 [1] CRAN (R 4.0.0)
## stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2)
## stringr * 1.4.0 2019-02-10 [1] CRAN (R 4.0.0)
## tibble * 3.0.4 2020-10-12 [1] CRAN (R 4.0.2)
## tidyr * 1.1.2 2020-08-27 [1] CRAN (R 4.0.2)
## tidyselect 1.1.0 2020-05-11 [1] CRAN (R 4.0.0)
## tidyverse * 1.3.0 2019-11-21 [1] CRAN (R 4.0.2)
## triebeard 0.3.0 2016-08-04 [1] CRAN (R 4.0.2)
## units 0.6-7 2020-06-13 [1] CRAN (R 4.0.0)
## urltools 1.7.3 2019-04-14 [1] CRAN (R 4.0.2)
## utf8 1.1.4 2018-05-24 [1] CRAN (R 4.0.0)
## V8 3.4.0 2020-11-04 [1] CRAN (R 4.0.2)
## vctrs 0.3.6 2020-12-17 [1] CRAN (R 4.0.2)
## withr 2.3.0 2020-09-22 [1] CRAN (R 4.0.2)
## xfun 0.19 2020-10-30 [1] CRAN (R 4.0.2)
## xml2 1.3.2 2020-04-23 [1] CRAN (R 4.0.0)
## yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.0)
##
## [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library
I’m going to assume this is also a wrestling move.↩︎
Technically we’re using the NUTS1 regional designations, a Eurostat standard.↩︎