Working with Mapillary Data¶
This short notebook demostrates how landlensdb can be used to load, process, visualize, and store street-view data from Mapillary servers.
Pre-requisites¶
Before getting started, you will need to have PostgreSQL and PostGIS installed.
PostgreSQL and PostGIS¶
Once PostgreSQL and PostGIS are installed, or if they are already installed, then you will need to have a PostGIS enabled PostgreSQL database to work with. To create one, use:
createdb <database_name> && psql <database_name> -c "CREATE EXTENSION POSTGIS"
Be sure to replace <database_name> with the name you want to call your database. For example, it could be:
createdb landlens && psql landlens -c "CREATE EXTENSION POSTGIS"
Mapillary Token¶
You will also need to have a Mapillary API token. This isnt necessary to use the library, but you will need it to follow the tutorial, which includes examples on connecting to Mapillary and downloading images.
.env File¶
Finally, you will need to save this information into a .env file. To create a .env file, use:
touch .env
Then, paste the following:
MLY_TOKEN=<token>
Replace the text enclosed by angular brackets <> with the actual text. For example, your MLY_TOKEN would be the Mapillary token you can acquire from a Mapillary account. Usually, these start with MLY|. The path variables are absolute paths to where you want files to be read or written. The database URL, if the database was created using the earlier instructions, would look like: postgresql://localhost:5432/landlens, assuming PostgreSQL is being served on localhost with port 5432 (the default). The DB_TABLE will be the table name to use in this tutorial. It can be anything, for example: mapillary_images.
Once this is done, you should be ready to proceed with the tutorial.
If you don't have landlens_db installed, you can do so with pip install landlensdb.
from landlensdb.handlers.cloud import Mapillary
from landlensdb.process.snap import create_bbox, get_osm_lines, snap_to_road_network
from landlensdb.handlers.db import Postgres
from landlensdb.geoclasses.geoimageframe import GeoImageFrame
Before we get started, we will need to load our Mapillary API token and other environmental variables. For simplicity, we will use the dotenv library to please install this and create a .env file to follow this tutorial. You will also need to make sure that pandas and geopandas are installed in order to manipulate some of the data required for the tutorial.
import os
import geopandas as gpd
import glob
import pandas as pd
from dotenv import load_dotenv
load_dotenv()
MLY_TOKEN = os.environ.get("MLY_TOKEN")
Loading Images from Mapillary¶
landlensdb was made to work with Mapillary data and it includes helper functions to make calls to the Mapillary API and download and convert Mapillary data into a format for landlensdb.
To use landlensdb to fetch data from Mapillary, you first need to initialize a Mapillary connection using your Mapillary Secret Token.
importer = Mapillary(MLY_TOKEN)
landlensdb offers a few functions to filter Mapillary data from their API. However, for more advanced filtering, we recommend that users use the mapillary-python-sdk and convert the resulting data into a GeoImageFrame.
Here is an example of how to load data using the fetch_by_id method of landlensdb:
image_id = 915374089313107
image = importer.fetch_by_id(image_id)
image
| altitude | atomic_scale | camera_parameters | camera_type | captured_at | compass_angle | computed_altitude | computed_compass_angle | computed_geometry | computed_rotation | ... | merge_cc | mesh | sequence | sfm_cluster | width | detections | quality_score | mly_id | name | image_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 41.782 | 1.002665 | 0.61739578749889,0.26131500830183,0.1242660260... | fisheye | 2019-10-23T22:29:42+09:00 | 99.299232 | 1.795589 | 102.951814 | POINT (140.95153462743 42.329677227362) | -1.0627190885041,-0.84029284280692,-1.15538369... | ... | 1.926644e+18 | {'id': '313263440182706', 'url': 'https://scon... | emgV_2cwMSoW9w7fkg7xJQ | {'id': '169747341731652', 'url': 'https://scon... | 4000 | {'data': [{'id': '916266259223890'}, {'id': '9... | 0.933333 | 915374089313107 | mly|915374089313107 | https://scontent.fhnl3-2.fna.fbcdn.net/m1/v/t6... |
1 rows × 23 columns
By default, landlensdb will download all fields from the Mapillary image endpoint and default to thumb_1024_url as the image_url, however, you may specify a subset of fields using the fields argument and only these fields will be downloaded. Note, you must supply at least the id, geometry, and one of the image url fields.
For example, using the fetch_within_bbox method of landlensdb:
bbox = [139.59,35.865358, 139.719, 35.882781]
start = '2022-03-16'
end = '2022-03-16'
fields = ['id', 'altitude', 'captured_at', 'camera_type', 'quality_score', 'thumb_1024_url',
'compass_angle', 'computed_compass_angle', 'computed_geometry', 'geometry']
images = importer.fetch_within_bbox(bbox, start_date=start, end_date=end, fields=fields,
max_images=100,# for limiting fetching images
max_workers=10)
images.head()
Fetching 14 tiles... Reached maximum number of images (100), stopping tile fetching Found 3996 total images After removing duplicates: 3996 unique images Limiting to 100 images for processing
Fetching metadata: 100%|██████████| 100/100 [00:05<00:00, 17.38it/s]
| altitude | captured_at | camera_type | quality_score | compass_angle | computed_compass_angle | computed_geometry | geometry | mly_id | name | image_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 38.502000 | 2022-11-07T01:11:41.352000+09:00 | perspective | 0.000000 | 304.154000 | 308.163758 | POINT (139.58766729639 35.888444642159) | POINT (139.5877 35.8884) | 656828185851650 | mly|656828185851650 | https://scontent.fhnl3-2.fna.fbcdn.net/m1/v/t6... |
| 1 | 14.000000 | 2024-01-19T12:03:23+09:00 | perspective | 0.000000 | 306.000000 | 312.757495 | POINT (139.57239765118 35.872508855934) | POINT (139.57244 35.87257) | 845839730628147 | mly|845839730628147 | https://scontent.fhnl3-2.fna.fbcdn.net/m1/v/t6... |
| 2 | -2.399936 | 2018-03-18T03:36:35.152000+09:00 | perspective | 0.764484 | 310.674866 | 300.832196 | POINT (139.59155887209 35.886212671385) | POINT (139.59158 35.88623) | 300714221533914 | mly|300714221533914 | https://scontent.fhnl3-2.fna.fbcdn.net/m1/v/t6... |
| 3 | 16.115421 | 2020-02-21T13:46:01.550000+09:00 | perspective | 0.904487 | 0.000000 | 335.390392 | POINT (139.57252004801 35.872856102707) | POINT (139.57252 35.87285) | 165253005445392 | mly|165253005445392 | https://scontent.fhnl3-2.fna.fbcdn.net/m1/v/t6... |
| 4 | 54.591000 | 2024-04-02T09:44:00+09:00 | perspective | 0.000000 | 269.794000 | 276.320324 | POINT (139.58541302864 35.877485024935) | POINT (139.58543 35.87749) | 959520636195873 | mly|959520636195873 | https://scontent.fhnl3-2.fna.fbcdn.net/m1/v/t6... |
It is also important to realize that Mapillary image urls are not permanent. So, landlensdb offers a method to download Mapillary images and return a new GeoImageFrame with the updated the image_url to the new location. It is good practice to always use the full path to file locations.
relative_path = "mapillary"
absolute_path = os.path.abspath(relative_path)
images = images.download_images_to_local(absolute_path, filename_column='name', max_workers=10)
images.head()
Downloading images: 100%|██████████| 100/100 [00:01<00:00, 56.39it/s]
| altitude | captured_at | camera_type | quality_score | compass_angle | computed_compass_angle | computed_geometry | geometry | mly_id | name | image_url | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 38.502000 | 2022-11-07T01:11:41.352000+09:00 | perspective | 0.000000 | 304.154000 | 308.163758 | POINT (139.58766729639 35.888444642159) | POINT (139.5877 35.8884) | 656828185851650 | mly|656828185851650 | /Users/iosefa/repos/landlensdb/docs/examples/m... |
| 1 | 14.000000 | 2024-01-19T12:03:23+09:00 | perspective | 0.000000 | 306.000000 | 312.757495 | POINT (139.57239765118 35.872508855934) | POINT (139.57244 35.87257) | 845839730628147 | mly|845839730628147 | /Users/iosefa/repos/landlensdb/docs/examples/m... |
| 2 | -2.399936 | 2018-03-18T03:36:35.152000+09:00 | perspective | 0.764484 | 310.674866 | 300.832196 | POINT (139.59155887209 35.886212671385) | POINT (139.59158 35.88623) | 300714221533914 | mly|300714221533914 | /Users/iosefa/repos/landlensdb/docs/examples/m... |
| 3 | 16.115421 | 2020-02-21T13:46:01.550000+09:00 | perspective | 0.904487 | 0.000000 | 335.390392 | POINT (139.57252004801 35.872856102707) | POINT (139.57252 35.87285) | 165253005445392 | mly|165253005445392 | /Users/iosefa/repos/landlensdb/docs/examples/m... |
| 4 | 54.591000 | 2024-04-02T09:44:00+09:00 | perspective | 0.000000 | 269.794000 | 276.320324 | POINT (139.58541302864 35.877485024935) | POINT (139.58543 35.87749) | 959520636195873 | mly|959520636195873 | /Users/iosefa/repos/landlensdb/docs/examples/m... |
| altitude | captured_at | camera_type | compass_angle | computed_compass_angle | computed_geometry | geometry | mly_id | name | image_url | quality_score | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 14.768889 | 2020-02-21T13:44:34.569000+09:00 | perspective | 9.554391 | 351.504697 | POINT (139.57339081065 35.871402809966) | POINT (139.5734 35.87141) | 864971387703143 | mly|864971387703143 | test_images/downloads/mly|864971387703143.jpg | 0.7 |
| 1 | 16.000000 | 2024-01-19T12:01:50+09:00 | perspective | 247.000000 | 310.752912 | POINT (139.57289114328 35.87219131719) | POINT (139.57279 35.87224) | 336839779179779 | mly|336839779179779 | test_images/downloads/mly|336839779179779.jpg | 0.7 |
| 2 | 48.680551 | 2020-08-01T12:01:21.643000+09:00 | perspective | 358.808411 | 1.589768 | POINT (139.58462410187 35.883294681154) | POINT (139.58463 35.8833) | 5653252761383733 | mly|5653252761383733 | test_images/downloads/mly|5653252761383733.jpg | 0.7 |
| 3 | 11.000000 | 2024-01-19T12:04:53+09:00 | perspective | 246.000000 | 55.156704 | POINT (139.57205573147 35.872939770667) | POINT (139.57206 35.87295) | 344214351810374 | mly|344214351810374 | test_images/downloads/mly|344214351810374.jpg | 0.7 |
| 4 | 9.022069 | 2020-03-09T13:32:58.202000+09:00 | perspective | 194.887604 | 168.303653 | POINT (139.58402274235 35.877453641423) | POINT (139.58415 35.87735) | 508816543794292 | mly|508816543794292 | test_images/downloads/mly|508816543794292.jpg | 0.7 |
Loading data from arbitrary sources¶
It is also possible to read from any OGC-recognized vector file format, including ESRI shapefile, geojson, and geopackage, or to create a GeoImageFrame in the same manner as a geopandas dataframe by initializing it with data so long as it has a name, image_url, and geometry column.
Data can also be imported from a PostreSQL postGIS enabled database. There is more information below on creating and exporting postgres tables for landlensdb.
When reading from postgres, it can be beneficial to load a subset of data. This can be important when the database contains upwards of tens of thousands of images. For this purpose, there are several database utility and query functions to select only a subset of the data in the database.
Processing Images¶
Now that we have loaded some data, we can perform some simple processing on the images. Check the documentation for the current processing functions available. Here is an example of how landlensdb can be used to snap images to road networks.
First, we need a road network to snap your images to. landlensdb also offers a helper function to download road networks from Open Street Map within a given bounding box.
bbox = images['geometry'].total_bounds
network = get_osm_lines(bbox)
Then, calling the snap_to_road_network will snap all points to the closest road network (within the provided threshold distance) and will create a new geometry column in the GeoImageFrame falled snapped_geometry to represent this new point.
snap_to_road_network(images, 100, network)
/Users/iosefa/repos/landlensdb/landlensdb/process/snap.py:232: UserWarning:
Not all images were snapped. Non-snapped images will not be added
to the snapped image table. To snap all images, try increasing the threshold
or changing the road network. The following images could not be snapped
to a road network: ['/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|341189448731335.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|2894556887433291.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|845839730628147.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|165253005445392.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|1578427066239162.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|4073999156023160.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|2022903504790710.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|798037910842482.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|321170896035731.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|135578586316907.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|977004729506385.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|151439446992900.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|366394412691407.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|2196359430499964.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|1074933626880597.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|259857353794309.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|1780421159052050.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|470266004254253.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|739435500086975.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|3529384387315502.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|344214351810374.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|7037649352990242.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|284181453385106.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|1348429235810254.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|163483822448611.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|266023851890350.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|882539549271017.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|250040466902808.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|298938741783046.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|463760911393728.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|1083992052788441.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|489004355634199.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|958784518225045.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|146933804077187.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|3877177859055675.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|1138747620595199.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|841487969775838.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|398261285903268.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|7275967072425300.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|1400369537274529.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|309864044006486.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|243705220835241.jpg', '/Users/iosefa/repos/landlensdb/docs/examples/mapillary/mly|178622602000292.jpg']
warnings.warn(
/Users/iosefa/repos/landlensdb/landlensdb/process/snap.py:129: UserWarning:
GeodataImageFrame contains rows with empty snapped_geometry. Non-snapped images will be skipped.
To snap all images, try increasing the threshold or changing the road network.
warnings.warn(
| altitude | captured_at | camera_type | compass_angle | computed_compass_angle | computed_geometry | geometry | mly_id | name | image_url | quality_score | snapped_geometry | snapped_angle | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 5.157323 | 2020-05-17T09:34:25.224000+09:00 | perspective | 345.032135 | 64.252294 | POINT (139.58263261308 35.877839236796) | POINT (139.58263 35.87783) | 299088425109738 | mly|299088425109738 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | POINT (139.58263 35.87784) | 247.485127 |
| 1 | 38.502000 | 2022-11-07T01:11:41.352000+09:00 | perspective | 304.154000 | 308.163758 | POINT (139.58766729639 35.888444642159) | POINT (139.5877 35.8884) | 656828185851650 | mly|656828185851650 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | POINT (139.5877 35.8884) | 304.460467 |
| 2 | 54.591000 | 2024-04-02T09:44:00+09:00 | perspective | 269.794000 | 276.320324 | POINT (139.58541302864 35.877485024935) | POINT (139.58543 35.87749) | 959520636195873 | mly|959520636195873 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | POINT (139.58543 35.87748) | 268.479139 |
| 3 | 12.000000 | 2024-01-19T12:04:21+09:00 | perspective | 319.000000 | 330.582321 | POINT (139.57227883013 35.872625965843) | POINT (139.57232 35.87267) | 341189448731335 | mly|341189448731335 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | None | NaN |
| 4 | 5.500537 | 2020-05-17T09:28:41.965000+09:00 | perspective | 343.608521 | 81.007946 | POINT (139.57913973808 35.876375848152) | POINT (139.57815 35.87591) | 2894556887433291 | mly|2894556887433291 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | None | NaN |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 95 | 14.880390 | 2020-02-21T13:44:53.100000+09:00 | perspective | 331.856595 | 336.501893 | POINT (139.57314536276 35.871769469654) | POINT (139.57314 35.87174) | 309864044006486 | mly|309864044006486 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | None | NaN |
| 96 | 4.887280 | 2020-05-17T09:19:59.208000+09:00 | perspective | 353.266785 | 335.405193 | POINT (139.57910854058 35.872199360711) | POINT (139.57911 35.8722) | 243705220835241 | mly|243705220835241 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | None | NaN |
| 97 | 7.213029 | 2020-05-17T09:41:26.497000+09:00 | perspective | 90.242986 | 84.969101 | POINT (139.58843748084 35.879393276315) | POINT (139.58844 35.87939) | 520722175762090 | mly|520722175762090 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | POINT (139.58844 35.8794) | 88.965971 |
| 98 | 10.585542 | 2020-03-09T13:34:18.698000+09:00 | perspective | 214.312256 | 75.909129 | POINT (139.58375978158 35.878759370556) | POINT (139.58361 35.87855) | 956597638469064 | mly|956597638469064 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | POINT (139.58372 35.8786) | 150.474872 |
| 99 | 10.000000 | 2024-01-19T12:18:53+09:00 | perspective | 224.000000 | 26.167133 | POINT (139.57051706948 35.872654878585) | POINT (139.5705 35.87266) | 178622602000292 | mly|178622602000292 | /Users/iosefa/repos/landlensdb/docs/examples/m... | 0.7 | None | NaN |
100 rows × 13 columns
Snapping to a local road network¶
It is also possible to load your own road network and snap to this. When doing this, it is important that all the image points are within a reasonable distance from any given road in your network and that the threshold is appropriately set. If you suspect that there are images far from a road, and you do need that image to be snapped to the closest road, then be sure to set a high enough threshold.
Here is an example of how this can be achieved:
roads_path = 'data/roads/*.shp'
road_files = glob.glob(roads_path)
roads = [gpd.read_file(road) for road in road_files]
network = pd.concat(roads, ignore_index=True)
snap_to_road_network(images, 100, network, realign_camera=True)
Visualizing Images¶
landlens_db provides a simple way to visualize its GeoImageFrames interactively using Folium. The map method of a GeoImageFrame will plot all images as markers on a map and will display the image on click along with any metadata set using the additional_properties argument as well as markers for any provided additional geometry.
images.map(
additional_properties=['altitude', 'camera_type'],
additional_geometries=[
{'geometry': 'computed_geometry', 'angle': 'computed_compass_angle', 'label': 'Computed'},
{'geometry': 'snapped_geometry', 'angle': 'snapped_angle', 'label': 'Snapped'},
]
)
/Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 3 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 4 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 5 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 7 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 8 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 11 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 12 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 21 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 23 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 27 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 30 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 31 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 32 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 33 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 35 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 37 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 38 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 44 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 45 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 47 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 48 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 51 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 54 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 55 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 56 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 60 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 63 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 64 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 66 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 67 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 69 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 72 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 73 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 79 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 81 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 82 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 84 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 87 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 89 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 94 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 95 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 96 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn( /Users/iosefa/repos/landlensdb/landlensdb/geoclasses/geoimageframe.py:442: UserWarning: Item at index 99 in 'snapped_geometry' is not a valid Point. Skipping. warnings.warn(
Storing Images¶
GeoImageFrame data can be stored in a variety of formats. Given that it is built on GeoPandas the GeoDataFrame class, it will take any geodataframe method to save data. For instance, to save a table as a geopackage, we simply call:
images.to_file('images_tutorial.gpkg')
However, in the current version when reading a saved vector format it is important to then initialize the GeoDataFrame as a GeoImageFrame if you want to make use of the features of landlensdb. For example:
images_gdf = gpd.read_file('images_tutorial.gpkg')
images = GeoImageFrame(images_gdf)
Saving to a PostgreSQL Database¶
landlens_db also offers functionality to store data in a PostGIS enabled PostgreSQL database. This is done by extending the to_postgis method of GeoPandas. There are some constraints, such as unique image_urls, that are automatically applied when storing data, as well as some data validity checks -- see the documentation for details.
To save a GeoImageFrame to a PostgreSQL table, you will need to first initiate a connection to a PostgreSQL database. You can do this using the ImageDB class:
db_con = Postgres('postgresql://localhost:5432/landlens')
This database must already exist and have PostGIS loaded.
Then, you can save using to_postgis:
images.to_postgis('tutorial', db_con.engine, if_exists="replace")
Updating an Existing Table¶
When saving to PostgreSQL, you can choose to handle existing tables. to_postgis offers the same fail, replace and append methods that GeoPandas offers, however, append requires that all data going in will not conflict with any existing data. Instead, it is possible to "upsert" (insert and update) data into existing tables using the upsert_images class method of Image_DB. You may choose to either update conflicting records or skip them by declaring "update" or "nothing" in the conflict argument of the function.
db_con.upsert_images(images, 'tutorial', conflict='update')
Querying an Existing Table¶
It is also possible to load and filter data from existing postgres connections. landlensdb offers simple filter functions to query and filter tables to provide a subset of the data. This can be important when working with very large datasets. For example, to load all images with an altitude greater than 50:
high_altitude_images = db_con.table('tutorial').filter(altitude__gt=50).all()
high_alt_map = high_altitude_images.map(
additional_properties=['altitude', 'camera_type'],
additional_geometries=[
{'geometry': 'geometry', 'angle': 'compass_angle', 'label': 'Base Geometry'},
])
high_alt_map.save('query_map.html')
high_alt_map