pygeoapi-plugins

pygeoapi plugins developed by the Center for Geospatial Solutions

OGC API - Features

Additional OGC API - Feature providers are listed below, along with a matrix of supported query parameters.

Provider	Property Filters/Display	Result Type	BBox	Datetime	Sort By	Skip Geometry	CQL	Transactions	CRS
CKAN	✅/✅	results/hits	❌	❌	✅	✅	❌	❌	✅
PseudoPostgreSQL	✅/✅	results/hits	✅	✅	✅	✅	✅	❌	✅
PostgreSQLGeoJSON	✅/✅	results/hits	✅	✅	✅	✅	✅	❌	❌
SPARQL	❌/✅	results/hits	❌	❌	❌	❌	❌	❌	❌
GeoPandas	✅/✅	results/hits	✅	✅	✅	✅	❌	✅	✅

The provider names listed in the table are only accessible in internetofwater/pygeoapi, otherwise the full python path is required.

CKAN

The CKAN Provider enables OGC API - Feature support at the collection level for a specific resource within the datastore-search endpoints of CKAN instances. It allows you to integrate CKAN resources into your pygeoapi instance. The provider definition for the CKAN Provider includes configuration options specific to CKAN. To use the CKAN Provider, you need to specify pygeoapi_plugins.provider.ckan.CKANProvider as the provider's name.

providers:
  - type: feature
    name: pygeoapi_plugins.provider.ckan.CKANProvider
    data: https://catalog.newmexicowaterdata.org/api/3/action/datastore_search
    resource_id: 08369d21-520b-439e-97e3-5ecb50737887
    id_field: _id
    x_field: LONDD
    y_field: LATDD

In this example, the CKAN Provider is configured to work with the specified CKAN resource.

• data: The URL endpoint for the datastore search API of the CKAN instance.
• resource_id: The identifier of the specific CKAN resource you want to access within the datastore.
• id_field: The field that serves as the unique identifier for features in the CKAN resource.
• x_field: The field representing the X-coordinate (longitude) for the features in the CKAN resource.
• y_field: The field representing the Y-coordinate (latitude) for the features in the CKAN resource.

PseudoPostgreSQL

The PseudoPostgreSQL Provider extends the PostgreSQL Provider in pygeoapi core with faster counting of numberMatched on large tables. Counting every row that matches a query can take longer than the query itself. Instead, the provider reads the number of rows the PostgreSQL query planner estimates the query will return, using EXPLAIN. If the estimate is below a limit, the provider runs a precise count instead, so numberMatched is exact for smaller results.

• The limit is set with the PSEUDO_COUNT_LIMIT environment variable (default 5000000). The original misspelling, PSUEDO_COUNT_LIMIT, is also accepted.
• Requests with resulttype=hits always return a precise count.
• If the estimate cannot be read, the provider logs a warning and falls back to a precise count.

numberMatched is approximate above the limit, as accurate as the table statistics PostgreSQL keeps. Running ANALYZE on the table keeps the estimate current.

To use the PseudoPostgreSQL Provider, you need to specify pygeoapi_plugins.provider.postgresql.PseudoPostgreSQLProvider as the provider's name. The rest of the configuration is the same as the PostgreSQL Provider in pygeoapi core.

providers:
  - type: feature
    name: pygeoapi_plugins.provider.postgresql.PseudoPostgreSQLProvider
    data:
      host: 127.0.0.1
      port: 5432
      dbname: test
      user: postgres
      password: ${POSTGRESQL_PASSWORD}
      search_path: [osm, public]
    id_field: osm_id
    table: hotosm_bdi_waterways
    geom_field: foo_geom

PostgreSQL GeoJSON

The PostgreSQL GeoJSON Provider extends the PostgreSQL Provider in pygeoapi core by rendering geometries as GeoJSON in the database, using PostGIS ST_AsGeoJSON. The core provider loads each geometry and converts it to GeoJSON in Python, which is slow for large or complex geometries. Rendering in the database skips that conversion, so feature responses build faster. Pair it with the JSON formatter to also skip re-encoding the GeoJSON for f=json.

Geometries are returned in the table's own coordinate reference system: ST_AsGeoJSON does not reproject, so the crs query parameter is not supported. Store geometries in WGS 84 (EPSG:4326) to serve standard GeoJSON.

To use the PostgreSQL GeoJSON Provider, you need to specify pygeoapi_plugins.provider.postgresql.PostgreSQLGeoJSONProvider as the provider's name. The rest of the configuration is the same as the PostgreSQL Provider in pygeoapi core.

providers:
  - type: feature
    name: pygeoapi_plugins.provider.postgresql.PostgreSQLGeoJSONProvider
    data:
      host: 127.0.0.1
      port: 5432
      dbname: test
      user: postgres
      password: ${POSTGRESQL_PASSWORD}
      search_path: [osm, public]
    id_field: osm_id
    table: hotosm_bdi_waterways
    geom_field: foo_geom
    max_decimal_digits: 6 # optional, default 9

max_decimal_digits sets how many decimal places ST_AsGeoJSON rounds coordinates to (default 9). Fewer digits make smaller responses; in degrees, 6 decimal places is about 0.1 m.

SPARQL

The SPARQL Provider is a wrapper for any pygeoapi feature provider that provides additional context, allowing integration of SPARQL-based data sources into a pygeoapi instance. By wrapping another feature provider, the SPARQL Provider inherits queryable capacities from the wrapped feature provider - adding SPARQL context to each resulting feature. The provider definition for the SPARQL Provider is similar to that of the wrapped provider, with the addition of specific SPARQL-related configuration options. To use the SPARQL Provider, you need to specify pygeoapi_plugins.provider.sparql.SPARQLProvider as the provider's name.

providers:
  - # Normal pygeoapi provider configuration
    type: feature
    data: /pygeoapi_plugins/tests/data/places.csv
    id_field: index
    geometry:
      x_field: lon
      y_field: lat
    #
    name: pygeoapi_plugins.provider.sparql.SPARQLProvider
    sparql_provider: CSV # Name of provider SPARQL is wrapping
    sparql_query:
      endpoint: https://dbpedia.org/sparql
      bind:
        name: uri
        variable: '?subject'
      prefixes:
        '': <http://dbpedia.org/resource/>
        dbpedia2: <http://dbpedia.org/property/>
        dbo: <http://dbpedia.org/ontology/>
      where:
        - subject: '?subject'
          predicate: dbo:populationTotal
          object: '?population'
        - subject: '?subject'
          predicate: dbo:country
          object: '?country'
        - subject: '?subject'
          predicate: '<http://dbpedia.org/property/leaderName>'
          object: '?leader'
      filter:
        - 'FILTER (isIRI(?leader) || isLiteral(?leader))'

In this example, the SPARQL Provider wraps the GeoJSON Provider. The SPARQL Provider only uses variables prefixed with sparql_ in the configuration.

• data: The path to the data file used by the wrapped provider (GeoJSON Provider in this case).
• id_field: The field that serves as the unique identifier for features in the data.
• sparql_provider: The name of the provider that will handle the SPARQL query results (GeoJSON Provider in this case).
• sparql_query: The SPARQL object holding the content of the SPARQL query.
  • endpoint: The SPARQL variable representing the graph IRI in the query.
  • bind:
    • name: Field in the wrapped properties block to query the graph with
    • variable: The SPARQL variable used for querying (e.g., ?subject). prefixes:
  • prefixes: Optional dictionary defining the prefixes used in the SPARQL query.
  • where: A list of mappings that define the WHERE clause of the SPARQL query. Each mapping includes:
    • subject: The subject of the triple pattern.
    • predicate: The predicate of the triple pattern.
    • object: The object of the triple pattern.
  • filter: A list of SPARQL filter expressions to apply to the results.

GeoPandas

The GeoPandas Provider enables OGC API - Feature support using GeoPandas as the backend. This integration can read in data files in any of the geospatial formats supported by GeoPandas.

id_field is the only field that is required to be labeled.

providers:
  - type: feature
    name: pygeoapi_plugins.provider.geopandas_.GeoPandasProvider
    # Example data
    data: 'https://www.hydroshare.org/resource/3295a17b4cc24d34bd6a5c5aaf753c50/data/contents/hu02.gpkg'
    id_field: id

You can also use plain CSV and read in points by providing an x_field and y_field in the config the same way you would with the default pygeoapi CSV provider.

OGC API - Tiles

Additional OGC API - Tile providers are listed below

MVT PostgreSQL

The MVT PostgreSQL Provider extends the core provider, to include additional supports for rendering features efficiently at low zoom. The threshold for applying limits can be controlled with the configuration option disable_at_z. The default value for the zoom threshold is 6.

The first way to filter the features that are rendered is by setting a minimum pixel size. The configuration option min_pixel provider will drop features that are less than a pixel in the rendered tile. The default value for minimum pixel size is 512. The following configuration would render half pixel features until zoom level 10.

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_postgresql.MVTPostgreSQLProvider_
    ...
    disable_at_z: 10
    mix_pixel: 256 # a full pixel in the tile

The second way to filter the features that are rendered in a tile is using a modified CQL expression. The CQL expression can be used to determine render priority based on non-geographic attributes of a feature The following configuration would only render features with a population larger than 100,000:

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_postgresql.MVTPostgreSQLProvider_
    ...
    tile_threshold: "population > 100000"

The CQL expression can be formatted with a z value (minimum zoom is 1). This allows the parameter filter to be dynamic based on the zoom level of the tile. The following configuration would render tiles of successively smaller populations for every subsequent zoom level increase:

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_postgresql.MVTPostgreSQLProvider_
    ...
    disable_at_z: 5
    tile_threshold: "population_served_count > 200000 / ({z} * 2)"
    # z{0}: population_served_count > 100000
    # z{1}: population_served_count > 100000
    # z{2}: population_served_count > 50000
    # z{3}: population_served_count > 33333
    # z{4}: population_served_count > 25000

The third configuration option tile_limit can be specified to enforce a maximum number of features in a single tile. This will apply to all tiles regardless of if the other filters are enabled by disable_at_z. Features will be ordered by the size of there bounding box, pruning the smallest feature until the feature limit is met for the tile. Point-based geometry are pruned at random.

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_postgresql.MVTPostgreSQLProvider_
    ...
    disable_at_z: 0 # Apply no CQL or Pixel Size filter
    tile_limit: 1000 # No more than 1000 features in a single tile

The fourth configuration option simplify_geometry can be specified to reduce the number of vertices at low zooms. This will apply to all tiles regardless of if the other filters are enabled by disable_at_z.

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_postgresql.MVTPostgreSQLProvider_
    simplify_geometry: true
    # Simplification tolerance value scales with zoom level:
    #   z{0}: tolerance = 0.1
    #   z{2}: tolerance = 0.1
    #   z{4}: tolerance = 0.01
    #   z{6}: tolerance = 0.001
    #   z{8}: tolerance = 0.0001
    simplify_method: ST_SimplifyPreserveTopology # default
    # Must be one of:
    #   ST_Simplify
    #   ST_SimplifyPreserveTopology
    #   ST_SimplifyVW
    #   ST_SnapToGrid
    ...

The fifth configuration option tile_limit can be specified to enforce a maximum size (in Mb) for a single tile. This will apply to all tiles regardless of if the other filters are enabled by disable_at_z.

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_postgresql.MVTPostgreSQLProvider_
    ...
    disable_at_z: 0 # Apply no CQL or Pixel Size filter
    tile_size: 10 # Tile size shall be no more than 10Mb

MVT PostgreSQL with Caching

There are two additional Postgres based MVT providers with caching of tiles to prevent significant server load on tile generation at low zoom levels. Both providers use the configuration option disable_cache_at_z to prevent storing tiles at high zoom levels if undesired.

Filesystem Cache

The filesystem cache uses a local filesystem to store pbf blobs in a typical z/y/x directory tree.

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_cache.MVTPostgresFilesystem
    ...
    cache_directory: /tmp/mvt_cache # Default directory if not specified
    disable_cache_at_z: 6 # Default value to disable caching if not specified

PostgreSQL Table Cache

The table cache uses an external table to store pbf blobs in a typical z/y/x relational database. Make sure the credentials have access to update the table and write if necessary.

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_cache.MVTPostgresCache
    ...
    table: osm_waterways
    mvt_table: osm_waterways_mvt # Default table is table with `_mvt` appended
    disable_cache_at_z: 25 # Cache everything (eventually)

Redis Cache

The Redis cache uses a combined tile key for the lookup in a Redis service.

providers:
  - type: tile
    name: pygeoapi_plugins.provider.mvt_cache.MVTPostgresRedis
    ...
    redis_host: ${REDIS_HOST} # Use REDIS_HOST from environment
    redis_port: ${REDIS_PORT} # Use REDIS_PORT from environment

Pre-caching tiles

Tiles are cached the first time they are requested. To fill a cache ahead of time, run the mvt-cache pre-cache command with your pygeoapi configuration. It renders and caches every tile from each cached collection's minimum zoom up to (not including) disable_cache_at_z, for each tiling scheme. A collection whose cache already holds its first tile is skipped.

# All collections with an MVT cache provider
pygeoapi plugins mvt-cache pre-cache $PYGEOAPI_CONFIG

# Only some collections
pygeoapi plugins mvt-cache pre-cache $PYGEOAPI_CONFIG --collection osm_waterways

# The configuration file defaults to the PYGEOAPI_CONFIG environment variable
pygeoapi plugins mvt-cache pre-cache

This replaces the pre_cache provider option, which rendered the tiles when the provider was loaded.

OGC API - Processes

Additional OGC API - Processes are listed below

Intersector

The intersection process uses GDAL Python bindings to return features from a collection using an existing Geospatial OGR based file as the spatial mask.

An example configuration in a pygeoapi configuration is below.

intersector:
  type: process
  processor:
    name: pygeoapi_plugins.process.intersect.IntersectionProcessor

• collection is always required
• url or file is required
  • url is the remote source of the Geospatial file
  • file is the encoded bytes of the Geospatial file

This plugin is used in https:/reference.geoconnex.us/.

Sitemap Generator

The Sitemap Generator process makes use of the XML formatter and OGC API - Features to generate a sitemap of the pygeoapi instance. This can be used with the python package sitemap-generator to generate a sitemap index. An example configuration in a pygeoapi configuration is below.

sitemap-generator:
  type: process
  processor:
    name: pygeoapi_plugins.process.sitemap.SitemapProcessor

Formatters

The pygeoapi plugins repository includes a variety of custom formatters to modify the response body from OGC API - Feature and OGC API - EDR GeoJSON responses. The following formatters are supported:

• JSON: pygeoapi_plugins.formatter.JSON — returns GeoJSON (application/geo+json) for f=json, serialized with orjson. Faster than pygeoapi's default JSON output, and leaves < and > unescaped. See JSON below.
• JSON-FG: pygeoapi_plugins.formatter.JSONFG — returns JSON-FG compliant GeoJSON (application/geo+json). Useful for systems that require OGR-style JSON-FG output.
• Parquet: pygeoapi_plugins.formatter.Parquet — returns an Apache Parquet binary (application/vnd.apache.parquet). Good for analytical workflows and columnar storage.
• Shapefile: pygeoapi_plugins.formatter.Shapefile — returns a zipped Shapefile (application/zip).
• KML: pygeoapi_plugins.formatter.KML — returns KML formatted output (application/vnd.google-earth.kml+xml).
• GPKG: pygeoapi_plugins.formatter.GPKG — returns a GeoPackage (application/geopackage+sqlite3).
• PGDUMP: pygeoapi_plugins.formatter.PGDUMP — returns SQL dump suitable for importing into PostGIS (application/sql).
• XML (Sitemap): pygeoapi_plugins.formatter.XML — generates a sitemap XML from feature collections (application/xml).

Usage

• To enable any of these formatters for a resource, add them to the resource formatters list in your pygeoapi configuration. Example (see docker/pygeoapi.config.yml for a full example):

resources:
  my_collection:
    type: collection
    title: My Collection
    providers:
      - type: feature
        name: pygeoapi_plugins.provider.geopandas_.GeoPandasProvider
        data: /path/to/data.geojson
        id_field: id
    formatters:
      - name: pygeoapi_plugins.formatter.JSONFG
      - name: pygeoapi_plugins.formatter.Shapefile
      - name: pygeoapi_plugins.formatter.GPKG
      - name: pygeoapi_plugins.formatter.KML
      - name: pygeoapi_plugins.formatter.PGDUMP
      - name: pygeoapi_plugins.formatter.XML
      - name: pygeoapi_plugins.formatter.Parquet

JSON

The JSON formatter replaces pygeoapi's default f=json output for the items of a collection. It serializes with orjson, which is about 10 times faster than pygeoapi's default for large responses, and it does not escape < and > as < and >.

It works with any feature provider, and is fastest with the PostgreSQL GeoJSON Provider. That provider keeps each geometry as the GeoJSON text from ST_AsGeoJSON, and the JSON formatter writes that text straight into the response, without parsing it into Python objects and encoding it again. Other outputs, such as HTML and JSON-LD, still get ordinary geometries, parsed only when they are read.

resources:
  my_collection:
    type: collection
    title: My Collection
    providers:
      - type: feature
        name: pygeoapi_plugins.provider.postgresql.PostgreSQLGeoJSONProvider
        data:
          host: 127.0.0.1
          dbname: test
          user: postgres
          password: ${POSTGRESQL_PASSWORD}
        id_field: osm_id
        table: hotosm_bdi_waterways
        geom_field: foo_geom
    formatters:
      - name: pygeoapi_plugins.formatter.JSON

On the 14,776 features of the test table, building and serializing all of them took about 0.3 s with this pairing, compared with about 1 s for the core PostgreSQL Provider with pygeoapi's default JSON output.

The JSON formatter only handles item lists (/collections/{collectionId}/items?f=json). Single items and other endpoints keep pygeoapi's default JSON output.