compatibility with pystac-client

[scottyhq]

I expect released pystac-client and pystac versions to work well together, and am currently encountering a KeyError: 'id' when opening saved search results from pystac-client with pystac.

Related issue #256, #302 and stac-utils/pystac-client#1

from pystac_client import Client # 0.1.1 
import pystac # 0.5.6

aoi = {'type': 'Polygon',
 'coordinates': [[[-108.29, 38.88],
   [-107.97, 38.88],
   [-107.97, 39.10],
   [-108.29, 39.10],
   [-108.29, 38.88]]]}

catalog = Client.open("https://cmr.earthdata.nasa.gov/stac/LPCLOUD")
results = catalog.search(collections=['HLSS30.v1.5'], #S2
                         intersects=aoi,
                         datetime='2021-01-01/2021-01-03',
                         )
print(f"{results.matched()} items found")

items = results.items_as_collection()
items.save('items.json')

pystac.read_file('items.json')

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-95-12b3d23b2c32> in <module>
     20 items.save('items.json')
     21 
---> 22 pystac.read_file('items.json')

/srv/conda/envs/notebook/lib/python3.8/site-packages/pystac/__init__.py in read_file(href)
     71         by the JSON read from the file located at HREF.
     72     """
---> 73     return STACObject.from_file(href)
     74 
     75 

/srv/conda/envs/notebook/lib/python3.8/site-packages/pystac/stac_object.py in from_file(cls, href)
    492 
    493         if cls == STACObject:
--> 494             o = STAC_IO.stac_object_from_dict(d, href=href)
    495         else:
    496             o = cls.from_dict(d, href=href)

/srv/conda/envs/notebook/lib/python3.8/site-packages/pystac/serialization/__init__.py in stac_object_from_dict(d, href, root)
     35 
     36     if info.object_type == STACObjectType.CATALOG:
---> 37         return Catalog.from_dict(d, href=href, root=root)
     38 
     39     if info.object_type == STACObjectType.COLLECTION:

/srv/conda/envs/notebook/lib/python3.8/site-packages/pystac/catalog.py in from_dict(cls, d, href, root)
    788         d = deepcopy(d)
    789 
--> 790         id = d.pop('id')
    791         description = d.pop('description')
    792         title = d.pop('title', None)

KeyError: 'id'

[scottyhq]

I also tried installing unrelased versions, but can't import pystac_client:

pip install git+https://github.com/stac-utils/pystac-client.git
pip install pystac==v1.0.0-beta.2

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-442652a99a3a> in <module>
----> 1 import pystac_client
      2 #print(pystac_client.__version__)

/srv/conda/envs/notebook/lib/python3.8/site-packages/pystac_client/__init__.py in <module>
      1 # flake8: noqa
----> 2 from pystac import STAC_IO
      3 import pystac.extensions.base
      4 
      5 from pystac_client.version import __version__

ImportError: cannot import name 'STAC_IO' from 'pystac' (/srv/conda/envs/notebook/lib/python3.8/site-packages/pystac/__init__.py)

[duckontheweb]

Thanks for reporting this @scottyhq.

@matthewhanson Based on this, I'm thinking it would be good to get your work on the stripped down ItemCollection into the 1.0 release. Do you want to put in a PR from https://github.com/stac-utils/pystac/blob/mah/itemcollection, or do you feel like there's more work that needs to be done on that?

[matthewhanson]

@duckontheweb Agree, I'll get a PR up this coming week (holiday Monday here in the states, but will push on that Tuesday)

[matthewhanson]

The issue with this scenario is that pystac.read_file is using STACObject.from_file so assumes that ItemCollection is itself a STAC Object, but here we concluded it should not be:
#302

We can discuss in detail at the STAC Ecosystem call, but even regardless of what we decide we may want to hold off adding it here and keeping it in pystac-client until that's been fully tested and working. Then we can port it over in a future version.

[duckontheweb]

Thanks for the reminder on that issue @matthewhanson, I had forgotten some of the details of that discussion and where we landed.

Here are my thoughts on this...

I expect released pystac-client and pystac versions to work well together

I agree in general. However, other libraries that use PySTAC (e.g. pystac-client, stactools, etc.) add new functionality that PySTAC cannot be expected to support. It seems like the scope of PySTAC is to support structures (and associated functionality) defined in the core STAC spec and "official" STAC extensions (those with a home in the stac-extensions org). ItemCollections are unfortunately a big gray area because they were originally in the core spec (removed in 1.0.0-beta.1), then as a fragment in the API spec, and now also in a WIP Item Collection extension.

It seems like the path to resolving this is:

1. Decide on a home for the ItemCollection spec (maybe this has already been decided and I've just missed it, so please pipe in if this is the case)
2. Remove duplicate definitions in the spec (e.g. remove the extension if we decide it should live in the API spec)
3. If it is an extension, then we should support it in PySTAC. If it's in the API spec, then it's beyond the scope of PySTAC to handle.

[duckontheweb]

We had a productive conversation about how to handle ItemCollections on a STAC call this morning and were able to come up with a path forward.

We will add support for ItemCollections in PySTAC as simple GeoJSON FeatureCollection containers for Items.

Here's my proposal for the initial PySTAC implementation:

• Serialization/deserialization

  Will be able to deserialize an ItemCollection instance from and serialize an instance to JSON. Any top-level fields in the ItemCollection should be preserved throughout the deserialization/serialization round-trip, but no special meaning will be assigned to anything except type and features.

• Iteration

  Will be able to loop over all Items in an ItemCollection using syntax like:

  item_collection: ItemCollection = ...
  for item in item_collection:
      ...

  If a feature is encountered that is not a STAC Item , the iteration will raise a STACTypeError.

We should also add methods for converting to/from Catalogs and Collections, but I don't think we should try to include that in the 1.0 PySTAC release. It's going to require some more careful consideration of how we handle things like stac_extensions and links fields, and may require some logic for merging an ItemCollection into an existing Catalog/Collection. If someone has a use-case for any of these conversions, let's write it up as a separate issue so we can discuss the feature requirements in more detail.

@scottyhq @matthewhanson Does this sound like a good start and will it meet your immediate needs?

[matthewhanson]

This all makes sense to me, here's the latest item_collection.py in pystac-client we can pull over. Needs to be updated to include typing where missing, docstrings, and tests.