Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GTC-3101 Add documentation for a bunch of operations and creation options #617

Merged
merged 3 commits into from
Jan 6, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 43 additions & 7 deletions app/models/pydantic/creation_options.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,24 +122,60 @@ class RasterTileSetAssetCreationOptions(StrictBaseModel):
"when input files are in different projections from each other."
)
)
pixel_meaning: str
pixel_meaning: str = Field(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should you specify it has to lowercase, spaces instead of underscores, not too long, etc.? Right now you'll just get some regex error response, might be useful to state it here.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

..., description="Short description of what the pixel value in the "
"raster represents. This is used to clarify the meaning of the raster "
"and distinguish multiple raster tile sets based on the same dataset "
"version."
)
data_type: DataType
nbits: Optional[int]
calc: Optional[str]
nbits: Optional[int] = Field(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to link to the documentation from GDAL in these?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

None,
description="Advanced option that lets GDAL compress the data even "
"more based on the number of bits you need."
)
calc: Optional[str] = Field(
None,
description="For raster sources or default assets, a raster algebra "
"expression, similar to gdal_calc, to "
"combine multiple raster layers into a new layer. Each source is "
"treated as a separate band, and is assigned a letter variable A-Z "
"based on the order in the sources. This expression can technically be "
"any valid NumPy expression.\n\nFor vector default assets, an SQL "
"expression specifying a calculation that yields the desired raster "
"value based on the fields of your vector dataset."
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about this reorganization of the text? I tried to make the two different modes more explicit and clarify the part about band variable assignment:
"There are two modes for this field, one for rasterizing vector sources and one for transforming one or more sources that are already raster. For rasterizing vector sources, this field should be an SQL expression that yields the desired raster value (presumably) based on the fields of your vector dataset. For raster sources, this should be a raster algebra expression, similar to that provided to gdal_calc, that transforms one or more input bands into one or more output bands. For use in this expression, each band in the sources is assigned an alphabetic variable (A-Z, then AA-AZ, etc.) in the order it exists in those sources, with those of the first source first, continuing with those of the second, and so on. So with two input sources of two bands each, they would be assigned to variables A and B (for the first source) and C and D (for the second source). NumPy is in scope, as np.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, took your suggested reorganization, with just slight modifications. Thanks!

band_count: int = 1
union_bands: bool = False
no_data: Optional[Union[List[NoDataType], NoDataType]]
rasterize_method: Optional[RasterizeMethod]
rasterize_method: Optional[RasterizeMethod] = Field(
RasterizeMethod.value,
description="For raster sources or default assets, 'value' (the "
"default) means use the value from the last or only band processed, "
"and 'count' means count the number of bands with data values."
)
resampling: ResamplingMethod = PIXETL_DEFAULT_RESAMPLING
order: Optional[Order]
order: Optional[Order] = Field(
None,
description="For vector default assets, order the features by the "
"calculated raster value. For 'asc', the features are ordered by "
"ascending calculated value so that the largest calculated value is "
"used in the raster when there are overlapping features. For 'desc', "
"the ordering is descending, so that the smallest calculated value "
"is used when there are overlaps."
)
overwrite: bool = False
subset: Optional[str]
grid: Grid
symbology: Optional[Symbology] = None
compute_stats: bool = True
compute_histogram: bool = False
process_locally: bool = True
auxiliary_assets: Optional[List[UUID]] = None
auxiliary_assets: Optional[List[UUID]] = Field(
None,
description="Asset IDs of additional rasters you might want to include "
"in your calc expression."
)
photometric: Optional[PhotometricType] = None
num_processes: Optional[StrictInt] = None
timeout_sec: Optional[StrictInt] = Field(
Expand Down Expand Up @@ -331,7 +367,7 @@ class RasterTileCacheCreationOptions(TileCacheBaseModel):
"default",
description="Name space to use for raster tile cache. "
"This will be part of the URI and will "
"allow to create multiple raster tile caches per version,",
"allow creation of multiple raster tile caches per version,",
)
symbology: Symbology = Field(..., description="Symbology to use for output tiles")
source_asset_id: str = Field(
Expand Down
8 changes: 6 additions & 2 deletions app/models/pydantic/versions.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,12 @@ class Version(BaseRecord):
metadata: Union[VersionMetadataOut, BaseModel]
status: VersionStatus = VersionStatus.pending

# Each element of assets is a tuple (asset_type, assert_uri, asset_id)
assets: List[Tuple[str, str, str]] = list()
assets: List[Tuple[str, str, str]] = Field(
list(),
description="List of saved (non-pending and non-failed) assets, with "
" elements in the form: [asset_type, asset_uri, asset_id]. The list "
"of assets is sorted by the creation time of each asset."
)


class VersionCreateIn(StrictBaseModel):
Expand Down
11 changes: 7 additions & 4 deletions app/routes/assets/asset.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,13 @@
"""Assets are replicas of the original source files.
"""Assets are usually alternate representations of the base dataset
version, sometimes combining in extra data from other datasets.

Assets might be served in different formats, attribute values might be
altered, additional attributes added, and feature resolution might have
changed. Assets are either managed or unmanaged. Managed assets are
created by the API and users can rely on data integrity. Unmanaged
assets are only loosely linked to a dataset version and users must
cannot rely on full integrity. We can only assume that unmanaged are
based on the same version and do not know the processing history.
cannot rely on full integrity. We can only assume that unmanaged assets
are based on the same version and do not know the processing history.
"""

from typing import List, Optional, Union
Expand Down Expand Up @@ -87,7 +88,9 @@ async def get_asset(
*,
asset_id: UUID = Path(...),
) -> AssetResponse:
"""Get a specific asset."""
"""Get a specific asset. This provides information on the asset, including
the asset id, the asset status, the asset URI, and creation & last update
times."""
try:
row: ORMAsset = await assets.get_asset(asset_id)
except RecordNotFoundError as e:
Expand Down
4 changes: 2 additions & 2 deletions app/routes/datasets/asset.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@ async def get_version_assets(
description="The number of assets per page. Default is `10`.",
),
) -> Union[PaginatedAssetsResponse, AssetsResponse]:
"""Get all assets for a given dataset version. The list of assets
is sorted by the creation time of each asset.
"""Get all assets for a given dataset version (including pending/failed assets).
The list of assets is sorted by the creation time of each asset.

Will attempt to paginate if `page[size]` or `page[number]` is
provided. Otherwise, it will attempt to return the entire list of
Expand Down
6 changes: 5 additions & 1 deletion app/routes/datasets/dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,11 @@ async def update_dataset(
request: DatasetUpdateIn,
user: User = Depends(get_owner),
) -> DatasetResponse:
"""Update metadata, accessibility or ownership of a dataset."""
"""Update metadata, accessibility or ownership of a dataset.

Individual fields of the metadata can be modified, without affecting other
existing fields.
"""
input_data: Dict = request.dict(exclude_none=True, by_alias=True)

if request.owner_id is not None:
Expand Down
15 changes: 12 additions & 3 deletions app/routes/datasets/versions.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,8 @@
async def get_version(
*, dv: Tuple[str, str] = Depends(dataset_version_dependency)
) -> VersionResponse:
"""Get basic metadata for a given version. The list of assets is sorted by
"""Get basic metadata for a given version. The list of assets only includes
saved (non-pending and non-failed) assets and is sorted by
the creation time of each asset."""

dataset, version = dv
Expand All @@ -106,8 +107,8 @@ async def add_new_version(
user: User = Depends(get_owner),
response: Response,
):
"""Create a version for a given dataset by uploading the geospatial/tabular
asset.
"""Create a version for a given dataset by uploading the tabular, vector,
or raster asset.

Only the dataset's owner or a user with `ADMIN` user role can do
this operation.
Expand Down Expand Up @@ -373,6 +374,14 @@ async def get_stats(dv: Tuple[str, str] = Depends(dataset_version_dependency)):
response_model=Union[FieldsMetadataResponse, RasterBandsMetadataResponse],
)
async def get_fields(dv: Tuple[str, str] = Depends(dataset_version_dependency)):
"""Get the fields of a version. For a version with a vector default asset,
these are the fields (attributes) of the features of the base vector dataset.

For a version with a raster default asset, the fields are all the raster
tile sets that use the same grid as the raster default asset. Also
included are some fields with special meaning such as 'area__ha',
'latitude', and 'longitude'.
"""
dataset, version = dv
orm_asset: ORMAsset = await assets.get_default_asset(dataset, version)

Expand Down
Loading