-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement MDArray API #433
Open
eschnett
wants to merge
19
commits into
yeesian:master
Choose a base branch
from
eschnett:eschnett/mdarray
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
203f923
Sketch for an MDArray implementation
eschnett fbc79a6
Make GADLGroup work
eschnett 85ba9d6
Add MDArray support
eschnett 4fbf316
Remove file
eschnett b85b5d5
Re-enable tests
eschnett 6b0721f
Simplify tests
eschnett 4c107fb
Add Dimension tests
eschnett a8e15a0
CI: Run benchmarks with Julia lts
eschnett 2292bdd
CI: Run benchmarks on macos-13
eschnett 5e39807
Add Attribute
eschnett 2b353f3
CI: Disable benchmarks
eschnett b57e23a
Test attributes
eschnett 55b25b1
Test high-level mdarray read/write functions
eschnett 42685aa
Restore all test cases
eschnett 2287f9f
Call error() when there are errors
eschnett 77d22bf
Change constructor API: Take parent as WeakRef
eschnett 36640fa
Reverse array indices
eschnett 4744e94
Use tuples instead of vectors for array dimensions
eschnett 9324fd1
Correct indices in MDArray.adviseread
eschnett File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,372 @@ | ||
abstract type AbstractExtendedDataType end | ||
# needs to have a `ptr::GDALExtendedDataTypeH` attribute | ||
|
||
abstract type AbstractEDTComponent end | ||
# needs to have a `ptr::GDALEDTComponentH` attribute | ||
|
||
abstract type AbstractGroup end | ||
# needs to have a `ptr::GDAL.GDALGroupH` attribute | ||
|
||
abstract type AbstractMDArray end | ||
# needs to have a `ptr::GDAL.GDALMDArrayH` attribute | ||
|
||
abstract type AbstractAttribute end | ||
# needs to have a `ptr::GDAL.GDALAttributeH` attribute | ||
|
||
abstract type AbstractDimension end | ||
# needs to have a `ptr::GDAL.GDALDimensionH` attribute | ||
|
||
################################################################################ | ||
|
||
# Question: Why do the `I...` types exist? The only difference seems | ||
# to be that they call a finalizer. This could instead be an option to | ||
# the constructor, simplifying the type hierarchy. | ||
|
||
mutable struct ExtendedDataType <: AbstractExtendedDataType | ||
ptr::GDAL.GDALExtendedDataTypeH | ||
end | ||
|
||
mutable struct EDTComponent <: AbstractEDTComponent | ||
ptr::GDAL.GDALEDTComponentH | ||
end | ||
|
||
mutable struct Group <: AbstractGroup | ||
ptr::GDAL.GDALGroupH | ||
|
||
Group(ptr::GDAL.GDALGroupH = C_NULL) = new(ptr) | ||
end | ||
|
||
mutable struct IGroup <: AbstractGroup | ||
ptr::GDAL.GDALGroupH | ||
|
||
function IGroup(ptr::GDAL.GDALGroupH = C_NULL) | ||
group = new(ptr) | ||
finalizer(destroy, group) | ||
return group | ||
end | ||
end | ||
|
||
mutable struct MDArray <: AbstractMDArray | ||
ptr::GDAL.GDALMDArrayH | ||
|
||
MDArray(ptr::GDAL.GDALMDArrayH = C_NULL) = new(ptr) | ||
end | ||
|
||
mutable struct IMDArray <: AbstractMDArray | ||
ptr::GDAL.GDALMDArrayH | ||
|
||
function IMDArray(ptr::GDAL.GDALMDArrayH = C_NULL) | ||
mdarray = new(ptr) | ||
finalizer(destroy, mdarray) | ||
return mdarray | ||
end | ||
end | ||
|
||
mutable struct Attribute <: AbstractAttribute | ||
ptr::GDAL.GDALAttributeH | ||
|
||
Attribute(ptr::GDAL.GDALAttributeH = C_NULL) = new(ptr) | ||
end | ||
|
||
mutable struct IAttribute <: AbstractAttribute | ||
ptr::GDAL.GDALAttributeH | ||
|
||
function IAttribute(ptr::GDAL.GDALAttributeH = C_NULL) | ||
attribute = new(ptr) | ||
finalizer(destroy, attribute) | ||
return attribute | ||
end | ||
end | ||
|
||
mutable struct Dimension <: AbstractDimension | ||
ptr::GDAL.GDALDimensionH | ||
|
||
Dimension(ptr::GDAL.GDALDimensionH = C_NULL) = new(ptr) | ||
end | ||
|
||
mutable struct IDimension <: AbstractDimension | ||
ptr::GDAL.GDALDimensionH | ||
|
||
function IDimension(ptr::GDAL.GDALDimensionH = C_NULL) | ||
dimension = new(ptr) | ||
finalizer(destroy, dimension) | ||
return dimension | ||
end | ||
end | ||
|
||
################################################################################ | ||
|
||
Base.unsafe_convert(::Type{Ptr{Cvoid}}, x::AbstractExtendedDataType) = x.ptr | ||
Base.unsafe_convert(::Type{Ptr{Cvoid}}, x::AbstractEDTComponent) = x.ptr | ||
Base.unsafe_convert(::Type{Ptr{Cvoid}}, x::AbstractGroup) = x.ptr | ||
Base.unsafe_convert(::Type{Ptr{Cvoid}}, x::AbstractMDArray) = x.ptr | ||
Base.unsafe_convert(::Type{Ptr{Cvoid}}, x::AbstractAttribute) = x.ptr | ||
Base.unsafe_convert(::Type{Ptr{Cvoid}}, x::AbstractDimension) = x.ptr | ||
|
||
################################################################################ | ||
|
||
function destroy(datatype::AbstractExtendedDataType)::Nothing | ||
GDAL.gdalextendeddatatyperelease(datatype) | ||
datatype.ptr = C_NULL | ||
return nothing | ||
end | ||
|
||
function destroy(edtcomponent::AbstractEDTComponent)::Nothing | ||
GDAL.gdaledtcomponentrelease(edtcomponent) | ||
edtcomponent.ptr = C_NULL | ||
return nothing | ||
end | ||
|
||
function destroy(group::AbstractGroup)::Nothing | ||
GDAL.gdalgrouprelease(group) | ||
group.ptr = C_NULL | ||
return nothing | ||
end | ||
|
||
function destroy(mdarray::AbstractMDArray)::Nothing | ||
GDAL.gdalmdarrayrelease(mdarray) | ||
mdarray.ptr = C_NULL | ||
return nothing | ||
end | ||
|
||
function destroy(attribute::AbstractAttribute)::Nothing | ||
GDAL.gdalattributerelease(attribute) | ||
attribute.ptr = C_NULL | ||
return nothing | ||
end | ||
|
||
function destroy(dimension::AbstractDimension)::Nothing | ||
GDAL.gdaldimensionrelease(dimension) | ||
dimension.ptr = C_NULL | ||
return nothing | ||
end | ||
|
||
################################################################################ | ||
|
||
# GDALGroup | ||
|
||
function getname(group::AbstractGroup)::AbstractString | ||
return GDAL.gdalgroupgetname(group) | ||
end | ||
|
||
function getfullname(group::AbstractGroup)::AbstractString | ||
return GDAL.gdalgroupgetfullname(group) | ||
end | ||
|
||
function getmdarraynames( | ||
group::AbstractGroup, | ||
options = nothing, | ||
)::AbstractVector{<:AbstractString} | ||
# TODO: allow options | ||
@assert options === nothing | ||
return GDAL.gdalgroupgetmdarraynames(group, C_NULL) | ||
end | ||
|
||
function openmdarray( | ||
group::AbstractGroup, | ||
name::AbstractString, | ||
options, | ||
)::AbstractMDArray | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IMDArray(GDAL.gdalgroupopenmdarray(group, name, C_NULL)) | ||
end | ||
|
||
function getgroupnames( | ||
group::AbstractGroup, | ||
options = nothing, | ||
)::AbstractVector{<:AbstractString} | ||
# TODO: allow options | ||
@assert options === nothing | ||
return GDAL.gdalgroupgetgroupnames(group, C_NULL) | ||
end | ||
|
||
function opengroup( | ||
group::AbstractGroup, | ||
name::AbstractString, | ||
options, | ||
)::AbstractGroup | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IGroup(GDAL.gdalgroupopengroup(group, name, C_NULL)) | ||
end | ||
|
||
function getvectorlayernames( | ||
group::AbstractGroup, | ||
options, | ||
)::AbstractVector{<:AbstractString} | ||
# TODO: allow options | ||
@assert options === nothing | ||
return GDAL.gdalgroupgetvectorlayernames(group, C_NULL) | ||
end | ||
|
||
function openvectorlayer(group::AbstractGroup, options)::AbstractFeatureLayer | ||
# TODO: allow options | ||
@assert options === nothing | ||
# TODO: Find out how to set `ownedby` and `spatialref`, probably by querying `group` | ||
return IFeatureLayer( | ||
GDAL.openvectorlayer(group, C_NULL), | ||
ownedby, | ||
spatialref, | ||
) | ||
end | ||
|
||
function getdimensions( | ||
group::AbstractGroup, | ||
options, | ||
)::AbstractVector{<:AbstractDimension} | ||
# TODO: allow options | ||
@assert options === nothing | ||
dimensioncountref = Ref{Csize_t}() | ||
dimensionshptr = GDAL.gdalgroupgetdimensions(group, ndimensions, C_NULL) | ||
dimensions = AbstractDimension[ | ||
IDimension(unsafe_load(dimensionhptr, n)) for | ||
n in 1:dimensionscountref[] | ||
] | ||
GDAL.vsifree(dimensionshptr) | ||
return dimensions | ||
end | ||
|
||
function creategroup( | ||
group::AbstractGroup, | ||
name::AbstractString, | ||
options, | ||
)::AbstractGroup | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IGroup(GDAL.gdalgroupcreategroup(group, name, C_NULL)) | ||
end | ||
|
||
function deletegroup(group::AbstractGroup, name::AbstractString, options)::Bool | ||
# TODO: allow options | ||
@assert options === nothing | ||
return GDAL.gdalgroupdeletegroup(group, name, C_NULL) | ||
end | ||
|
||
function createdimension( | ||
group::AbstractGroup, | ||
name::AbstractString, | ||
type::AbstractString, | ||
direction::AbstractString, | ||
size::Integer, | ||
options, | ||
)::AbstractDimension | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IDimension( | ||
GDAL.gdalgroupcreatedimension( | ||
group, | ||
name, | ||
type, | ||
direction, | ||
size, | ||
C_NULL, | ||
), | ||
) | ||
end | ||
|
||
function createmdarray( | ||
group::AbstractGroup, | ||
name::AbstractString, | ||
dimensions::AbstractVector{<:AbstractDimension}, | ||
datatype::AbstractExtendedDataType, | ||
options, | ||
)::AbstractMDArray | ||
# TODO: allow options | ||
@assert options === nothing | ||
dimensionhptrs = Ptr{Cvoid}[convert(Ptr{Cvoid}, dim) for dim in dimensions] | ||
return IMDArray( | ||
GDAL.gdalgroupcreatemdarray( | ||
group, | ||
name, | ||
length(dimensionhptrs), | ||
dimensionhptrs, | ||
datatype, | ||
C_NULL, | ||
), | ||
) | ||
end | ||
|
||
function deletemdarray( | ||
group::AbstractGroup, | ||
name::AbstractString, | ||
options, | ||
)::Bool | ||
# TODO: allow options | ||
@assert options === nothing | ||
return GDAL.gdalgroupdeletemdarray(group, name, C_NULL) | ||
end | ||
|
||
# gettotalcopycost | ||
# copyfrom | ||
|
||
function getstructuralinfo( | ||
group::AbstractGroup, | ||
)::AbstractVector{<:AbstractString} | ||
return GDAL.gdalgroupgetstructuralinfo(group) | ||
end | ||
|
||
function openmdarrayfromfullname( | ||
group::AbstractGroup, | ||
fullname::AbstractString, | ||
options, | ||
)::AbstractMDArray | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IMDArray( | ||
GDAL.gdalgroupopenmdarrayfromfullname(group, fullname, C_NULL), | ||
) | ||
end | ||
|
||
function resolvemdarray( | ||
group::AbstractGroup, | ||
name::AbstractString, | ||
startingpath::AbstractString, | ||
options, | ||
)::AbstractMDArray | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IMDArray( | ||
GDAL.gdalgroupresolvemdarray(group, name, startingpath, C_NULL), | ||
) | ||
end | ||
|
||
function opengroupfromfullname( | ||
group::AbstractGroup, | ||
fullname::AbstractString, | ||
options, | ||
)::AbstractGroup | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IGroup(GDAL.gdalgroupopengroupfromfullname(group, fullname, C_NULL)) | ||
end | ||
|
||
function opendimensionfromfullname( | ||
group::AbstractGroup, | ||
fullname::AbstractString, | ||
options, | ||
)::AbstractDimension | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IDimension( | ||
GDAL.gdalgroupopendimensionfromfullname(group, fullname, C_NULL), | ||
) | ||
end | ||
|
||
# clearstatistics | ||
|
||
function rename(group::AbstractGroup, newname::AbstractString)::Bool | ||
return GDAL.gdalgrouprename(group, newname) | ||
end | ||
|
||
function subsetdimensionfromselection( | ||
group::AbstractGroup, | ||
selection::AbstractString, | ||
options, | ||
)::AbstractGroup | ||
# TODO: allow options | ||
@assert options === nothing | ||
return IGroup( | ||
GDAL.gdalgroupsubsetdimensionfromselection(group, selection, C_NULL), | ||
) | ||
end |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably want this to be <: AbstractDiskArray so chunked reads work automatically.