-
Notifications
You must be signed in to change notification settings - Fork 199
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(zip): cd header and zip64 info generation implemented (#2792)
- Loading branch information
1 parent
7f0bbb1
commit d1cf361
Showing
75 changed files
with
863 additions
and
336 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,21 @@ | ||
| # Apache Arrow | ||
|
|
||
|  | ||
|   | ||
|  | ||
|
|
||
| - *[`@loaders.gl/arrow`](/docs/modules/arrow)* - loaders.gl implementation | ||
| - *[Apache Arrow](https://arrow.apache.org/)* - A specification for large in-memory columnar data. | ||
| - *[ArrowJS](https://arrow.apache.org/docs/js)* - official documentation on ArrowJS API. | ||
| - *[ArrowJS](/docs/arrowjs)* - loaders.gl documentation on ArrowJS API. | ||
|
|
||
| The Apache Arrow project specifies a language-independent binary columnar memory format. It enables zero-copy shared memory and streaming messaging, interprocess communication, and is supported by many programming languages and data libraries. | ||
|
|
||
| This Apache Arrow specification supports encoding vectors and table-like containers of flat and nested data. | ||
|
|
||
| The Arrow spec is performance-optimized to eliminate memory copies and aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors. | ||
|
|
||
| Apache Arrow is emerging as a de-facto standard for large in-memory columnar data (Spark, Pandas, Drill, ...). | ||
|
|
||
| By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication. | ||
|
|
||
| For more information, see [ArrowJS](/docs/arrowjs) documentation. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| # GeoArrow | ||
|
|
||
|  | ||
|   | ||
|  | ||
|
|
||
| - *[`@loaders.gl/arrow`](/docs/modules/arrow)* - loaders.gl implementation | ||
| - *[GeoArrow Specification](https://github.com/geoarrow/geoarrow) | ||
| - *[Apache Arrow](https://arrow.apache.org/)* - A specification for large in-memory columnar data. | ||
| - *[ArrowJS](/docs/arrowjs)* - loaders.gl documentation on ArrowJS API. | ||
|
|
||
| ## Overview | ||
|
|
||
| GeoArrow is a specification for storing geospatial data in Apache Arrow memory layout. It ensures geospatial tools can interoperate and leverage the growing Apache Arrow ecosystem. | ||
|
|
||
| GeoArrow enables each row in an Arrow table to represent a feature as defined by the OGC Simple Feature Access standard (i.e. Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, and GeometryCollection). | ||
|
|
||
| Aside from geometry, simple features can also have additional standard Arrow columns that provide additional non-spatial attributes for the feature. | ||
|
|
||
| Geospatial tabular data where one or more columns contains feature geometries and remaining columns define feature attributes. The GeoArrow specification defines how such vector features (geometries) can be stored in Arrow (and Arrow-compatible) data structures. | ||
|
|
||
| Note that GeoArrow is not a separate format from Apache Arrow rather, the GeoArrow specification simply describes additional conventions for metadata and layout of geospatial data. This means that a valid GeoArrow file is always a valid Arrow file. This is done through [Arrow extension type](https://arrow.apache.org/docs/format/Columnar.html#extension-types) definitions that ensure type-level metadata (e.g., CRS) is propagated when used in Arrow implementations. | ||
|
|
||
|
|
||
| ## Relationship with GeoParquet | ||
|
|
||
| The [GeoParquet specification](https://github.com/opengeospatial/geoparquet) is closely related to GeoArrow. Notable differences: | ||
|
|
||
| - GeoParquet is a file-level metadata specification | ||
| - GeoArrow is a field-level metadata and memory layout specification | ||
|
|
||
| ## Geometry Types | ||
|
|
||
| | Geometry type | Read | Write | Description | | ||
| | -------------------------- | ---- | ----- | ----------- | | ||
| | `geoarrow.multipolygon` | ✅ | ❌ | | | ||
| | `geoarrow.polygon` | ✅ | ❌ | | | ||
| | `geoarrow.multipoint` | ✅ | ❌ | | | ||
| | `geoarrow.point` | ✅ | ❌ | | | ||
| | `geoarrow.multilinestring` | ✅ | ❌ | | | ||
| | `geoarrow.linestring` | ✅ | ❌ | | | ||
| | `geoarrow.wkb` | ❌ | ❌ | | | ||
| | `geoarrow.wkt` | ❌ | ❌ | | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,6 +1,6 @@ | ||
| { | ||
| "lerna": "2.9.1", | ||
| "version": "4.0.3", | ||
| "version": "4.0.4", | ||
| "command": { | ||
| "publish": {}, | ||
| "bootstrap": {} | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| // loaders.gl, MIT license | ||
| // Copyright (c) vis.gl contributors | ||
|
|
||
| import type {Loader, LoaderWithParser, LoaderOptions} from '@loaders.gl/loader-utils'; | ||
| import {ArrowWorkerLoader} from './arrow-loader'; | ||
| import type {GeoJSONTable, GeoJSONTableBatch, BinaryGeometry} from '@loaders.gl/schema'; | ||
| import type {ArrowTable, ArrowTableBatch} from './lib/arrow-table'; | ||
| import {parseGeoArrowSync} from './parsers/parse-geoarrow-sync'; | ||
| import {parseGeoArrowInBatches} from './parsers/parse-geoarrow-in-batches'; | ||
|
|
||
| // __VERSION__ is injected by babel-plugin-version-inline | ||
| // @ts-ignore TS2304: Cannot find name '__VERSION__'. | ||
| const VERSION = typeof __VERSION__ !== 'undefined' ? __VERSION__ : 'latest'; | ||
|
|
||
| export type GeoArrowLoaderOptions = LoaderOptions & { | ||
| arrow?: { | ||
| shape: 'arrow-table' | 'binary-geometry'; | ||
| }; | ||
| }; | ||
|
|
||
| /** ArrowJS table loader */ | ||
| export const GeoArrowWorkerLoader: Loader< | ||
| ArrowTable | BinaryGeometry, | ||
| never, | ||
| GeoArrowLoaderOptions | ||
| > = { | ||
| ...ArrowWorkerLoader, | ||
| options: { | ||
| arrow: { | ||
| shape: 'arrow-table' | ||
| } | ||
| } | ||
| }; | ||
|
|
||
| /** | ||
| * GeoArrowLoader loads an Apache Arrow table, parses GeoArrow type extension data | ||
| * to convert it to a GeoJSON table or a BinaryGeometry | ||
| */ | ||
| export const GeoArrowLoader: LoaderWithParser< | ||
| ArrowTable | GeoJSONTable, // | BinaryGeometry, | ||
| ArrowTableBatch | GeoJSONTableBatch, // | BinaryGeometry, | ||
| GeoArrowLoaderOptions | ||
| > = { | ||
| ...ArrowWorkerLoader, | ||
| options: { | ||
| arrow: { | ||
| shape: 'arrow-table' | ||
| } | ||
| }, | ||
| parse: async (arraybuffer: ArrayBuffer, options?: GeoArrowLoaderOptions) => | ||
| parseGeoArrowSync(arraybuffer, options?.arrow), | ||
| parseSync: (arraybuffer: ArrayBuffer, options?: GeoArrowLoaderOptions) => | ||
| parseGeoArrowSync(arraybuffer, options?.arrow), | ||
| parseInBatches: parseGeoArrowInBatches | ||
| }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| // import type {} from '@loaders.gl/loader-utils'; | ||
|
|
||
| import type {WriterWithEncoder, WriterOptions} from '@loaders.gl/loader-utils'; | ||
| import {GeoJSONTable, BinaryGeometry} from '@loaders.gl/schema'; | ||
| import {encodeGeoArrowSync} from './lib/encode-geoarrow'; | ||
|
|
||
| // __VERSION__ is injected by babel-plugin-version-inline | ||
| // @ts-ignore TS2304: Cannot find name '__VERSION__'. | ||
| const VERSION = typeof __VERSION__ !== 'undefined' ? __VERSION__ : 'latest'; | ||
|
|
||
| type ArrowWriterOptions = WriterOptions & { | ||
| arrow?: {}; | ||
| }; | ||
|
|
||
| /** Apache Arrow writer */ | ||
| export const GeoArrowWriter: WriterWithEncoder< | ||
| GeoJSONTable | BinaryGeometry, | ||
| never, | ||
| ArrowWriterOptions | ||
| > = { | ||
| name: 'Apache Arrow', | ||
| id: 'arrow', | ||
| module: 'arrow', | ||
| version: VERSION, | ||
| extensions: ['arrow', 'feather'], | ||
| mimeTypes: [ | ||
| 'application/vnd.apache.arrow.file', | ||
| 'application/vnd.apache.arrow.stream', | ||
| 'application/octet-stream' | ||
| ], | ||
| binary: true, | ||
| options: {}, | ||
| encode: async function encodeArrow(data, options?): Promise<ArrayBuffer> { | ||
| // @ts-expect-error | ||
| return encodeGeoArrowSync(data); | ||
| }, | ||
| encodeSync(data, options?) { | ||
| // @ts-expect-error | ||
| return encodeGeoArrowSync(data); | ||
| } | ||
| }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.