Skip to content

Commit

Permalink
Extended documentation on the HTTP frontend to cover the new service
Browse files Browse the repository at this point in the history
  • Loading branch information
iagaponenko committed Nov 15, 2024
1 parent 1a85967 commit 0d6e30c
Show file tree
Hide file tree
Showing 10 changed files with 356 additions and 139 deletions.
14 changes: 7 additions & 7 deletions doc/admin/data-table-indexes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -247,7 +247,7 @@ Where:
The required *base* name of the table where the index will be created.

``overlap`` : *number* := ``0``
The optional *overlap* flagg indicating a sub-type of the *chunk* table. The value should be one of the following:
The optional *overlap* flag indicating a sub-type of the *chunk* table. The value should be one of the following:

- ``1`` : *full overlap*
- ``0`` : *chunk*
Expand Down Expand Up @@ -301,9 +301,9 @@ Where:
types (numeric, etc.). Otherwise, an index creation request will fail.

``ascending`` : *number*
The required sorting order of the column in the index. It translates into ``ASC`` or ``DESC`` optiona
in the key definition in ``key_part``. A value of ``0`` will be interpreted as ``DESC``. Any other positive number
will be imterpreted as to ``ASC``.
The required sorting order of the column in the index. It translates into ``ASC`` or ``DESC`` options
in the key definition in ``key_part``. A value of ``0`` will be interpreted as ``DESC``.
Any other positive number will be imterpreted as to ``ASC``.

``auth_key`` : *string*
The required zauthorization key.
Expand Down Expand Up @@ -388,7 +388,7 @@ Where:
The required *base* name of the table where the index will be created.

``overlap`` : *number* := ``0``
The optional *overlap* flagg indicating a sub-type of the *chunk* table. The value should be one of the following:
The optional *overlap* flag indicating a sub-type of the *chunk* table. The value should be one of the following:

- ``1`` : *full overlap*
- ``0`` : *chunk*
Expand Down Expand Up @@ -439,10 +439,10 @@ Where the service path has the following parameters:
``table`` : *string*
The name of the table for which the indexes are required to be collected.

The optional query parameyter is
The optional query parameter is

``overlap`` : *number* := ``0``
The optional *overlap* flagg indicating a sub-type of the *chunk* table. The value should be one of the following:
The optional *overlap* flag indicating a sub-type of the *chunk* table. The value should be one of the following:

- ``1`` : *full overlap*
- ``0`` : *chunk*
Expand Down
2 changes: 1 addition & 1 deletion doc/ingest/api/advanced/directors.rst
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ The following table illustrates how JSON configurations for all the above-mentio
| "longitude_key" : "dec" | |
| } | **Note**: Attributes ``latitude_key`` and |
| | ``longitude_key`` were not provided. |
| | is allowed for the dependent tables. |
| | This is allowed for the dependent tables. |
| | |
| | .. code-block:: json |
| | |
Expand Down
1 change: 0 additions & 1 deletion doc/ingest/api/advanced/transactions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,6 @@ Things to consider:
factor limiting the performance of the workflow.
- Any failure to ingest a contribution will result in aborting the entire transaction. This can significantly
impact the workflow's performance, especially if the amount of data to be ingested is large.
impact the workflow's performance, especially if the amount of data to be ingested is large.

Best use:

Expand Down
8 changes: 4 additions & 4 deletions doc/ingest/api/concepts/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@ it should be registered using:

- :ref:`ingest-db-table-management-register-db` (REST)

The newely registered database will be always in the *unpublished* state. If the database already exists in
the Replication/Ingest and it's in the *published* state it should be *unpublished* state using:
The newly registered database will be always in the *unpublished* state. If the database already exists in
the Replication/Ingest and it's in the *published* state it should be put into the *unpublished* state using:

- :ref:`ingest-db-table-management-unpublish-db` (REST)

Expand Down Expand Up @@ -159,8 +159,8 @@ The REST API for initiating the contribuiton requests is covered in the followin
Monitor the progress of the ingest activities
----------------------------------------------

The workflow should always be avare about the progress of the ingest activities, and about the status of the
contribution requests. This is need for (at least) three reasons:
The workflow should always be aware about the progress of the ingest activities, and about the status of the
contribution requests. This is needed for (at least) three reasons:

#. To know when the ingest activities are finished
#. To know when the ingest activities (and which requests) are failed
Expand Down
19 changes: 9 additions & 10 deletions doc/ingest/api/concepts/transactions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,13 @@
Transactions
============

The distributed transaction mechanism is one of the key technologies that was
implemented in the Qserv Ingest system to allow for the incremental updates of the overall state of the data and metadata
while ensuring the consistency of the ingested catalogs. Transactions also play an important role in allowing
the high-performance ingest activities to be performed in a distributed environment. Transactions if used correct may
significantly increase the level of parallelism of the ingest workflows. Transactions are not visible to end users.
The distributed transaction mechanism is a crucial technology in the Qserv Ingest system. It enables incremental updates
to the data and metadata while maintaining the consistency of ingested catalogs. Transactions also facilitate high-performance
ingest activities in a distributed environment. When used correctly, transactions can significantly enhance the parallelism
of ingest workflows. Note that transactions are not visible to end users.

Transactions are open in a scope of a database. It's a responsibility of the workflows to manage transactions as needed
for the ingest activities uisng the following REST services:
for the ingest activities using the following REST services:

- :ref:`ingest-trans-management` (REST)

Expand Down Expand Up @@ -68,10 +67,10 @@ to Qserv users and is queriable. Here is an illustration of a query and the corr
Checkpointing
-------------

Transactions also provide the checkpointing mechanism that allows rolling back to a prior consistent state of the affected tables
should any problem occur during the ingest activities. Transactions may spans across many workers and tables located
at the workers. It's up to the workflow to decide what contrubutions to ingest and in what order to ingest those in
a scope of each transaction.
Transactions also provide a checkpointing mechanism that allows rolling back to a prior consistent state of the affected tables
should any problem occur during the ingest activities. Transactions may span across many workers and tables located
at the workers. It's up to the workflow to decide what contributions to ingest and in what order to ingest those in
the scope of each transaction.

The following diagram illustrates the concept of the transactions in Qserv. There are 3 transactions that are started and
commited sequentially (in the real life scenarios the transactions can be and should be started and commited in parallel,
Expand Down
2 changes: 1 addition & 1 deletion doc/ingest/api/index.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

.. note::

Information in this guide corresponds to the version **38** of the Qserv REST API. Keep in mind
Information in this guide corresponds to the version **39** of the Qserv REST API. Keep in mind
that each implementation of the API has a specific version. The version number will change
if any changes to the implementation or the API that might affect users will be made.
The current document will be kept updated to reflect the latest version of the API.
Expand Down
10 changes: 5 additions & 5 deletions doc/user/http-frontend-general.rst
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ Example:
"name" : "http",
"id" : 8,
"instance_id" : "qserv-prod",
"version" : 38,
"version" : 39,
"success" : 1
}
Expand All @@ -137,27 +137,27 @@ the following request for checking the status of an ongoing query might look lik

.. code-block:: bash
curl -k 'https://localhost:4041/query-async/status/1234?version=38' -X GET
curl -k 'https://localhost:4041/query-async/status/1234?version=39' -X GET
For other HTTP methods used by the API, the version number must be included in the body of the request as shown below:

.. code-block:: bash
curl -k 'https://localhost:4041/query-async' -X POST \
-H 'Content-Type: application/json' \
-d'{"version":38,"query":"SELECT ..."}'
-d'{"version":39,"query":"SELECT ..."}'
If the number does not match expectations, such a request will fail and the service will return the following
response. Here is an example of what will happen if the wrong version number ``29`` is specified instead
of ``38`` (as specified in the example above):
of ``39`` (as specified in the example above):

.. code-block:: json
{ "success" : 0,
"error" : "The requested version 29 of the API is not in the range
supported by the service.",
"error_ext" : {
"max_version" : 38,
"max_version" : 39,
"min_version" : 30
},
"warning" : ""
Expand Down
Loading

0 comments on commit 0d6e30c

Please sign in to comment.