You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
FQNs are utilized within TDFs and are the identifiers of attributes outside of the policy administration/database. There is a concept in the platform of attribute FQN reindexing. FQNs in the lookup table can get out of sync if:
an update was made to an attribute object (namespace name, definition name, value value) via RPC but there was a connection issue and the upsert to replace the FQNs containing the change failed
a manual database change was made altering any of the same policy objects
A functionality was added to the platform binary to reindex FQNs and there is an open issue to add it to otdfctl.
However, some potential issues have become clear over time.
It should be possible to run a re-index on an interval
Because reindexing is actually crucial for accessing TDFs containing the intended updated state of the FQN, consistency could be guaranteed at said interval.
The upsert queries have room for optimization and efficiency
The upsert db query for a namespace only updates the https://<namespace> FQN, the definition upsert query only updates the rows matching https://<namespace>/attr/<attribute definition name>, and so on.
It is not a problem because the CLI reindex command runs every query for every item, but optimization will be more important as at-scale deployments of the platform are realized and the volume of items in the database goes up.
partially fix#1603
- refactors FQN upserts to use a batch SQL query and removes
`upsertAttrFqn` function in favor of calling SQLC generated functions
directly
- simplifies `AttrFqnReindex` function used by CLI
- refactors Namespaces/Attributes/Values Create and UnsafeUpdate DB
layer methods to return result from Get method directly (allowing code
reuse and FQN retrieval after upsert)
---------
Co-authored-by: Jake Van Vorhis <[email protected]>
Background
FQNs are utilized within TDFs and are the identifiers of attributes outside of the policy administration/database. There is a concept in the platform of attribute FQN reindexing. FQNs in the lookup table can get out of sync if:
A functionality was added to the platform binary to reindex FQNs and there is an open issue to add it to otdfctl.
However, some potential issues have become clear over time.
Because reindexing is actually crucial for accessing TDFs containing the intended updated state of the FQN, consistency could be guaranteed at said interval.
upsert
queries have room for optimization and efficiencyThe
upsert
db query for a namespace only updates thehttps://<namespace>
FQN, the definitionupsert
query only updates the rows matchinghttps://<namespace>/attr/<attribute definition name>
, and so on.It is not a problem because the CLI reindex command runs every query for every item, but optimization will be more important as at-scale deployments of the platform are realized and the volume of items in the database goes up.
An improvement was recently unblocked by the SQLC migration see comments.
Acceptance Criteria
upsert
query overwrites the new namespace name in every FQN built from the namespace ID/name being upsertedupsert
query overwrites the new namespace name in every FQN built from the definition ID/name being upsertedupsert
query overwrites the new namespace name in every FQN built from the value ID/name being upsertedThe text was updated successfully, but these errors were encountered: