Come up with way to store user specific preferences for use in the Console

[kschiffer]

Summary

As part of the ongoing UX improvement plans for the Console, it will be beneficial to store certain user preferences in the user record so they can be accessed universally.

Current Situation

Right now, it is only possible to store attributes to entities (including users). Using this interface is only good for storing strings. This means that while we could store some data as simple strings and/or JSON, this approach has a couple of drawbacks:

• This is using attributes (supposedly) against their original purpose
• While simple strings can be indexed and searched, it's not possible with JSON formatted data
• Not easy to mark such special attributes so that they don't interfere with the other uses of attributes

I believe this why we should think of a new way to store such data.

Why do we need this? Who uses it, and when?

We need this for purposes such as:

• Storing bookmarks/favorite entities so they can be accessed more easily when navigating the Console
• Storing user preferences such as: dark mode, customized dashboard layouts, different list view layouts, default sort orders, default event filters, etc
• Storing payload formatters for different purposes to apply them easily when onboarding different end devices

There are various other ways we could make use of such user data.

Proposed Implementation

First of all, I want to point out that while using browser-based storage such as cookies/local storage can be used to address this issue as well, the biggest drawback is that such preferences would be attached to the browser and as such lost whenever a user logs in from a different browser (eg when using mobile). This is less dramatic for some settings such as preferred sort orders, but would become more frustrating when storing bookmarks such as favorite end devices or gateways.

As such, I see two remaining options:

Backend-for-frontend

This approach would mean that we set up a small service (eg. express.js server) that would connect to a database to store such user-defined data and generally act as a middleware between Console and backend.

Pros:

• Quite flexible
• Could be used to do other things such as acting as middleware for connections to the backend and providing a simple interface to facilitate the tasks of the Console

Cons:

• Lots of added complexity (as in failure potential) and extra maintenance
• There would need to be a mechanism that keeps the two databases reasonably in sync, eg. removing bookmarks for deleted entities
• Performance overhead, as the added complexity and fetching steps can slow down the overall experience

Extend user registry

This is I guess the most intuitive approach, where we would add new fields to the user schema that are then handled exclusively by the Console to store user preferences.
Pros:

• Relatively simple to implement and maintain
• Possible to query for certain data

Cons:

• Storing and maintaining data that is exclusively used by the Console
• There might be limits on how much the schema can be modified and expanded before it starts to impact performance

I myself would lean to the latter approach but I might not understand all possible concerns, so I'd like to gather some opinions here so we can decide which approach to take.

cc @adriansmares @johanstokking @KrishnaIyer

Contributing

• I can help by doing more research.
• I can help by implementing the feature after the proposal above is approved.
• I can help by testing the feature before it's released.

Code of Conduct

• I agree to follow TTN's Community Code of Conduct.

[johanstokking]

Extend user registry

This is I guess the most intuitive approach, where we would add new fields to the user schema that are then handled exclusively by the Console to store user preferences. Pros:

* Relatively simple to implement and maintain

* Possible to query for certain data

Cons:

* Storing and maintaining data that is exclusively used by the Console

* There might be limits on how much the schema can be modified and expanded before it starts to impact performance

I think this is the way to go. I don't see an issue with the first con here; the Console is a first class component in TTS and it just needs to persist state, just like other components persist (private) state.

• Storing bookmarks/favorite entities so they can be accessed more easily when navigating the Console

• Storing user preferences such as: dark mode, customized dashboard layouts, different list view layouts, default sort orders, default event filters, etc

These are simply columns per user, right? For booksmarks/favorites or recently viewed there would be an array column basically.

• Storing payload formatters for different purposes to apply them easily when onboarding different end devices

I'm not too sure about this though. I'm not against this, but we should be very mindful about adding something like this. Is this a customer request?

[adriansmares]

In general I am on the same boat regarding the storage - it should stay in the Identity Server, and it probably should be under two forms:

1. A per user table with the extra Console settings. I suggest that we make this one strongly typed (so not a proto Struct buy really have a fixed form, in order to support individual field level patches). Here we would have per user preferences like custom layouts, sort orders, and other beauties. The underlying table would have a jsonb column, but API wise we should allow per-field patches (similar to how billing and configuration work today with tenants).
   It would be fine to also add this patchable Struct directly to the User model - I am just a bit weary against having huge columns on a single table.
2. A many-to-many table for the bookmarks, linking entities to users. The argument here against some large column is that we can clean up this relation easier when a user or entity is purged - otherwise it is annoying to track down and update potentially many entities due to the row format.

---

Update on (1):

As I mentioned before, we don't need to add multiple columns to the table such that every time we need to add something new we have to write a migration. We can have something like:

message UserPreferences {
  bool dark_mode = 1;
  message DashboardLayouts {
  ...
  }
  DashboardLayouts dashboard_layouts = 2;
  ...
}

message User {
  ...
  UserPreferences preferences = 99;
}

This is in turn backed in the users table by a jsonb preferences column which contains the marshalled preferences. We can then update the preferences on a field by field basis, and any changes (new fields, new messages, deprecated/removed fields and messages) are just updates to the API, which don't require any database migrations.

[nicholaspcr]

I'm not so sure if its ideal to store the console user preferences on the Identity Server as it is a set of information which aren't attached to entities but rather to desired user behaviors on the front-end. Having said that, I still agree that its probably the best choice to store said information on the IS purely due to the decrease of complexity that this approach brings.

I also agree with the two approaches of storage given by @adriansmares.

[kschiffer]

Great, then let's go with the IS approach.

These are simply columns per user, right?

Let's see:

• Bookmarks: many-to-many table, or array column
• Dark mode: boolean in the user record
• Dashboard layout: this can potentially be more complex, since it should store panel ID, order and possibly a configuration; is it a good idea to store this as a JSON?
• Sort orders: this should ideally be saved per table, so I guess this would also be a JSON in the user table?
• Payload formatters: let's skip this for now, it was just a first idea

@KrishnaIyer can you help to schedule this in conjunction with the Console design updates?

[KrishnaIyer]

Yes. This is added as an item to https://github.com/TheThingsIndustries/product-management/issues/29.

[KrishnaIyer]

@nicholaspcr: Do you have sufficient input in the issue to start implementing this?

[nicholaspcr]

@KrishnaIyer yeah the suggestion made by Adrian is good solution. Will work on this after finishing the contact_info removal issues.

[nicholaspcr]

Updating this issue with the changes made to the IS, this should unblock the front-end issues related to these features.
Tagging @kschiffer so that the Console changes can be distributed to the front-end team.

---

Changes introduced by #6890

A new field console_preferences was introduced to user proto message:

{
    "console_theme": "('CONSOLE_THEME_SYSTEM'|'CONSOLE_THEME_LIGHT'|'CONSOLE_THEME_DARK')",
    "dashboard_layouts": {
        "api_key":      "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')",
        "application":  "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')",
        "collaborator": "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')",
        "end_device":   "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')",
        "gateway":      "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')",
        "organization": "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')",
        "overview":     "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')",
        "user":         "('DASHBOARD_LAYOUT_TABLE', 'DASHBOARD_LAYOUT_LIST', 'DASHBOARD_LAYOUT_GRID')"
    },
    "sort_by": {
        "api_key":       "same fields allowed in the respective list entity request",
        "application":   "same fields allowed in the respective list entity request",
        "collaborator":  "same fields allowed in the respective list entity request",
        "end_device":    "same fields allowed in the respective list entity request",
        "gateway":       "same fields allowed in the respective list entity request",
        "organization":  "same fields allowed in the respective list entity request",
        "user":          "same fields allowed in the respective list entity request"
    }
}

The idea here is that the front-end team can add new fields to the console_preferences proto definition without
requiring a migration, since the bytes of the object is what is stored on the database. So this field is malleable
enough for new fields at any point.

---

Changes introduced by #6968

A new registry was added UserBookmarkRegistry.

lorawan-stack/api/ttn/lorawan/v3/user_services.proto

Lines 176 to 208 in 282b6a3

	service UserBookmarkRegistry {
	// Create a bookmark for the given user.
	rpc Create(CreateUserBookmarkRequest) returns (UserBookmark) {
	option (google.api.http) = {
	post: "/users/bookmarks"
	body: "*"
	};
	}
	// List the bookmarks for the given user.
	rpc List(ListUserBookmarksRequest) returns (UserBookmarks) {
	option (google.api.http) = {get: "/users/{user_ids.user_id}/bookmarks"};
	}
	// Delete the given user's bookmark.
	rpc Delete(DeleteUserBookmarkRequest) returns (google.protobuf.Empty) {
	option (google.api.http) = {
	delete: "/users/{user_ids.user_id}/bookmarks/applications/{entity_ids.application_ids.application_id}",
	additional_bindings {delete: "/users/{user_ids.user_id}/bookmarks/clients/{entity_ids.client_ids.client_id}"}
	additional_bindings {delete: "/users/{user_ids.user_id}/bookmarks/gateways/{entity_ids.gateway_ids.gateway_id}"}
	additional_bindings {delete: "/users/{user_ids.user_id}/bookmarks/organizations/{entity_ids.organization_ids.organization_id}"}
	additional_bindings {delete: "/users/{user_ids.user_id}/bookmarks/users/{entity_ids.user_ids.user_id}"}
	additional_bindings {delete: "/users/{user_ids.user_id}/bookmarks/applications/{entity_ids.device_ids.application_ids.application_id}/devices/{entity_ids.device_ids.device_id}"}
	};
	}
	// Delete a list of bookmarks of the given user.
	// This operation is atomic; either all bookmarks are deleted or none.
	// Bookmarks not found are skipped and no error is returned.
	rpc BatchDelete(BatchDeleteUserBookmarksRequest) returns (google.protobuf.Empty) {
	option (google.api.http) = {delete: "/users/{user_ids.user_id}/bookmarks/batch"};
	}
	}

The operations are quite straight forward. Bookmarks are entities that can be soft-deleted but not by any of the
registry operations, only when an entity referenced is deleted that the bookmarks is soft-deleted.

So when user-X has a bookmark to entity-Y, if entity-Y is deleted and restored the bookmark will remain present and it
will be returned in the ListBookmarks request.

Additionally there is support for listing deleted bookmarks:

lorawan-stack/api/ttn/lorawan/v3/user.proto

Lines 554 to 555 in 282b6a3

	// Only return recently deleted bookmarks.
	bool deleted = 5;

But I'm not sure if it is wise to show deleted bookmarks, unless requested by users later on.

[kschiffer]

Thank you! I believe we can close this issue then, since all Console related issues are filed individually.

cc @ryaplots