feat: dont use group name as primary key

[fengelniederhammer]

resolves #1157

Please review and merge #1500 and #1505 first.

preview URL: https://1157-dont-use-group-name.loculus.org/

Summary

URLs use the group id now in the backend and on the website (e.g. /group/123).

Screenshot

PR Checklist

• [ ] All necessary documentation has been adapted.
• The implemented feature is covered by an appropriate test.

[JonasKellerer]

LGTM

[chaoran-chen]

https://1157-dont-use-group-name.loculus.org/dummy-organism/my_sequences/insdc_ingest_group?

I am getting an error message here:

I didn't check whether this was already the case in the past or introduced by this PR, but now I see the group twice in the dropdown menu (on the same page as above):

[fengelniederhammer]

Thanks, good catch. group is indeed not a field anymore, it should be groupName or groupId.

Seing the same group twice might be a result of the fact that the group actually exists twice? How is this group created?

[corneliusroemer]

Must be a bug somewhere, a new group seems to be created whenever one logs in:

The group is created by the ingest pipeline, the ingest pipeline checks whether a certain group exists, if it doesn't, it recreates.

So the PR requires some adjustment of the ingest code!

[corneliusroemer]

Ingest group creation logic needs to be adjusted to prevent repeated creation of the same insdc_ingest_group multiple times

Relevant code is here:
https://github.com/loculus-project/loculus/blob/2933576b5eeebdf431325a2863f44333addecee8/ingest/scripts/submit_to_loculus.py#L49-L79

[corneliusroemer]

Should we hide that groupid? It doesn't contain any useful information?

[fengelniederhammer]

I tried fixed the website (I found a couple more forgotten adaption).

@corneliusroemer I tried to adapt the script. Can you please check whether that might be correct? My Python isn't that good and I don't know how to test whether it works.

[fengelniederhammer]

Should we hide that groupid? It doesn't contain any useful information?

We could make it less prominent, but I added it because it's the relevant id now.

[chaoran-chen]

https://1157-dont-use-group-name.loculus.org/ebola-zaire/my_sequences/1? is empty, which is a sign that the ingest script is not working?

[corneliusroemer]

I've made it work with the ingest pipeline now, thanks for getting started with it @fengelniederhammer - there was a small bug but it's my bad as I have not documented how to test the ingest pipeline at all, other than in CI.

@chaoran-chen it works now:

resolved

[corneliusroemer]

Group id doesn't seem to make it to lapis:

https://1157-dont-use-group-name.loculus.org/seq/LOC_0001HFK.1

Chaoran's point also is still valid, that my sequences doesn't show anything despite sequences being there:
https://1157-dont-use-group-name.loculus.org/ebola-zaire/my_sequences/1?

[fengelniederhammer]

I rebased on the latest main.

The group id does make it to LAPIS: https://1157-dont-use-group-name.loculus.org/seq/LOC_000007L.1

Also the ingest seems to work: https://1157-dont-use-group-name.loculus.org/ebola-zaire/search?

(Thanks @corneliusroemer for fixing!)

[corneliusroemer]

Looks great! All issues appear fixed

[corneliusroemer]

I've improved search ui config: #1541, can be merged into this PR