[ENH] - Teams List / Look-up Table

[pjdohertygis]

Which app?

All Apps

Describe requirements

Real-world Problem: New teams sign up for SARCOP throughout the year and also with great urgency during incidents in States that we have not responded to before (e.g. 2024 NM Wildfires). When Teams "join" they want to see their team name in SARCOP for accountability.

Technical Problem: Currently we are spending many hours per month maintaining domains across several feature layers (Sandbox waypoints, Sandbox RST, Sandbox tracklogs, Deployment waypoints, Deployment RST, Deployment Tracklogs, Team Fact Sheet, US&R Catalog, etc.). This requires editing json and prone to error.

Proposed Solution: We want one team list to rule them all, aka "The List". All apps should use this team list when entering data. There are a few approaches worth considering.

COA1: Controlled Input (Domainless)

• Create a table feature layer, edit this when new teams are added
• QuickCapture - utilize table feature layer as project user input, this will stamp all incoming data (waypoints, tracklogs, RST, etc).
• Dashboards - filters can be used without domains. This has an added advantage on incidents, the list will be short.
• Intel Manager - filters can be used without domains.
• Smart Forms (Intel Manager / Field Maps) - entering data will require some sort of Arcade script, potentially that taps into the "The List" and or a list of usernames. Currently we do not know how to do this. We also need to see if Field Maps "filters" can work without domains. One idea for this is create a username / team name look-up, however, sometimes task forces deviate from this (e.g., Los Angeles County deploys as CA-TF2 when out of state but... CA-RTF when in-state). [ENH] - Intel Manager 2.0 Presets or Favorites #289

COA2: Controlled Domain (Update Domains)

• Create a table feature layer, edit this when new teams are added
• ETL - can we use Notebooks/Python something else that programmatically updates all of the domains at once from "The List", here is an example https://community.esri.com/t5/arcgis-api-for-python-questions/updating-feature-service-coded-value-domain-with/td-p/848244
• GeoJobe - is this an option? maybe in the short-term? https://geo-jobe.com/mapthis/get-to-know-our-toolbox-admin-tools-update-field-domains-for-feature-layers/

Alternatives

Continue to spend many hours maintaining several duplicate lists.... even with more staff this is prone to error.

Other Relevant Info

This issue has been open since 2020...

Priority impact

p1 - time sensitive

Impact

This has the potential to save the admin team hundreds of hours per year and also more quickly onboard new teams.

---

Original 2020 Post
Since we are using a secured system with usernames, shouldn't we be able to create a look-up table of usernames that indicates the team? With QuickCapture, we only have one Project User Input so we lose the option to fill-out the team name field (right now, we have it set to "Squad Number". In addition, Team Name must be manually entered in the Favorites for every Survey123 input (while they are revamping the workflow for this, it still seems redundant to have to manually enter something we should already know).

Here is a table we can use for testing: https://napsg.maps.arcgis.com/home/item.html?id=b88f39b77922455b8ad89abbe19cd92c

Requirements:

• It has to scale to +100,000 points
• Dynamic
• Easy to update when new teams join the system
• Work in all ArcGIS Apps
• Bonus if it is driven by the Hub Supporting Team Group

Constraints:

• We will only have control over user naming conventions that we create. Some outside agencies may use their own.
• We don't want to rely on IT / Contract support to maintain

Options to explore:

1. ArcGIS Online Hosted Feature Layer Joins: https://doc.arcgis.com/en/arcgis-online/analyze/join-features.htm @afackler has some experience using these. Do we know if these are scalable (100,000 points)? Can they be used to filter data in Dashboards and Web Apps?

2. Arcade: This would be one impressive expression.... it would need to cover all 28 FEMA Teams and then countless SUSAR Teams. It might be difficult to keep up to date if the Teams list is dynamic?

3. Jupyter Notebook: Use Python "in the cloud" https://developers.arcgis.com/python/guide/using-the-jupyter-notebook-environment/

[IsmaelInRedlands]

I like the idea of creating a lookup table very much. It will eliminate user entry errors and makes data collection simpler. I think a Python script is a good approach. Another option is a webhook. See details at: https://community.esri.com/t5/arcgis-quickcapture-blog/data-enrichment-with-arcgis-quickcapture-and-integromat/ba-p/892203

[t-mahon]

How about using 'USERNAME' question type to then run a calculation for Team Name. For FEMA Task Forces, the username format is XXTF#. Saving the username to a field (can be null to prevent unnecessary pass through data) and then calculate the Team Name by taking the first 5 characters. Formatting could be made prettier by inserting the dash if that's preferred by the powers that be.

This may cause a few issues for the state teams, but perhaps the lure of less items to fill out will help them standardize. This may scale better then having a list and might even be able to handle squad numbers when more FEMA team logins become available. Example: XXTF#_SQ# from that we can take substr(${agol_id},0,5) for Team Name and substr(${agol_id},7,9) for Squad Name. This will reduce repeat user inputs down to Search Type.

[IsmaelInRedlands]

@t-mahon If people adhere to the XXTF# format, I think your proposal would make the survey easier to complete and will reduce user-input errors. It will also be the easiest to maintain from a Survey123 perspective (no CSV table needed). I guess the issue is what happens if people do not adhere.
It may help to calculate the Team name and Squad automatically using the technique you describe and let the user ignore the inputs (if they are correct), or manually update them, if they are not. In fact, what about hiding the Team and Squad questions if you find a good match, and only show them when absolutely necessary? I am guessing that it thins may drive people to adopt the format. If you adopt it, then you do not have to complete the tea and squad. If you dont, then you need to type them in manually.

[t-mahon]

@IsmaelInRedlands good idea on displaying/hiding the fields based on username schema compatibility. I mocked up a couple of tests, at the moment it looks like the test is case-sensitive, but other than that, it works. So this would mean we could define a username schema, it's easy if you use it, harder if you don't. If their AGOL username doesn't fit schema, then they're prompted to enter a team name.

With the FEMA assets there's obviously greater control, the state and local assets are a bit more freewheeling. I know for FDNY, we adopted the name_FDNY_internalorg (tmahon_FDNY_IMT, admin_FDNY_SOCTF, etc) format. Based on how we found our names displayed in cooperative groups and we felt that agency identification was important. I'm sure other agencies have made their own determinations.

Questions:

• Is this worth trying to fit into v8?
• Under the current AGOL login scheme, it would only define Task Force.
• Can FEMA deploy extra logins? If so, would it make sense to go to: XXTF#_Sq# each squad gets a login for the team. Then we could whack up that name into the TF and Sq fields of the survey. Also makes the test more accurate as we can look for _SQ in a certain position.
• Leave the Squad Leader field in the schema, and an IST or savvy TIS could add in GIS/Intel Manager based off the who was assigned to which squad, if needed.

Probably should've tried this earlier in the v8 development, apologies for getting on the ball late.

[pjdohertygis]

Team - I think @jdokemaps and I have a short-term workaround to reconcile S123 and QuickCapture, while also allowing for future development.

1. Add a new field called team_name_filter - this will be the "end state" for team name that we filter on. It has no domains and is just string.
2. Survey123 - this will auto calculate from from the team_name dropdown. If it does not match one of the standard teams in our dropdown at the time of publishing, then it prompts the user to enter into team_name_filter directly. Ideally this will also have constraints / input-masks
3. QuickCapture - the Project User Input will be pointing to team_name_filter and we can use the input type = choice list and copy and paste from the choices in Survey123 for team_name.

When new established teams emerge we can easily add them to QuickCapture without a breaking change. For Survey123, we will need to wait until a pause in the incident or new incident to add the new team names into the domain. We are going to test this out on 8b right now.

@IsmaelInRedlands @t-mahon

[IsmaelInRedlands]

Starting with 3.12, you can reference in Survey123 a CSV file stored in ArcGIS as an item.
The advantage of this is that if the CSV file is updated in ArcGIS Online, the app automatically downloads it so the list choices are up to date.
If you want to give this a go:
-Create a CSV file with one column called name and the other label
-Copy your CSV into the media folder of your form
-Reference your CSV file using the select_one_from_file myFile.csv question type
-Publish you survey
-Next upload your CSV file from the media folder as a new item in ArcGIS Online. Share the item accordingly.
-In Survey123 Connect, go to linked content and link your survey to your CSV item
-Once linked, you can remove your CSV file from your media folder
-As you upload new versions of your CSV file to ArcGIS Online, the app should update the list automatically (as long as you have 3.12).
It may be a bit early to do the above given that people will unlikely have 3.12 in their devices, but this is something that could help you in the future.
We are also working towards allowing a list to be populated from a feature service query in the future (this is well documented in the Early Adopter Community website)

[pjdohertygis]

@IsmaelInRedlands this is awesome! We will look into this soon. For now we have a short-term solution for both Survey123 and QuickCapture (we think).

[t-mahon]

@pjdohertygis and @jdokemaps that sounds like solid work around for the moment. The team_name_filter idea is a good one, and I don't think we have many team additions in the straight FEMA system. Obviously for the state teams that's more dynamic. Down the road if we can steer the naming conventions in a certain way that may be a help.

@IsmaelInRedlands That's going to be a helpful feature. We might be using it in the coming weeks for a FDNY IMT mission.

[pjdohertygis]

Here is a video that further explains the concept, although limited to Survey123 https://www.youtube.com/watch?t=293&v=EZRhNU_SPQY&feature=youtu.be

[pjdohertygis]

Added to Planner for investigation.

[afackler]

I tested the CSV list for S123 and it works as expected. I just followed Ismael's steps, though he's right you need Connect/ app version 3.12 or higher for this to work (I was using 3.13). The CSV downloads with the survey and automatically redownloads when you open the form if changes have been made to the CSV on AGOL. With this, you don't have to republish the whole survey if you want to add or remove team names. Since the sheet is downloaded as well, it works offline.

[pjdohertygis]

Nice work @afackler!

• Parity with QuickCapture?
• Could the Team Name question be filled in the background based on their login so it is one less question for them to answer?

[afackler]

Answering the first question - I can't do anything with the CSV in QuickCapture. There's nothing in the options to even suggest it's possible. I think it would be cool to have this capability - being able to change a list without republishing the whole project, but for the time being, I don't see any way of doing it.

As for the lookup, I took an approach of changing the survey's team name calculation to return the username of who’s signed in: property("username"). Then I assigned usernames to teams in the CSV linked to the survey (you would have 3 rows, 3 different usernames, but all have the same team name. I put a snapshot of this below). Though this worked on the front end, i.e. in the survey it would show the correct data, it did not bring the label to the actual database. Even so, that'd still only be the label, and making filters wouldn't be possible anyway.

If we were doing this once, we could make a really, really long if statement in the calculation. That'd probably work, but that's not practical. I'm working on making the notes easier to read and will put them in the USAR OneNote

[pjdohertygis]

For later consideration.... we may want to pull the Team Name field out and stop using domains altogether on the feature layer.

Survey123

• In Survey123 we used the Team Name field as a domain so people have a dropdown.
• When they fill this out, it also fills out the Team Filter field.
• Team Other only gets used when they don’t exist in the dropdown.

QuickCapture

• In QuickCapture we ONLY fill out Team Name Filter
• Team Name cannot be filled because it has a domain, which is why we made Team Name Filter in the first place.

Therefore, we could… stop using Team Name in Survey123 and only use Team Name Filter. We’d update the dropdown for the Team Name Filter question, but leave the domain blank. Because the dropdown will format the structure for us.

This would reduce confusion over Team Name fields in the long run and reduce the need to update the feature layer each time we add a new team. We'd only update the dropdowns in the UI for QuickCapture and Survey123.

Anyone opposed to this?

[pjdohertygis]

@kstrube is making progress on this one - stay tuned!

[pjdohertygis]

Let's make sure we add a Team Name field to segments layer as per #190

[pjdohertygis]

@kstrube one additional layer we will want to test is our interest survey - we can use the built-in Survey123 choice list from layer OR just stick with domains - discuss pros and cons of both.
https://survey123.arcgis.com/share/e7ffaf9005bc4da6b5acdd4d6f3a139c